Image Warping and Mosaicing

Part 1 & 2. Shoot the Pictures and Recover Homographies

In this part, I took photos to be used in this project, which are shown in each respective part in this project. These photos are taken by fixing the center of projection and rotating the camera around the center of projection. For each series of images, I took two photos that overlap about 50% with the previous photo. Then, I used the provided Correspondence Tool to define correspondences between the photos. The correspondences are used to recover homographies between the photos using the least squares method. To that end, we define the homography matrix \( H \), and solve for the unknowns \( a, b, c, d, e, f, g, h \).

\[ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \\ \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} wx' \\ wy' \\ w \end{bmatrix} \] If we expand this out, we have the following system of equations: \[ \begin{aligned} ax + by + c &= wx' \\ dx + ey + f &= wy' \\ gx + hy + 1 &= w \end{aligned} \] This implies: \[ \begin{aligned} ax + by + c &= (gx + hy + 1)x' \\ dx + ey + f &= (gx + hy + 1)y' \end{aligned} \] Expanding further: \[ \begin{aligned} ax + by + c - gxx' - hyx' &= x' \\ dx + ey + f - gxy' - hyy' &= y' \end{aligned} \] This can be written as the following matrix form: \[ \begin{bmatrix} x & y & 1 & 0 & 0 & 0 & -xx' & -yx' \\ 0 & 0 & 0 & x & y & 1 & -xy' & -yy' \\ \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \\ e \\ f \\ g \\ h \end{bmatrix} = \begin{bmatrix} x' \\ y' \end{bmatrix} \]

When the number of point correspondences \( n \) exceeds four, the system of equations becomes overdetermined, meaning there are more equations than unknowns. In such cases, it is generally impossible to find an exact solution that satisfies all equations simultaneously due to measurement noise and inaccuracies. To address this, we use the least squares method, which seeks to find the parameter vector \( \mathbf{h} \) that minimizes the sum of the squared differences between the left and right sides of the equations. Mathematically, this can be expressed as:

\[ \min_{\mathbf{h}} \|A\mathbf{h} - \mathbf{b}\|_2^2,\ \ \mathbf{h} = \begin{bmatrix} a & b & c & d & e & f & g & h \end{bmatrix}^\top \] Using least squares: \[ \mathbf{h} = (A^\top A)^{-1} A^\top \mathbf{b} \] Once vector \( \mathbf{h} \) is found, the homography matrix \( H \) can be reconstructed. \[ H = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \\ \end{bmatrix} \]

Part 3. Warping Images and Image Rectification

In this part, I warped the images using the homographies recovered in the previous part. In the correspondences tool, I selected four points that define the corners of the rectification region. Then, I manually defined the rectangular region where the corners should be warped to. The homography matrix is used to warp the image to the rectangular region. Here are some results:

Chocolate Ad

Rectified Chocolate Ad

Taco Menu

Rectified Taco Menu

Part 4. Blend the Images into a Mosaic

For this section I blended the images into a mosaic. I first warped the images to the same coordinate system using the homographies recovered in the previous part. Then, I blended the images using two methods: naive blending and Laplacian blending. Naive blending simply overlays the transformed image on top of reference image. Laplacian blending uses a 2-level Gaussian stack (i.e. high and low frequency components) to blend the two images. For each example, I created a custom mask for Laplacian blending. Here are some results: