Smart Object Extraction from a Picture and Blending into Another
Highlights
A new efficient method for recovering reliable local sets of dense correspondences between two images with shared content.
The method could be applied for:
-
automatic adjusting of the source’ tonal characteristics to match a reference,
-
transferring a known mask to a new image,
-
kernel estimation for image de-blurring
Innovation
The approach simultaneously recovers both a robust set of dense correspondences between sufficiently similar regions in two images and a global non-linear parametric color transformation model. A new coarse-to-fine scheme is utilized in which nearest-neighbor field computations using Generalized PatchMatch [Barnes et al. 2010] are interleaved with fitting a global non-linear parametric color model and aggregating consistent matching regions using locally adaptive constraints.
Below is an example of color transfer.
The reference image (a) was taken indoors using a flash, while the source image (b) was taken outdoors, against a completely different background, and under natural illumination. The correspondence algorithm detects parts of the woman’s face and dress as shared content (c), and fits a parametric color transfer model (d). The appearance of the woman in the result (e) matches the reference (a).
Advantages
-
New correspondence method that combines dense local matching with robustness to outliers.
-
Identification of correspondences between non-rigid objects with significant variance in their appearance characteristics, including dramatically different pose, lighting, viewpoint and sharpness.
-
Synergy of two worlds advantages: dense, like optical flow and stereo reconstruction methods, and robust to geometric and photometric variations, like sparse feature matching.
The method for computing a reliable dense set of correspondences between two images specifically designed to handle a third scenario, where the input images share some common content, but may differ significantly due to a variety of factors, such as non-rigid changes in the scene, changes in lighting and/or tone mapping, and different cameras and lenses. This scenario often arises in personal photo albums, which typically contain repeating subjects photographed under different conditions.
Patent Status
Granted US 9,014,470