Smart Object Extraction from a Picture and Blending into Another

Highlights

A new efficient method for recovering reliable local sets of dense correspondences between two images with shared content.

The method could be applied for:

• automatic adjusting of the source’ tonal characteristics to match a reference,
• transferring a known mask to a new image,
• kernel estimation for image de-blurring

Innovation

The approach simultaneously recovers both a robust set of dense correspondences between sufficiently similar regions in two images and a global non-linear parametric color transformation model. A new coarse-to-fine scheme is utilized in which nearest-neighbor field computations using Generalized PatchMatch [Barnes et al. 2010] are interleaved with fitting a global non-linear parametric color model and aggregating consistent matching regions using locally adaptive constraints.

Below is an example of color transfer.

The reference image (a) was taken indoors using a flash, while the source image (b) was taken outdoors, against a completely different background, and under natural illumination. The correspondence algorithm detects parts of the woman’s face and dress as shared content (c), and fits a parametric color transfer model (d). The appearance of the woman in the result (e) matches the reference (a).

Advantages

• New correspondence method that combines dense local matching with robustness to outliers.
• Identification of correspondences between non-rigid objects with significant variance in their appearance characteristics, including dramatically different pose, lighting, viewpoint and sharpness.
• Synergy of two worlds advantages:  dense, like optical flow and stereo reconstruction methods, and robust to geometric and photometric variations, like sparse feature matching.

The method for computing a reliable dense set of correspondences between two images specifically designed to handle a third scenario, where the input images share some common content, but may differ significantly due to a variety of factors, such as non-rigid changes in the scene, changes in lighting and/or tone mapping, and different cameras and lenses. This scenario often arises in personal photo albums, which typically contain repeating subjects photographed under different conditions.

Current development stage

TRL4 Technology validated in lab

Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. The methods may be implemented in software, hardware, or a combination thereof. The order of method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.

Patent Status

Granted US 9,014,470