Background

Image manipulation has become extremely prevalent in recent years due to its commercial importance and the sheer number of popular image driven media forums. With the click of a button, users are able to sharpen images, layer them with flattering filters and even create unique looking sketches and profiles!

The ability to edit images on a high-level, however, remains the domain of experts using complicated software. For example, one needs a professional graphic artist to rearrange objects in a scene while preserving the picture’s natural elements. Even manipulating the shape of various objects while maintaining its integrity is not a simple task—how do you turn a sedan into a sports car while maintaining the color and style of the original picture?

Technical Barriers & Solutions

One of the technical challenges preventing the ease of image manipulation is that a dataset containing a large number of highly related images to the target image is currently required to “train” software in how to appropriately edit the target image. This is not problematic when dealing with common images such as faces or cars. Indeed, current algorithms depend upon these common images to allow us to make easy adjustments to popular images.  Technology, however, is unable to create algorithms for uncommon images, severely limiting today’s easy editing software.

Innovation: Deep Image Manipulation

Our innovative technology—deep image manipulation–invokes deep learning to engage a series of neural networks that allows users to change the structure of an image and easily adapt pixels within an image to be a new shape and/or structure while preserving the image’s original “DNA.”

We have found that the key to enabling single image training is extensive augmentation of the input image that proposes a novel augmentation method. Our network learns to map between a primitive representation of the image (e.g. edges) to the image itself. At manipulation time, our generator allows for making general image changes by modifying the primitive input representation and mapping it through the network. We have extensively evaluated our method and found that performs remarkably well.

Benefits of Our Deep Manipulation Technology

  • The technology only needs one picture to collect data: a training dataset is not needed; the target image is the only data required.
  • Preserves the DNA of the image far more accurately than other machine-learning based approaches available today.
  • Our technology makes image manipulation easy for everyone: it can be used by amateur consumers, but the deep learning that underscores it also makes it powerful enough for graphics’ professionals and designers.

Current Status

Proof of Concept established

Examples

Watch our Technology in action here: https://bit.ly/3f7ETXe

Graphic Examples of our Technology at Work

Figure 3: Results of our SP on challenging image manipulation tasks. Left: the edge-image pair used to train our method. Center: switching the positions between the middle car and the car on the right. Right: removing the car on the left and filling in the background space left by its removal. In both cases our method was able to synthesize very attractive output images.