3D Object Manipulation Using Generative Neural Networks

Zoran Amit, HUJI, School of Computer Science and Engineering, Computer Science
Bezalel Shimon, HUJI, School of Computer Science and Engineering



3D Modeling, Deep Neural Networks, Computer Aided Design


Generative Neural Networks, 3D Objects

3D object generation is the new frontier in the fields of computer vision and graphics. Whether for 2D to 3D reconstruction tasks, or manipulating and fine tuning an objects’ final form, the demand is high for a deep neural network that can tackle this general problem. Animators and CAD designers have a keen interest in streamlining their workflows to produce 3D geometries in an intuitive and expressive manner. While domain specific algorithms exist that solve generation or manipulation in their respective parametric space, such as that of character rigging, none such holistic solutions exist for all objects and forms. This could be attributed to a difficulty in amassing 3D datasets, which are expensive to scan or construct when compared to the ease and pervasiveness of the camera. The limited datasets that do exist are class specific, often exhibiting cars or chairs, and are relatively small in size. Therefore, we propose a generative model that trains on a single 3D object, and in turn can produce novel manipulations on that object that is both expressive and true to the source. Click here for video demo

Our Innovation
We have developed a design tool for manipulating arbitrary 3D geometries and volumes in a manner that is intuitive, continuously fluid, and preserves the source’s identifying features such as details, patterns, and texture. Our users interact with an attractor-based manipulator, with manageable degrees of freedom. A proxy object tightly coupled to the source’s geometry is manipulated, resulting in changes to the source. We then utilize a deep-neural-net to map this manipulation into 3D reconstructions of geometries of higher detail. This approach of a deep neural network, trained on a single object shows promising results in reconstruction as well as decreased costs for 3D data collection.


This approach has the potential to surpass the state-of-the-art in 3D reconstruction in a wide range of <b>industrial</b> applications where required, including self-driving cars, warehouse robots, drones, and aerial mapping, as well as CAD and animation modeling. Internally, we can automate the process of generating whole datasets of a single source object, to further train other 3D networks.


Contact for more information:

Anna Pellivert
Contact ME: