Imagining hidden supporting objects using volumetric conditional GANs and differentiable stability scores
Hector Basevi and Aleš Leonardis
This webpage contains resources associated with the paper Imagining Hidden Supporting Objects using Volumetric Conditional GANs and Differentiable Stability Scores to appear at the 33rd British Machine Vision Conference in November 2022.
The paper: research.birmingham.ac.uk.
The data: Coming soon! (After BMVC.)
Any questions: Contact Hector
We present a system trained to imagine the plausible hidden content of scenes using differentiable context from a network trained to score the physical stability of scenes. This feedback on physical stability is crucial to:
-
Imagining scenes which are themselves stable through the imagination of hidden supporting objects.
-
Replicating the distribution of hidden objects found in the training data.
Our system consists of a 3D (voxel) conditional Wasserstein generative adversarial network which receives conditioning on visible depth and semantic information and produces a complete volumetric semantic labelling of the scene. The labelling can then be parsed into a set of object instances for use in other tasks. Crucially, feedback on physical stability context is provided by a stability scoring network and treated as conditioning rather than as an optimisation objective.
We can visualise an example by parsing the outputs of a non-generative system, a generative system without stability conditioning, and our full generative system with stability conditioning. We see when simulating the behaviour of the parsed objects that only our full system imagines a scene which is stable.
| Non-generative | Generative without stability | Generative with stability |
|---|---|---|