View on GitHub


Imagining hidden supporting objects using volumetric conditional GANs and differentiable stability scores

Hector Basevi and AleŇ° Leonardis

This webpage contains resources associated with the paper Imagining Hidden Supporting Objects using Volumetric Conditional GANs and Differentiable Stability Scores to appear at the 33rd British Machine Vision Conference in November 2022.

The paper:

The data: Coming soon! (After BMVC.)

Any questions: Contact Hector

We present a system trained to imagine the plausible hidden content of scenes using differentiable context from a network trained to score the physical stability of scenes. This feedback on physical stability is crucial to:

Our system consists of a 3D (voxel) conditional Wasserstein generative adversarial network which receives conditioning on visible depth and semantic information and produces a complete volumetric semantic labelling of the scene. The labelling can then be parsed into a set of object instances for use in other tasks. Crucially, feedback on physical stability context is provided by a stability scoring network and treated as conditioning rather than as an optimisation objective.

System design

We can visualise an example by parsing the outputs of a non-generative system, a generative system without stability conditioning, and our full generative system with stability conditioning. We see when simulating the behaviour of the parsed objects that only our full system imagines a scene which is stable.

Non-generative Generative without stability Generative with stability