Facebook AI introduces a new imaging model called “IC-GAN” that creates high quality images of unknown objects and scenes


Source: https://ai.facebook.com/blog/instance-condition-gans/

Generative Adversarial Networks (GANs) have been used for some years to create photorealistic images of objects or scenes that are very similar in style and content. Previously, however, these models could only produce output related to the datasets they were trained on – which had limitations as these files were typically less diverse than when generating new ideas. A conventional GAN ​​trained on images of cars shows impressive results when asked to generate other images of cars or cars. However, the trained GAN is likely to fail if it receives a flower or an object outside of its automotive record.

The failure to display non-identical objects from the training dataset is a major limitation and definitely needs to be addressed to meet the demand. Facebook is trying to solve the above problem by introducing Instance-related GAN (IC-GAN). the IC-GAN is a new imaging model that, with a few inputs, can produce high quality images even if it does not appear in the training set. The special thing about the IC-GAN model is that it can generate realistic, unforeseen combinations of images – for example a camel in the snow or zebras walking through an urban cityscape.

Unsurprisingly, IC-GANs could be used to create visual examples of datasets with these new capabilities. This would allow artists and creators alike to create richer AI-generated content by creating art from photos or videos, just as an artist could draw a picture with pencils and brushes.

In the proposed approach, the IC-GAN can be used with both marked and unmarked records. It applies a GAN framework to model a mixture of local clustering in one image and overlapping samples from different regions or neighborhoods in images that are similar to their nearest neighbors in the same data set. They use “neighbors” for the input layer into Discriminator’s neural network, which helps the generator create realistic looking sample images by mixing real pixels with predicted ones. Thus, the model could use data sets more efficiently because it avoids the problem of partitioning into small clusters.


Extending data with IC-GAN can help model builders include elements or objects that are not normally found in the training set. In addition, this approach can be an effective way to enhance your research efforts as it works across domains and generates more diverse examples of object recognition models. Conventional GAN ​​models can only generate images of zebras in the grassland, as their training data consists exclusively of such images. If you try to train a GAN with urban areas like New York City or Los Angeles it will fail because there aren’t enough examples from which the model can learn what features define those landscapes – like buildings or trees. The IC-GAN model can generate novel combinations of data by including objects that are not normally seen in standard data sets. For example, research shows how she created a picture of cows walking on the sand – this is just one option they choose from many options.

Paper: https://arxiv.org/abs/2109.05070?

Code: https://github.com/facebookresearch/ic_gan?

Source: https://ai.facebook.com/blog/instance-condition-gans/


Leave A Reply