generative adversarial networks (gans) are a type of deep neural network used to generate synthetic images. the architecture comprises two deep neural networks, a generator and a discriminator, which work against each other (thus, “adversarial”). the generator generates new data instances, while the discriminator evaluates the data for authenticity and decides whether each instance of data is “real” from the training dataset, or “fake” from the generator.
together, the generator and discriminator are trained to work against each other until the generator is able to create realistic synthetic data that the discriminator can no longer determine is fake. after successful training, the data produced by the generator can be used to create new synthetic data, for potential use as input to other deep neural networks.
gans are versatile in that they can learn to generate new instances of any datatype, such as synthetic images of faces, new songs in a certain style, or text of a specific genre.
training a gan
using an example of creating synthetic images of money, let’s walk through the specific parts and functions of a gan architecture.
- noise is fed into the generator. since the generator hasn’t been trained yet, the output will look like noise in the beginning.
- training data and the output of the generator is sent to the discriminator, which is being trained in parallel to identify real/fake images. the output of the discriminator at the beginning will not be very accurate as this portion of the network is also being trained and accuracy will improve over time.
- feedback: the output of the discriminator can be fed back to the generator and the discriminator, which can use this information to update parameters and attempt to improve on the accuracy.
the goal of the discriminator, when shown an instance from the true dataset, is to recognize those images that are authentic. meanwhile, the generator is creating new, synthetic images that it passes to the discriminator. it does so in the hopes that they, too, will be deemed authentic, even though they are fake. the goal of the generator is to generate passable images: to lie without being caught. the goal of the discriminator is to identify images coming from the generator as fake.
matlab® and deep learning toolbox™ let you build gans network architectures using automatic differentiation, , and shared weights.
applications of generative adversarial networks
handwriting generation: as with the image example, gans are used to create synthetic data. this can be used to supplement smaller datasets that need more examples of data in order to train accurate deep learning models. one example is handwriting detection: in order to train a deep neural network on handwriting, thousands of samples of training data are needed, and to collect this data manually may be time-intensive.
synthetic handwriting generation using gans.
scene generation: conditional gans are a specific type of gan takes advantage of labels, while the original gan does not assume labels will be present. conditional gans can be used in applications such as scene generation, where there must be a certain organization to the information. take the example of scene generation for automated driving. the road and sidewalk must be located below the buildings and sky. a synthetic image created for this example that does not adhere to the location of the road will immediately be determined as fake and unusable in an automated driving application.
image-to-image translation (pix2pix) using conditional gans.
audio and speech applications: gans are also used for applications such as text-to-speech synthesis, voice conversion, and speech enhancement. gans provide significant advantage over traditional audio and speech implementations as they can generate new samples rather than simply augment existing signals. one example in which gans are used for sound synthesis is to create synthetic version of drum sounds:
note: gans can be powerful in generating new synthetic data for use in many applications, yet it is often challenging to arrive at accurate results due to many failure modes that may take place. matlab lets you .