OpenAI’s fiction-spewing AI is learning to generate images

Artificial intelligence / Machine learning

OpenAI’s fiction-spewing AI is learning to generate images

By training GPT-2 on pixels instead of words, the model can accept half an image and predict how to complete it.

At its core, GPT-2 is a powerful prediction engine. It learned to grasp the structure of the English language by looking at billions of examples of words, sentences, and paragraphs, scraped from the corners of the internet. With that structure, it could then manipulate words into new sentences by statistically predicting the order in which they should appear.

So researchers at OpenAI decided to swap the words for pixels and train the same algorithm on images in ImageNet, the most popular image bank for deep learning. Because the algorithm was designed to work with one-dimensional data, i.e.: strings of text, they unfurled the images into a single sequence of pixels. They found that the new model, named iGPT, was still able to grasp the two-dimensional structures of the visual world. Given the sequence of pixels for the first half of an image, it could predict the second half in ways that a human would deem sensible.

Below, you can see a few examples. The left-most column is the input, the right-most column is the original, and the middle columns are iGPT’s predicted completions. (See more examples here .)

OPENAI

The results are startlingly impressive and demonstrate a new path for using unsupervised learning, which trains on unlabeled data, in the development of computer vision systems. While early computer vision systems in the mid-2000s trialed such techniques before, they fell out of favor as supervised learning, which uses labeled data, proved far more successful. The benefit of unsupervised learning, however, is that it allows an AI system to learn about the world without a human filter, and significantly reduces the manual labor of labeling data.

The fact that iGPT uses the same algorithm as GPT-2 also shows its promising adaptability across domains. This is in line with OpenAI’s ultimate ambition to achieve more generalizable machine intelligence.

At the same time, the method presents a concerning new way to create deepfake images. Generative adversarial networks , the most common category of algorithms used to create deepfakes in the past, must be trained on highly curated data. To get a GAN to generate a face, for example, its training data should only include faces. iGPT, by contrast, simply learns enough of the structure of the visual world across millions and billions of examples to spit out images that could feasibly exist within it. While training the model is still computationally expensive, offering a natural barrier to its access, that may not be the case for long.

OpenAI did not grant an interview request, and therefore did not provide additional context for  future plans with regards to its research. But in an internal policy team meeting that MIT Technology Review attended last year, its policy director Jack Clark mused about the future risks of GPT-style generation, including what would happen if it were applied to images. “Video is coming,” he said, projecting where he saw the field’s research trajectory going. “In probably five years, you’ll have conditional video generation over a five to ten second horizon. The sort of thing I’m imagining is eventually you’ll be able to put a photo of Angela Merkel as the condition, with an explosion next to her, and it will generate a likely output, which will be Angela Merkel getting killed.”