Recently, Simon Colton, Ph.D. student Michel Valstar, and Maja Pantic from Imperial College London department of computing (London, UK) won the prestigious award for the best machine-intelligence demonstration at the British Computer Society's Specialist Group on Artificial Intelligence (SGAI) conference, held at Cambridge University. The SGAI award, voted by some of the world's foremost authorities on artificial intelligence (AI), recognized the team's innovative and creative automated software technology--The Painting Fool, which creates portraits by reading the facial expressions of a human subject and interpreting the emotion conveyed by expressions.
Colton, who designed The Painting Fool, said he was excited by the challenge of creating an automated artist, which produces beautiful pieces of artwork. He also said his work raised some interesting questions about how people perceive creativity. "I'm a computational creativity researcher, which is a sub-area of AI research, so The Painting Fool can and has been used to test out some theories about how people perceive software as creative, or not. For instance, because people have to watch The Painting Fool produce its artwork with painstaking detail they seem to project more value onto it than they would if Photoshop produced something in milliseconds," he said.
To create a painting, The Painting Fool program first takes an image of its subject, via a video camera, and segments it into a number of regions of color. It then takes each segment and simulates how a human artist might paint that region. For instance, The Painting Fool may choose to imitate an artist's brushstrokes by painting a circular region with ever decreasing circles or straight lines with fine strokes.
In addition to simulating the artist's style of work, it also replicates what art materials, such as acrylics, pastels, chalk, or watercolors would look like.
"In recent developments, I've enabled the choice of color palette and materials to be controlled by other software. For the competition we used machine-vision software to control the painting styles, so the portraits were appropriate to the emotion being displayed by the sitter," Colton explained.
Colton said this technology allows the The Painting Fool to use a number of techniques to depict emotions. For instance, if a person displays anger it may paint the subject in bold angry reds and black and use harsh brushstrokes to represent rage. This type of intelligent interpretation of human feelings was possible because of the work of fellow contributor Michel Valstar, a doctoral student of Maja Pantic. Michel contributed software that helped The Painting Fool interpret human emotions. As part of his degree, Michel developed the Facial Action Analysis system used by The Painting Fool to recognize complicated facial features.
"I developed a computer system that can detect 23 out of a total 27 facial muscle actions, known as Action Units, or AUs. These AUs help to describe typical expressions of the six basic emotions: anger, disgust, fear, happiness, sadness, or surprise," explained Michel.
Apart from detecting which AUs were present, Michel says, this technology was the first in the world to actually pinpoint when an AU starts, reaches its peak, and starts to return to its normal expression. "We use this knowledge to select from a video frame where the emotion shown was most intense. We call that the apex frame," he said. The apex frame is sent to The Painting Fool program, together with information about which emotion was detected and where the facial points were at that time. Michel said he was pleased to see his work actually demonstrated in the real world.
Colton said he would also like to purchase a robotic arm to do the painting in the real world, instead of a computer screen. He is also working on enabling The Painting Fool to design and paint its own 3-D scenes. "I also hope to announce a new project with a linguist from Oxford University to get The Painting Fool to read the front page of the New York Times and paint a picture that represents the events and emotional content of the story."
