ECE team develops facial recognition software

December 22, 2008

Rowan University College of Engineering students have spent the fall working with Mona Lisa, and they are pleased to report - scientifically speaking - that the iconic subject of one of Leonardo da Vinci's greatest works is quite happy.

Electrical and computer engineering (ECE) majors Jessica Dennis, 22, of Mt. Olive, and Michael Ulrich, 23, of Riverside, have been developing facial expression recognition software as part of a senior elective, "Artificial Neural Networks," taught by ECE chair Dr. Shreekanth Mandayam. One of the expressions they evaluated was that of the Mona Lisa, perhaps the most famous face in the history of art, whose enigmatic look has long been the subject of speculation.

The students started their work, a type of artificial intelligence, feeding into their software program faces from the Japanese Female Facial Expression (JAFFE) Database, a standard catalog used worldwide by engineers and other scientists, among others, for various experiments and projects. JAFFE includes photographs of numerous women exhibiting seven expressions: anger, disgust, fear, happy, neutral, sad and surprise.

According to Dennis, her and Ulrich's goal was to develop a software program that has the ability to read facial expressions, which is not quite the same as reading emotions but may be a cue to them.

The challenge after incorporating the JAFFE database was to code the expressions in a way in which the software program could recognize the expressions and evaluate future ones by comparison.

"A computer can only deal with numbers," Mandayam explained. "If you can get a small set of numbers that correspond, for instance, to a happy or angry expression, you can automate the program's decision-making process about future images it is shown."

To get those numbers, the students focused on four key measurements in 210 photos: eyebrow height and width and mouth height and width, considered the most significant in determining expressions and emotions. They also examined secondary measurements, including how wide eyes open (most obvious in surprise) and jawlines (strong indicators of disgust). Processing the images in binary code, Dennis and Ulrich used 140 of the shots to "train" the software and 70 to "test" it.

After testing JAFFE photos, they also tested photos of people at Rowan and other recognizable subjects. The duo determined their software program had an overall 76.5-percent accuracy rate for all seven expressions with which they started and a 94.2-percent accuracy rate for surprise, fear and anger alone. The highest rating-that is the emotion that the software identified right the most frequently-was surprise, at just over 87 percent. The lowest was fear, at 68.8 percent.

In the case of the Mona Lisa, the program was 82.5 percent confident that the artist's subject was happy.

"Because the Mona Lisa doesn't have any facial hair, which was the style back then, we were worried that the results would not be accurate," Ulrich said. "To our surprise, we found that even without a dominant facial feature like eyebrows, the network was still able to distinguish her facial expression with a high confidence."

The Rowan Engineering artificial intelligence efforts is more than a curiosity, the students and their professor said. In theory, it could be used for human-computer interactions (such as enabling a computer in a car to determine if a driver is overly tired), face image compression (useful for videos), synthetic face animation (for movies and video games), videoconferencing, and crowd control (evaluating facial expressions as a security measure at, for instance, a heated soccer game).

While they were pleased with the outcomes of their work, it is not yet foolproof. Dennis noted, as an example, that one subject they evaluated - The Incredible Hulk - could only be determined to be angry at a 56-percent confidence level.

"We can't suggest our program be used for nonhumans or humans who have been exposed to gamma radiation," she noted.



###