We already know that gestures are an important component in communication. They help us to communicate information more clearly, and they can make that information more memorable to our listeners.
Emotions are a kind of information too. That's right, reading someone's emotion isn't as simple as reading his or her facial expression. What people often call our body language, how we stand, how we hold our shoulders, and whether we fidget, for instance, can all reveal information about what we're feeling, even if our faces are working to hide that emotion.
In a study, scientists intentionally mixed up signals in order to see how people's abilities to read facial expressions are affected by gestures. Though participants were instructed to focus only on the facial expressions of the figures in trying to decipher their emotions, it became clear that gestures were not being isolated and ignored. When shown static images of incongruous matches, for example, an angry face paired with a frightened body, participants' abilities to correctly identify the emotions signaled by facial expressions dropped from 81% to 64%.
In addition, an electroencephalogram revealed unusually large voltage spikes in a key brain area for face recognition when participants encountered these mismatched emotional cues. Those spikes show that our brains are surprised by these mismatches, and they send red flags.
What's that mean? Our processing of gestures is bound up with our processing of facial expressions. In fact, it may even be the case that as far as processing goes, our brains don't distinguish between the two kinds of visual cues.