The study, published 18 May in IEEE Robotics and Automation Letters, trained a vision language model to recognize human emotions during robot handover tasks. Unlike conventional facial analysis systems that focus narrowly on expressions, the VLM processed full-scene context — body language, hand gestures, environmental cues. It achieved a semantic similarity score of 0.86 compared to human observers' emotion labels, beating the conventional AI's 0.77.
In a second experiment with 40 volunteers, the team programmed a robot to make deliberate errors during collaboration tasks. When the robot apologized, 31 out of 40 participants preferred emotionally adaptive apologies over pre-scripted ones. But post-task surveys revealed that emotional adaptivity barely moved the needle on trust. Functional failure — the robot dropping an object, missing a handoff — degraded trust scores sharply regardless of apology quality.
The VLM's emotion predictions aligned well with third-party human observers watching interactions. But when measured against participants' self-reported internal feelings, accuracy dropped significantly. Lead researcher Seung Chan Hong summarized the gap: "While the VLM is a good observer of outward social cues, it isn't a mind reader."
The results underscore a hard constraint for human-robot collaboration: emotional intelligence is valued, but competence is non-negotiable. As Hong put it, "A personalized apology acts as a social lubricant, but it cannot repair the trust lost by the robot failing its physical task."



