'Social Master' GPT-4! Know how to interpret expressions and speculate on psychology

Imagine you are at a vibrant cocktail party filled with lively conversations and the clink of glasses.

At this time, you are a leisurely observer, hiding in the corner happily. Yet even without being at the center of a party, you can easily figure out the social relationships between different people, understand what's going on, and even decipher overt and covert social messages by reading people's verbal and nonverbal cues.

What if an LLM could reproduce this level of social skills? No, that’s what Koko Mind is.

Just open a video, and the model will start to analyze the character's expression and draw conclusions about the character's emotion.

Then, you can also ask questions in the prompt column on the right to let AI further analyze the undercurrent of social puzzles in the video.

(To be honest, this is difficult for some people)

Picture

Koko Mind contains 150 complex multi-party social interactions and free text questions and answers.

To ensure data diversity and scalability and avoid data contamination, all social interactions, questions and answers are generated by GPT-4 and subsequently verified by human experts.

The analysis data is based on three different sources:

• GPT-4-only: This subset is only composed of GPT-4 Created via prompts.
  
• Based on movies: To avoid data contamination, this part of the data is based on various scenes extracted from movies released after 2022. GPT-4 was responsible for shaping these scenes, adding its own elements while retaining the core essence.
  
• Based on ToMi: This section contains data supported by the simulated dataset ToMi, which involves moving physical objects to different places, which is psychological A classic test of a theory. Of course, these social interactions must be modified and expanded by GPT-4.

The proportions of the three data sources are as follows:

Pictures

For each social interaction, researchers will ask various questions to explore the following aspects closely related to social understanding.

• # Theory of Mind: Questions that assess understanding of other people's mental states and perspectives.
  
• Social Norms: Questions designed to identify social values ​​and norms in a situation.
  
• Emotion Recognition: Problems aimed at identifying and understanding emotional elements in context.
  
• Social Relationships: Focus on interpersonal dynamics and relationships.
  
• Counterfactual questions: Hypothetical queries designed to explore alternative outcomes or possibilities.
  
• Social Advice: A question that proposes advice or suggested action relevant to a specific situation.

The researchers used text-davinci-003 as a reference to evaluate different models after AlpacaEval.

In which the researchers removed the nonverbal cues in the brackets (e.g., nervously drinking coffee, etc.) from the context.

The following are some interesting points:

• Among the two models, compared to Claude, GPT-4 Demonstrate greater certainty and confidence in identifying winning models.
  
• Claude outperforms GPT-4 when the context has no non-verbal cues and the interaction is either entirely generated by GPT-4 or based on movies 4.
  
• And if the context contains non-verbal clues, GPT-4 is always better than Claude.

(One possible explanation is that GPT-4 is a multi-modal model that can better understand additional non-verbal information.)

In the blog, the researchers drew tables to clearly see the performance of each model.

Picture

The results, while exciting in many ways, also have certain limitations. First, Koko Mind is relatively small, which may limit the broad applicability and comprehensiveness of the researchers' conclusions.

Secondly, all interactions in Koko Mind are generated by GPT-4 and require manual verification, which makes the dataset difficult to expand.

Also, although Koko Mind provided human-verified answers in the dataset, the researchers did not use these answers as a reference when evaluating, and since these answers were generated by GPT-4 , so they may be biased towards GPT-4.

Future research could focus on how to evaluate models on human-validated machine-generated reference answers.

Of course, despite the limitations of one kind or another, researchers still regard Koko Mind as a springboard for future research related to social intelligence, multi-modal language models, etc.

The above is the detailed content of 'Social Master' GPT-4! Know how to interpret expressions and speculate on psychology. For more information, please follow other related articles on the PHP Chinese website!