SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

by   Xavier Alameda-Pineda, et al.

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at


page 1

page 4

page 6

page 8

page 9

page 14


Face-to-Face Co-Located Human-Human Social Interaction Analysis using Nonverbal Cues: A Survey

This work presents a systematic review of recent efforts (since 2010) ai...

Joint learning of Social Groups, Individuals Action and Sub-group Activities in Videos

The state-of-the art solutions for human activity understanding from a v...

F-formation Detection: Individuating Free-standing Conversational Groups in Images

Detection of groups of interacting people is a very interesting and usef...

Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation

Body language is an eye-catching social signal and its automatic analysi...

Social Cue Analysis using Transfer Entropy

Robots that work close to humans need to understand and use social cues ...

Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback

Behavioral cues play a significant part in human communication and cogni...

Detecting Gaze Towards Eyes in Natural Social Interactions and its Use in Child Assessment

Eye contact is a crucial element of non-verbal communication that signif...

Please sign up or login with your details

Forgot password? Click here to reset