Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions

by   Swathikiran Sudhakaran, et al.

In this paper, we present a novel deep learning based approach for addressing the problem of interaction recognition from a first person perspective. The proposed approach uses a pair of convolutional neural networks, whose parameters are shared, for extracting frame level features from successive frames of the video. The frame level features are then aggregated using a convolutional long short-term memory. The hidden state of the convolutional long short-term memory, after all the input video frames are processed, is used for classification in to the respective categories. The two branches of the convolutional neural network perform feature encoding on a short time interval whereas the convolutional long short term memory encodes the changes on a longer temporal duration. In our network the spatio-temporal structure of the input is preserved till the very final processing stage. Experimental results show that our method outperforms the state of the art on most recent first person interactions datasets that involve complex ego-motion. In particular, on UTKinect-FirstPerson it competes with methods that use depth image and skeletal joints information along with RGB images, while it surpasses all previous methods that use only RGB images by more than 20


page 1

page 2

page 3

page 4

page 5

page 6


Learning to Detect Violent Videos using Convolutional Long Short-Term Memory

Developing a technique for the automatic analysis of surveillance videos...

Lip Reading Using Convolutional Auto Encoders as Feature Extractor

Visual recognition of speech using the lip movement is called Lip-readin...

Multimodal Video-based Apparent Personality Recognition Using Long Short-Term Memory and Convolutional Neural Networks

Personality computing and affective computing, where the recognition of ...

Rapid Autonomous Car Control based on Spatial and Temporal Visual Cues

We present a novel approach to modern car control utilizing a combinatio...

AdaFrame: Adaptive Frame Selection for Fast Video Recognition

We present AdaFrame, a framework that adaptively selects relevant frames...

Relational Long Short-Term Memory for Video Action Recognition

Spatial and temporal relationships, both short-range and long-range, bet...

SummaryNet: A Multi-Stage Deep Learning Model for Automatic Video Summarisation

Video summarisation can be posed as the task of extracting important par...

Please sign up or login with your details

Forgot password? Click here to reset