Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition

Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, Wei Zhang
The University of Sydney, SenseTime Research, The Chinese University of Hong Kong

Abstract

• Novel compact motion representation method, named Optical Flow guided Feature (OFF)
• OFF can be embedded in any framework.

1. Introduction

• Temporal information is the key.
• Optical flow is useful motion representation, but inefficient.
• 3D CNN does not perform as well as Two-stream networks with optical flow.
• OFF is a new feature representation from orthogonal space of optical flow on feature level.
  
  • Spatial gradients of feature maps in horizontal, vertical directions
  • Temporal gradients

2. Related Work

• Hand-crafted features
• Deep-features
  
  • Optical flow
  • 3D CNN
  • RNN
• OFF
  
  • Well captures the motion patterns
  • Complementary to other motion representations

3. Optical Flow Guided Feature : OFF

• Optical Flow
  
  • 
    
    • : pixel at the location of a frame t
    • : spatial pixel displacement in each axes
• Apply at feature level
  
  • 
    
    • : mapping function for extracting features from image
    • : parameters in
• According to definition of optical flow
  
  • 
  • 
    
    • 
    • : feature level optical flow
• OFF :
  
  • Orthogonal to feature level optical flow and changes as it changes.
  • Encodes spatial-temporal information orthogonally and complementarily to

4. Using Optical Flow Guided Feature in CNN

4.1. Network Architecture

Feature Generation Sub-network

• BN-Inception for extracting feature map

OFF Sub-network

• 1x1 convolutional layer
• Apply Sobel operator for spatial gradients
  
  • 
  • 
• Element-wise subtraction for temporal gradients
  
  • 
• Concatenate features from lower level.

Classification Sub-network

• Multiple inner-product classifiers for each features
• Classification scores are averaged

4.2. Network Training

• th segment on level :
• Classification score of :
• 
  
  • is average pooling for summarizing scores
• Cross-entropy loss for each level
  
  • 
    
    • : number of categories
    • : ground-truth class label
• Two-stage training
  
  • Train feature generation sub-network first.
  • Train classification sub-network with feature network frozen.

4.3. Network Testing

• Test under TSN framework
• 25 segments are sampled from RGB
• th segment is treated as Frame

5. Experiments and Evaluations

5.1. Datasets and Implementation Details

• UCF-101 / HMDB-51 datasets
• 4 NVIDIA TITAN X GPUs
• Caffe & OpenMPI
• Train feature generation network by TSN method
• Train OFF sub-networks from scratch with feature generation networks frozen.

5.2. Experimental Investigations of OFF

• Efficiency
  
  • State-of-the art among real-time methods

• Effectiveness
  
  • Investigate the roustness of OFF when applying different inputs.

• Comparison
  
  • 2.0%/5.7% gain compared with the baseline Two-Stream TSN

6. Conclusion

• OFF is fast(200fps) and robust.
• The result with only RGB input is comparable to Two-stream approaches.
• Complementary to other motion representations.

PoTion : Pose MoTion Representation for Action Recognition

Vasileios Choutas, Philippe Weinzaepfel, Jérôme Revaud, Cordelia Schmid
Inria, NAVER LABS Europe

Abstract

• State-of-the-art methods for action recognition rely on two-stream networks
• Try to process appearance and motion jointly
• Fixed-sized representation that encodes pose over clip

1. Introduction

• Human pose can be added to multi-stream architecture
• 3D skeleton
  
  • Limited to case where the data is available
• 2D poses
  
  • Used when fully-visible
  • Features(hand-crafted, CNN) around the human joints
• Zolfaghari et al.
  
  • Pose stream for semantic segmentation maps
  • FC + spatio-temporal CNN
• Propose to focus on the movement of a few relevant keypoints
  
  • Fixed-sized representation, does not depend on the duration of the clip.
• Overview of PoTion
  
  • Obtain heat maps for human joint by running pose estimator (Part Affinity Fields).
  • Colorize heat maps depending on the relevant time.
  • Obtain PoTion representation for clip.
  • Train shallow CNN architecture for action classification.
• Contributions
  
  • Clip-level representation that encodes human pose motion, PoTion
  • Study PoTion representation and CNN for action recognition
  • Combin PoTion with two-stream architecture

2. Related Work

• CNNs for action recognition
  
  • 2D + RNN
  • 3D CNN
  • Two-stream
• Motion representation
  
  • Optical flow, etc.
• Pose representation
  
  • 3D skeleton
  • 2D pose
  • Approach of Zolfaghari et al.

3. PoTion representation

3.1. Extracting joint heatmaps

• Obtain human joint heatmaps by Part Affinity Fields
• Part Affinity Fields
  
  • Robust to occlusion and truncation
  • Output : joint heatmap & affinity map
  • Use only joint heatmap
• Is the likelihood of pixel

3.2. Time-dependent heat map colorization

• ‘Colorize’ according to relative time of this frame to
• For C = 2
  
  • 
• For multiple channels C
  
  • Split T frames into C-1 regularly sampled intervals
• 

3.3. Aggregation of colorized heatmaps

• Compute sum of the colorized heat maps
  
  • 
• Obtain invariant representation by normalizing channel independently.
  
  • 
• Compute intensity image
  
  • 
  • Encodes how much time a joint stays at each location
• Normalized PoTion representation : divide by intensity
  
  • 
  • All locations of the motion trajectory are weighted eqally

4. CNN on PoTion representation

• Network architecture

  • 3 blocks with 2 convolutional layers in each block
  • 3 x 3 kernel , stride 2 and then stride 1
  • Global average pooling + FC + softmax after all blocks
  • Batch normalization, ReLU after each Conv.

• Implementation details

  • Xavier initialization and train from scratch
  • Dropout of 0.25
  • Adam optimizer
  • Batch size of 32
  • 4 hours on single GPU (Titan X)
  • Data augmentation
    
    • Flipping by swapping channels was efficient
    • Random cropping, smoothing heat maps, shifting did not gain

5. Experimental results

5.1. Datasets and metrics

• HMDB
• JHMDB
• UCF101
• Kinetics
• Report mean classification accuracy.

5.2 PoTion representation

• Number of channels
• Aggregation techniques

5.3. CNN on PoTion

• Data augmentation
• Network Architecture
  
  • Number of convolution layers per block : 2
  • Number of blocks : 3
  • Number of convolution filters : (128, 256, 512)

5.4. Imapct of pose estimation

• Groundtruth pose : 4% gain
• Crop frames centered on the actor (GT-JHMDB) : 6% gain

5.5. Comparison to the state of the art

• Multi-stream approach
  
  • Up to +8% on TSN / Up to 3% on I3D
• Comparison to Zolfahari et al.
  
  • Improvement due to improved pose motion representation
• Comparison to the state of the art
  
  • Outperform all existing approaches
• Detailed analysis
  
  • Clear improvement when well defined motion
  • Low performance when object is more important than motion
• Results on Kinetics
  
  • Accuracy decreased by 1~2%
  • Reasons of decrease
    
    • Actors are partially visible
    • Feature erratic camera
    • Multiple shots per clip

6. Conclusion

• PoTion encodes the motion over entire clip into fixed-size representation.
• Classification with PoTion representation and shallow CNN.
• Leads to state-of-the-art performance on JHMDB, HMDB, and UCF-101.

Our discussion

• Why not use 3D convolution for capturing temporal features?