A biomechanical-constrained temporal learning framework for lightweight skeleton-based exercise recognition

Abstract

The recognition of exercises using skeletal pose sequences is a significant fitness technology, rehabilitation monitoring, and sports analytics. Nevertheless, the current practices tend to ignore the basic biomechanical processes of human motion. This paper presents a Biomechanical-Aware Temporal Learning (BATL), framework that incorporates human kinematic constraints and deep temporal model. The method takes into account three biomechanical limitations objects consisting of joint angle consistency, velocity smoothness, and bone length stability that are not explicitly coded but rather acquired during the learning course of model parameters. Moreover, BATL uses the phase discovery module which is unsupervised and automatically identifies important temporal divisions (preparation, execution and recovery) without manual annotation. BATL was tested on 146 video samples (157 different samples in total, including 146 squats, 126 push-ups, 146 bicep curls, 146 shoulder-presses) with an estimated test accuracy of 93.33% ± 0.94% percent (mean ± standard deviation on 5-fold cross-validation with random seeds 42, 123, 456, 789, 101). It has 3.57M parameters, requires 13.6 MB memory, and inference can be performed in BatL-only mode only in 5.2 ms per 30-frames sequence (end-to-end pipeline depending on OpenPose pose estimate in 1.574 seconds). It outperformed the baseline techniques by 6.22%-14.44%, and ablation experiment showed that incorporation of biomechanical constraints actually improved accurately by 7.78%. These results imply that even inductive bias involving the use of domain knowledge can be more effective than the benefits that can only be achieved with a more complicated model. Overall, biomechanical knowledge and deep learning offer certain prospects to the activity recognition system, in particular, movement quality and time.