Back-Hand-Pose: 3D Hand Pose Estimation for a Wrist-worn Camera via Dorsum Deformation Network
summary
A wrist-mounted fisheye RGB prototype uses a two-stream ResNet18 + LSTM Kalman pipeline called DorsalNet that regresses a 19D finger representation from RGB plus motion history images of dorsal deformation, then recovers 3D pose via inverse kinematics. Evaluated on 225,000 frames from 5 users it runs in real time, achieves 8.81° mean joint-angle error for user-specific models (9.77° general), boosts dynamic-gesture recognition ~20%, and reaches 75% accuracy on 11 grasp types while struggling most with thumb and pinky and forearm roll.
AI-written summary · about
chapters (14)
- 0:00Introduction and motivation
- 1:32Dorsal hand deformation insight
- 3:00System overview and hardware
- 4:05Preprocessing and motion features
- 4:51Dorsal Net architecture
- 6:01Hand representation details
- 7:38Evaluation and data collection
- 8:26Accuracy and baseline comparison
- 9:16Lighting robustness tests
- 10:02Ablation study results
- 10:50Grasp recognition performance
- 11:13Applications and demos
- 12:34Summary and future work
- 13:27Limitations and improvements
related talks
07mFingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by ...
10mProactive Sensing for Improving Hand Pose Estimation
19mBackHand: Sensing Hand Gestures via Back of the Hand
10mDetecting 3D Hand Pointing Direction from RGB-D Data in Wide-Ranging HRI Scenarios
05mTouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images
06mWR-Hand: Wearable Armband can Track User's Hand
20mToward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines
08mSo Predictable! Continuous 3D Hand Trajectory Prediction in Virtual Reality