Abstract:Multi-person multi-view 3D human pose estimation aims to predict the positions of multiple human keypoints from multi-view images and is considered a fundamental problem in computer vision. The lack of depth information and the high computational cost of 3D processing make pose estimation from RGB images complex and prone to inaccuracies. To address these challenges, various effective solutions have been proposed. Among them, voxel-based methods utilize camera parameters to extract 3D voxel features from multi-view images. However, the discrete nature of voxel representation inevitably introduces quantization errors. To mitigate this limitation, a novel method named PoseNeRF is proposed, which integrates neural radiance fields (NeRF) into multi-person multi-view 3D human pose estimation through an end-to-end differentiable dual-branch joint training framework. The proposed method consists of three main components: the NeRF branch, the pose branch, and a shared parameter mechanism. Specifically, the NeRF branch is trained using enhanced multi-view image features, allowing the embedded geometric multilayer perceptrons (G-MLP) to represent the opacity of specific spatial locations. The pose branch predicts the central positions and keypoints of humans from 3D voxel features. The shared parameter mechanism leverages the opacity information provided by the G-MLP in the NeRF branch to refine the voxel feature representation. The effectiveness of the proposed method is validated through extensive experiments on the CMU Panoptic, Campus, and Shelf datasets. On the CMU Panoptic dataset, the proposed method achieves notable improvements over VoxelPose and Faster VoxelPose in the AP25 metric, with gains of 2.1% and 6.0%, respectively. In terms of MPJPE, the proposed method reduces the error by 1.4 mm compared to Faster VoxelPose. Consistent performance gains are also observed on the Campus and Shelf datasets compared to VoxelPose.