Design of 3D Hand Mesh Reconstruction from Monocular Image

Abstract:

In recent years, as the impact of deep learning on people's lives has grown, more and more attention has been paid to the development of this field. The task of human hand pose and shape estimation from an RGB image has been a long-standing problem in the field of computer vision. Unlike common hand posture prediction, which only predicts the coordinates of the skeletal points of the hand, this task restores the original shape of the hand. Many places will apply this task such as augmented reality and virtual reality, but the task is still very challenging because the hand occupies a relatively small part of the image area, and the hand movements are flexible and easy to block. In this paper, we propose a complete end-to-end network architecture to obtain 3D mesh hand shapes from RGB hand images. In the encoder part, we use ResNet-50 to extract the image features. For better regression of model parameters later, we obtain some 2D feature maps, such as 2D heatmap and mask images through some convolutional layers. In the model parameter regression part, we use the fully connected layer for iterative regression of the model parameters. We add the hand mesh coordinate correction part to solve the less natural defects. We treat the hand model (MANO) generated by the model as the rough initial hand model, then enter the graphical convolutional network layer to regress the offset of each coordinate point. Finally, we add it to the initial hand model to get the final hand model. The experimental result shows that our model can 3D hand mesh reconstruction well from only the monocular Image.

Artitechture:

We propose a neural network that can accurately and efficiently predict 3D hand meshes from monocular RGB images. Our model consists of three main parts. The first Extract features part is to extract intermediate representations of the hand to facilitate the next parameter regression, playing a simplifying and guiding role. The second part, the MANO part, regresses the parameters of the parametric hand model MANO to produce a rough hand model first. The final part is the Adjust part, which generates the offset of the vertex coordinates on the hand mesh to correct our previous crude hand mesh and finally obtain the final hand mesh. In the future, we believe 3D hand mesh prediction will continue to be a very important field, and 3D hand mesh prediction will be indispensable for future games, VR and AR. Finally, we also hope that we can connect with applications and form a better system.

Fig 1¡GOur proposed framework

Made by ¦¿¹Å´