Articulated body pose estimation

Perception of human beings in their neighboring environment is an important capability that robots must possess. If a person uses gestures to point to a particular object, then the interacting machine should be able to understand the situation in real world context. Thus pose estimation is an important and challenging problem in computer vision, and many algorithms have been deployed in solving this problem over the last two decades. Many solutions involve training complex models with large data sets.

Pose estimation is a difficult problem and an active subject of research because the human body has 244 degrees of freedom with 230 joints. Although not all movements between joints are evident, the human body is composed of 10 large parts with 20 degrees of freedom. Algorithms must account for large variability introduced by differences in appearance due to clothing, body shape, size, and hairstyles. Additionally, the results may be ambiguous due to partial occlusions from self-articulation, such as a person's hand covering their face, or occlusions from external objects. Finally, most algorithms estimate pose from monocular (two-dimensional) images, taken from a normal camera. Other issues include varying lighting and camera configurations. The difficulties are compounded if there are additional performance requirements. These images lack the three-dimensional information of an actual body pose, leading to further ambiguities. There is recent work in this area wherein images from RGBD cameras provide information about color and depth.[3]

There is a need to develop accurate, tether-less, vision-based articulated body pose estimation systems to recover the pose of bodies, such as the human body, a hand, or non-human creatures. Such a system has several foreseeable applications, including the following:

• Markerless motion capture for human-computer interfaces,
• Physiotherapy,
• Human image synthesis,
• Ergonomics studies,
• Robot control, and
• Visual surveillance.

The typical articulated body pose estimation system involves a model-based approach, in which the pose estimation is achieved by maximizing/minimizing a similarity/dissimilarity between an observation (input) and a template model. Different kinds of sensors have been explored for use in making the observation, including the following:

• Visible wavelength imagery,
• Long-wave thermal infrared imagery,[4]
• Time-of-flight imagery, and
• Laser range scanner imagery.

These sensors produce intermediate representations that are directly used by the model. The representations include the following:

• Image appearance,
• Voxel (volume element) reconstruction,
• 3D point clouds, and sum of Gaussian kernels[5]
• 3D surface meshes.

The basic idea of part based model can be attributed to the human skeleton. Any object having the property of articulation can be broken down into smaller parts wherein each part can take different orientations, resulting in different articulations of the same object. Different scales and orientations of the main object can be articulated to scales and orientations of the corresponding parts. To formulate the model so that it can be represented in mathematical terms, the parts are connected to each other using springs. As such, the model is also known as a spring model. The degree of closeness between each part is accounted for by the compression and expansion of the springs. There is geometric constraint on the orientation of springs. For example, limbs of legs cannot move 360 degrees. Hence parts cannot have that extreme orientation. This reduces the possible permutations.[6]

The spring model forms a graph G(V,E) where V (nodes) corresponds to the parts and E (edges) represents springs connecting two neighboring parts. Each location in the image can be reached by the ${\displaystyle x}$ and ${\displaystyle y}$ coordinates of the pixel location. Let ${\displaystyle \mathbf {p} _{i}(x,\,y)}$ be point at ${\displaystyle \mathbf {i} ^{th}}$ location. Then the cost associated in joining the spring between ${\displaystyle \mathbf {i} ^{th}}$ and the ${\displaystyle \mathbf {j} ^{th}}$ point can be given by ${\displaystyle S(\mathbf {p} _{i},\,\mathbf {p} _{j})=S(\mathbf {p} _{i}-\mathbf {p} _{j})}$. Hence the total cost associated in placing ${\displaystyle l}$ components at locations ${\displaystyle \mathbf {P} _{l}}$ is given by

${\displaystyle S(\mathbf {P} _{l})=\displaystyle \sum _{i=1}^{l}\;\displaystyle \sum _{j=1}^{i}\;\mathbf {s} _{ij}(\mathbf {p} _{i},\,\mathbf {p} _{j})}$

The above equation simply represents the spring model used to describe body pose. To estimate pose from images, cost or energy function must be minimized. This energy function consists of two terms. The first is related to how each component matches the image data and the second deals with how much the oriented (deformed) parts match, thus accounting for articulation along with object detection.[7]

The part models, also known as pictorial structures, are of one the basic models on which other efficient models are built by slight modification. One such example is the flexible mixture model which reduces the database of hundreds or thousands of deformed parts by exploiting the notion of local rigidity.[8]

The kinematic skeleton is constructed by a tree-structured chain, as illustrated in the Figure.[9] Each rigid body segment has its local coordinate system that can be transformed to the world coordinate system via a 4×4 transformation matrix ${\displaystyle T_{l}}$,

${\displaystyle T_{l}=T_{\operatorname {par} (l)}R_{l},}$

where ${\displaystyle R_{l}}$ denotes the local transformation from body segment ${\displaystyle S_{l}}$ to its parent ${\displaystyle \operatorname {par} (S_{l})}$. Each joint in the body has 3 degrees of freedom (DoF) rotation. Given a transformation matrix ${\displaystyle T_{l}}$ , the joint position at the T-pose can be transferred to its corresponding position in the world coordination. In many works, the 3D joint rotation is expressed as a normalized quaternion ${\displaystyle [x,y,z,w]}$ due to its continuity that can facilitate gradient-based optimization in the parameter estimation.

A commercially successful but specialized computer vision-based articulated body pose estimation technique is optical motion capture. This approach involves placing markers on the individual at strategic locations to capture the 6 degrees-of-freedom of each body part.

A number of groups and companies are researching pose estimation, including groups at Brown University, Carnegie Mellon University, MPI Saarbruecken, Stanford University, the University of California, San Diego, the University of Toronto, the École Centrale Paris, ETH Zurich, National University of Sciences and Technology (NUST),[14] and the University of California, Irvine.

At present, several companies are working on articulated body pose estimation.

• Bodylabs: Bodylabs is a Manhattan-based software provider of human-aware artificial intelligence.

• Michael J. Black, Professor at Brown University
• Research Project Page of German Cheung at Carnegie Mellon University
• Homepage of Dr.-Ing at MPI Saarbruecken
• Markerless Motion Capture Project at Stanford
• Computer Vision and Robotics Research Laboratory at the University of California, San Diego
• Research Projects of David J. Fleet at the University of Toronto
• Ronald Poppe at the University of Twente.
• Professor Nikos Paragios at the Ecole Centrale de Paris
• Articulated Pose Estimation with Flexible Mixtures of Parts Project at UC Irvine
• http://screenrant.com/crazy3dtechnologyjamescameronavatarkofi3367/
• 2D articulated human pose estimation software
• Articulated Pose Estimation with Flexible Mixtures of Parts