Estimating 3D Motion and Forces of Human–Object Interactions from Internet Videos

Abstract

In this paper, we introduce a method to automatically reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person together with the object pose, the contact positions and the contact forces exerted on the human body. The main contributions of this work are three-fold. First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of the interactions. This is cast as a large-scale trajectory optimization problem. Second, we develop a method to automatically recognize from the input video the 2D position and timing of contacts between the person and the object or the ground, thereby significantly simplifying the complexity of the optimization. Third, we validate our approach on a recent video + MoCap dataset capturing typical parkour actions, and demonstrate its performance on a new dataset of Internet videos showing people manipulating a variety of tools in unconstrained environments.

Appendices

Outline of the Appendix

In this appendix, we provide additional technical details of the proposed method. In Appendix A, we provide a comprehensive description of the parametric human and object model we use for the trajectory optimization. Then, in Appendix B we give details of the ground contact force generators mentioned in the main paper (Sect. 4.3).

Parametric Human and Object Models

Human model We model the human body as a multi-body system consisting of a set of rotating joints and rigid links connecting them. We adopt the joint definition of the SMPL model (Loper et al. 2015) and approximate the human skeleton as a kinematic tree with 24 joints: one free-floating joint and 23 spherical joints. Figure 12 illustrates our human model in a canonical pose. A free-floating joint consists of a 3-dof translation in ${\mathbb {R}}^3$ and a 3-dof rotation in SO(3); we model the pelvis by a free-floating joint to describe the person’s body orientation and translation in the world coordinate frame. A spherical joint is a 3-dof rotation; it represents the relative rotation between two connected links in our model. In practice, we use unit quaternions to represent 3D rotations and axis-angles to describe angular velocities. As a result, the configuration vector of our human model $q^\mathrm {h}$ is a concatenation of the configuration vectors of the 23 spherical joints (dimension 4) and the free-floating pelvis joint (dimension 7), hence of dimension 99. The corresponding human joint velocity ${\dot{q}}^\mathrm {h}$ is of dimension $23\times 3+6=75$ (by replacing the quaternions with axis-angles). For simplicity, in the main paper we do not distinguish this difference in dimension and consider both $q^\mathrm {h}$ and ${\dot{q}}^\mathrm {h}$ to be represented using axis-angles, hence of the same dimension $n_q^\mathrm {h}=75$. In addition, based on these 24 joints, we define 18 “virtual markers” (shown as colored spheres in Fig. 12) that represent the 18 OpenPose joints. These markers are used instead of the 24 joints to compute the re-projection errors with respect to the OpenPose 2D detections.

Object models All four objects, namely barbell, hammer, scythe and spade, are modeled as a non-deformable rigid line stick. The configuration $q^\mathrm {o}$ represents the 6-dof displacement of the stick handle, as illustrated in Fig. 13. In practice, $q^\mathrm {o}$ is a 7-dimensional vector containing the 3D translation and 4D quaternion rotation of the free-floating handle end. The object joint velocity ${\dot{q}}^\mathrm {o}$ is of dimension 6 (by replacing the quaternion with an axis-angle). The handtools that we are modelling have the stick handle as the contact area. We ignore the handle’s thickness and represent the contact area using the line segment between the two endpoints of the handle. Depending on the number of human joints in contact with the object, we associate the same number of contact points to the object’s local coordinate frame. These contact points can be located at any point along the feasible contact area. In practice, all object contact points together with the endpoint corresponding to the head of the handtool are implemented as “virtual” prismatic joints of dimension 1.

Fig. 12

Our human model in the reference posture. The skeleton consists of one free-floating basis joint corresponding to pelvis, and 23 spherical joints. The colored spheres are 18 virtual markers that correspond to 18 OpenPose joints. Each marker is associated to a semantic joint in our model

Fig. 13

All four handtools are represented by a single object model shown in this image. The object model consists of 1 free-floating basis joint corresponding to the handle end point (red sphere), 1 prismatic joint corresponding to the head of the tool (green sphere), and several prismatic joints corresponding to the location of the contact points (grey translucent spheres in the middle). The contact points should lie on the feasible contact area (grey stick) formed by the two endpoints

Generators of the Ground Contact Forces

In this section, we describe the generators $g^{(3)}_n$ and $g^{(6)}_{kn}$ for computing the contact forces exerted by the ground on the person. Recall from the main paper that we consider different contact models depending on the type of the joint. We model the planar contacts between the human sole and the ground by fitting the point contact model [given by Eq. (9) in the main paper] at each of the four sole vertices. For other types of ground contacts, e.g. the knee-ground contact, we apply the point contact model directly at the human joint. We model the ground as a 2D plane $G = \{p\in {\mathbb {R}}^3|a^Tp=b\}$ with a normal vector $a\in {\mathbb {R}}^3$, $a\ne 0$, $b\in {\mathbb {R}}$ and a friction coefficient $\mu$. In the following, we first provide the expression of the 3D generators $g^{(3)}_n$ for modeling point contact forces and then derive the 6D generators $g^{(6)}_{kn}$ for modeling planar contact forces.

3D generators $g^{(3)}_n$ for point contact forces Let $p_k$ be the position of a contact point k located on the ground surface, i.e. $a^Tp_k=b$. We define at contact point k a right-hand coordinate frame C whose xz-plane overlaps the plane G and whose y-axis points towards the gravity direction, i.e., the opposite direction to the ground normal a. During point contact, it is a common assumption that the ground exerts only linear reaction forces on the contact point c. In other words, the spatial contact force expressed in the local frame C can be expressed as

$$\begin{aligned} ^C\phi = \begin{pmatrix} f \\ {\mathbf {0}}_{3\times 1} \end{pmatrix}, \end{aligned}$$

(17)

where the linear component f must lie in the second-order cone ${\mathcal {K}}^3 = \{f=(f_x,f_y,f_z)^T|\sqrt{f_x^2 + f_z^2} \le -f_y \tan \mu \}$, which can be approximated by the pyramid ${{\mathcal {K}}^3}^\prime = \{f=\sum _{n=1}^4{\lambda _n g^{(3)}_n}|\lambda _n\ge 0\}$, with a set of 3D-generators

$$\begin{aligned} g^{(3)}_1&= \left( \sin {\mu }, -\cos {\mu }, 0\right) ^T, \end{aligned}$$

(18)

$$\begin{aligned} g^{(3)}_2&= \left( -\sin {\mu }, -\cos {\mu }, 0\right) ^T, \end{aligned}$$

(19)

$$\begin{aligned} g^{(3)}_3&= \left( 0, -\cos {\mu }, \sin {\mu }\right) ^T, \end{aligned}$$

(20)

$$\begin{aligned} g^{(3)}_4&= \left( 0, -\cos {\mu }, -\sin {\mu }\right) ^T, \end{aligned}$$

(21)

where $\mu$ is the friction coefficient. More formally, we are approximating the friction cone ${\mathcal {K}}^3$ with the conic hull ${{\mathcal {K}}^3}^\prime$ spanned by 4 points on the boundary of ${\mathcal {K}}^3$, namely, $g^{(3)}_n$ with $n=1,2,3,4$.

6D generators $g^{(6)}_{kn}$ for planar (sole) contact forces Here we show how to obtain the 6D generator $g^{(6)}_{kn}$ from $g^{(3)}_{n}$ and the contact point position $p_k$. As described in the main paper, we approximate human sole as a rectangle area with 4 contact points. We assume that the sole overlaps the ground plane G during contact. Similar to the point contact, we define 5 parallel coordinate frames $C_k$, one at each of the four sole contact points, plus a frame A at the ankle joint. Note that the frames $C_k$ and A are parallel to each other, i.e., there is no rotation but only translation when passing from one frame to another. We can write the contact force at contact point k as the 6D spatial force

$$\begin{aligned} ^{C_k}\phi _k = \sum _{n=1}^4 \lambda _{kn} \begin{pmatrix} g^{(3)}_n \\ {\mathbf {0}}_{3\times 1} \end{pmatrix} , \text { with } \lambda _{kn} \ge 0. \end{aligned}$$

(22)

We denote by $^Ap_k$ the position of contact point $c_k$ in the ankle frame A, and by $^AX_{C_k}^*$ the matrix converting spatial forces from frame $C_k$ to frame A. We can then express the contact force in frame A:

$$\begin{aligned} ^A\phi&= \sum _{k=1}^4{{^AX_{C_k}^*}^{C_k}\phi _k} \end{aligned}$$

(23)

$$\begin{aligned}&= \sum _{k=1}^4{\begin{pmatrix} I_3 &{} ^Ap_k\times \\ 0_3 &{} I_3\\ \end{pmatrix}^{-T}{^{C_k}\phi _k}} \end{aligned}$$

(24)

$$\begin{aligned}&=\sum _{k=1}^4\sum _{n=1}^4 \lambda _{kn} g^{(6)}_{kn}, \end{aligned}$$

(25)

where

$$\begin{aligned} g^{(6)}_{kn}= \begin{pmatrix} g^{(3)}_n \\ ^Ap_k\times g^{(3)}_n \end{pmatrix}. \end{aligned}$$

(26)