Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation

SIGGRAPH Asia 2026

TL;DR: Given text, we generate whole-body human motion interacting with multiple rigid or articulated objects, by representing objects as tracked surface keypoints and modeling human–object contact with a spatio-temporal distance field.

Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation

Xiaogang Peng1, Zeyu Han1, Zichong Meng1, Yiming Xie1, Jihua Zhu2, Gang Hua3, Huaizu Jiang1
1. Northeastern University    2. Xi'an Jiaotong University    3. Amazon
Paper Code Video

TL;DR: Given text, we generate whole-body human motion interacting with multiple rigid or articulated objects, by representing objects as tracked surface keypoints and modeling human–object contact with a spatio-temporal distance field.

Teaser
Figure 1: Our approach enables: (a) whole-body interaction of a variable number of rigid objects from text prompts; (b) interaction with articulated objects exhibiting diverse joint dynamics (e.g., laptop, drawers, and knobs); and (c) text-driven whole-body motion synthesis guided by sparse object waypoints or full object motion trajectories.

Abstract

Daily activities require humans to coordinate whole-body motion with the motion of surrounding objects. Despite recent progress in human-object interaction (HOI) generation, most existing methods assume interactions with a single rigid object and do not extend well to scenarios involving a variable number of objects or articulated objects with diverse joint mechanisms. We propose surface keypoint trajectories as an object motion representation: for each rigid component, whether a standalone object or one part of an articulated assembly, we track a small set of non-collinear surface points over time. This representation handles multi-object coordination and diverse articulation mechanisms directly from point dynamics without requiring explicit joint-type specification. To model when and where each body region contacts each object, we introduce a spatio-temporal contact distance field that extends distance-based contact modeling to whole-body, multi-object, and articulated settings. We factorize HOI generation into three stages: generating object motions from text or waypoints, predicting the contact distance field, and synthesizing whole-body motion with contact-guided optimization. Experiments on ParaHome, HIMO, ARCTIC, and OMOMO demonstrate better or comparable performance to existing methods across single-object, multi-object, and articulated interaction settings.

Methodology

Method overview diagram

Figure 2: Framework Overview. Our pipeline consists of three stages. Stage 1 (Object Motion Generation) employs an object-motion diffusion model fo to generate object surface keypoint trajectories conditioned on the initial HOI state, the text prompt, and object geometry encoded with Basis Point Sets (BPS). Stage 2 (Contact Distance Field Prediction) uses a diffusion model fc to predict a human–object contact distance field based on the generated object motion, the text prompt, and the object geometry. Stage 3 (Body Motion Synthesis) applies a marker-motion diffusion model fh to synthesize the final human motion conditioned on the text prompt, the generated object motion, and the predicted contact distance field. We further perform contact optimization to alleviate contact and penetration artifacts, and finally recover the human and object meshes.

Results Gallery

Qualitative results across datasets. Filter by dataset and swipe / drag to explore more.

Comparison with Baselines

Qualitative comparison against baseline methods on four benchmarks. Each benchmark is its own swipeable window — drag to see more examples.

Contact Prediction Visualization

Visualization of the predicted human–object contact distance field. Line color encodes distance: red marks points in direct contact, pink marks points that are close but not touching, and the absence of a line indicates the points are far apart.

Citation

@inproceedings{doe2025deepmotion,
    title={Deep Learning for Human Motion Prediction},
    author={Doe, Jane and Smith, John and Johnson, Alice},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2025}
    }