Non-rigid Reconstruction with a Single Moving RGB-D Camera

ICPR 2018

Shafeeq Elanattil1,2
Peyman Moghadam1,2
Sridha Sridharan2
Clinton Fookes2
Mark Cox1

Autonomous Systems Laboratory, CSIRO Data61, Brisbane, Australia1
Queensland University of Technology, Brisbane, Australia2


Reconstruction results. Scene reconstruction results at different time instances using the proposed approach. Our method provides a complete reconstruction of rigid and non-rigid objects in the scene.

We present a novel non-rigid reconstruction method using a moving RGB-D camera. Current approaches use only non-rigid part of the scene and completely ignore the rigid background. Non-rigid parts often lack sufficient geometric and photometric information for tracking large frame-to-frame motion. Our approach uses camera pose estimated from the rigid background for foreground tracking. This enables robust foreground tracking in situations where large frame-to-frame motion occurs. Moreover, we are proposing a multi-scale deformation graph which improves non-rigid tracking without compromising the quality of the reconstruction. We are also contributing a synthetic dataset which is made publically available for evaluating non-rigid reconstruction methods. The dataset provides frame-by-frame ground truth geometry of the scene, the camera trajectory, and masks for background foreground. Experimental results show that our approach is more robust in handling larger frame-to-frame motions and provides better reconstruction compared to state-of-the-art approaches.


Paper

Shafeeq Elanattil, Peyman Moghadam, Sridha Sridharan, Clinton Fookes, Mark Cox

Non-rigid Reconstruction with a Single Moving RGB-D Camera

ICPR 2018.

[pdf] [bibtex]

Video


Overview


Block diagram of the proposed method.

Pipeline of our segmentation method. Input (a) depth and (b) RGB images. (c) Labelled depth image after connected component labelling. (d) Skeleton detected from the RGB image. (e) Final segmented foreground.



The above Figure shows the high-level block diagram of the proposed approach. Our segmentation method separates RGB-D data into foreground and background. The non-rigid and rigid reconstruction modules process foreground and background separately. The camera pose estimated by rigid reconstruction module is used to improve non-rigid tracking. Our method uses a multi-scale deformation graph for non-rigid tracking.

Please see the paper for more details.

Limitations

Our approach is mainly dependent upon the initial segmentation module. One wrong segmentation itself causes bad camera motion estimation and results in huge error accumulation in reconstruction. We are using a simple frame-to-frame segmentation approach that works well in simple conditions like a person standing isolated from other objects in the scene. The skeleton detection package openpose detects wrong skeletons in the background in case of human limb-like structures or shadows from the human body. This kind of problems can be rectified by using reinforced background modeling approaches along with our approach. Another main drawback is that our approach fails to track highly articulated motions of the human body. Usage of skeleton-based deformation modeling can be used for reducing this limitation. The system cannot handle topological changes such as object separation and closed-to-open motions.

Acknowledgements

We thank ChanohPark, Agniva Sengupta and Eranda Tennakoon for helpful discussions. This webpage template is taken from humans working on 3D who borrowed it from some colorful folks.