Multi-Pose Object Alignment for 3D Gaussian Splatting

Overview

Capturing an object in several poses exposes surfaces that are occluded in a single pose. Combining these observations for 3D Gaussian Splatting requires consistent camera geometry: rotating the object changes its relationship to the cameras, while structure from motion and 3DGS assume a static scene.

This project uses a rigid ArUco marker carrier to recover those changes and align the captures in one object-centered coordinate frame. I developed the calibration and pose-estimation pipeline and derived the transformations for both camera extrinsics and sparse point clouds.

Multi-view capture setup with phone cameras, controlled lighting, an object and a rigid ArUco marker carrier. — Capture setup from the project poster: multiple phone viewpoints and a rigid marker carrier provide the observations used for cross-pose alignment.

Capture and Calibration

The setup combines multiple phone viewpoints, controlled lighting, and an ArUco marker carrier rigidly attached to the object. The fixed marker-to-object relationship supplies a geometric reference across object poses.

Calibrate the cameras. ChArUco observations estimate camera intrinsics and lens distortion.
Capture multiple poses. Images are grouped by object pose, preserving the relationship between each image and its capture configuration.
Estimate marker poses. ArUco corner detection and PnP recover six-degree-of-freedom poses relative to the cameras.
Reconstruct each capture. Structure from motion provides camera extrinsics and a sparse point cloud for each object pose.

Alignment Pipeline

The central step is an SE(3) transformation chain that maps all captures into a reference object frame. Camera extrinsics and sparse geometry undergo the same coordinate change so their projections remain consistent.

CalibrationArUco pose estimationCross-pose alignmentJoint 3DGS optimization

Map each camera into the local coordinates associated with its object pose.
Transform each pose into a selected reference object frame.
Apply the corresponding transformation to the sparse point cloud.
Export the unified camera and geometry representation for 3DGS.

The transformed observations represent the rotating-object capture as an equivalent static object viewed by different cameras. This gives joint reconstruction access to surfaces observed in different poses.

Reconstruction and Evaluation

After distortion correction, foreground masking, and cross-pose alignment, the combined cameras and sparse geometry initialize the 3D Gaussian representation. Differentiable rendering then optimizes Gaussian positions, scales, rotations, opacity, and appearance.

The evaluation design separates geometric consistency from novel-view quality:

Geometry: marker reprojection error and cross-pose rotation and translation residuals.
Rendering: PSNR, SSIM, and LPIPS on held-out views.
Coverage: single-pose versus multi-pose observations, with visual inspection of occluded surfaces, duplicated geometry, and ghosting.

The alignment stage operates on the reconstruction inputs, making views from different object poses available to the downstream 3DGS optimizer in a common coordinate system.