MARS

MARS (Multi Asset Reconstruction for Simulation) is an innovative automated pipeline designed to transform 2D images into physics-ready 3D scene assets specifically for robotics training. This powerful toolchain offers a comprehensive solution that encompasses detection, segmentation, reconstruction, and validation processes to create high-quality 3D simulations.

Key Features

• Advanced Object Detection: Utilize hybrid vision-language models such as Qwen 2.5 VL and GroundingDINO for accurate object detection in images.
• Precision Segmentation: Leverage the Segment Anything Model (SAM) to generate precise object masks from images.
• Complete 3D Reconstruction: Generate complete 3D geometry and textures using SAM 3D Objects.
• Physics Property Estimation: Automatically estimate important physical properties including mass, friction, and inertia for realistic simulations.
• Robust Scene Validation: Validate the scenes using PyBullet physics simulation to ensure physical integrity.
• Multi-format Export: Effortlessly export assets to various formats such as USD, MJCF, and URDF, ensuring compatibility with numerous simulation environments.

Pipeline Overview

The MARS pipeline executes the following stages:

Image → Detection → Segmentation → 3D Reconstruction → Physics → Validation → Export

Each stage works collaboratively to facilitate smooth transitions from raw images to detailed 3D assets ready for simulations.

Monitoring and Reporting

Upon completion of a pipeline run, MARS delivers a detailed TUI summary that includes:

• Detection Results: A summary of detected objects along with their confidence scores and bounding boxes.
• Segmentation Results: Information on mask areas, stability scores, and Intersection over Union (IoU) predictions.
• 3D Reconstruction Results: Detailed metrics on mesh characteristics including vertex and face counts.
• Physics Properties: Thorough physical property data for each object.
• Validation Insights: Stability scores and displacement measurements post-validation.

Example Usage

Input: A simple 2D image of a real-world setup.

Output: A physics-ready 3D scene imported into Isaac Sim.

Development and Customization

MARS offers extensive opportunities for development with its modular architecture. The project structure is designed for maximum flexibility, allowing for immediate code editing without the need to rebuild the container. Configuration files are readily available for users to customize the behavior of various pipeline components, including detection, segmentation, and physics estimation settings.

For more detailed information on setup, usage, and customization, refer to the documentation provided within the repository. MARS paves the way for efficient and effective robotics training by delivering high-fidelity 3D assets derived from real-world imagery.