3dqa-benchmark

Overview of the 3DQA Benchmark

The 3DQA Benchmark provides a comprehensive analysis of AI-generated 3D assets, focusing on a public census of 2,307 real generative-AI exports. This project allows users to evaluate the quality of assets produced by various generative AI models, thus facilitating the understanding of defect prevalence across different generators.

Key Features

• Corpus of Assets: An extensive collection of 2,307 assets sourced from 23 different generators hosted in the 3D Arena dataset on Hugging Face.
• Robust Measurement: Measurements are taken using batch_lint.py and batch_lint_parallel.py, ensuring accuracy without hand-typed data or extrapolations. The geometry-only assessments are deterministic and executed in millisecond-scale per asset.
• Defect Analysis: Each generator’s assets are analyzed for defects such as:
  • Non-watertight
  • Non-manifold geometry
  • Missing UVs
  • Inverted normals
  • Inconsistent normals
  • Floating debris
  • Over poly budgets
  • Vertex-colour only assets

Generator	Total Assets	Not Watertight	Non-Manifold	UVs Missing	Over Poly Budget
ALL	2307	40.1%	9.7%	39.7%	61.9%
3DTopia-XL	101	36.6%	0.0%	0.0%	48.5%
Real3D	101	100.0%	44.6%	100.0%	99.0%
...	...	...	...	...	...

Measurement Insights

The benchmark provides various statistics, including:

• Lint Errors: Out of the assets analyzed, 303 were classified as malformed or unreadable, impacting overall defect rates.
• Asset Performance: The median face count among the sampled assets was found to be 77,402, with a proportion exceeding the typical Unity-mobile budget of 30,000 triangles.
• Healing Rates: Among defective assets (1501 total), 45.0% managed to heal fully when subjected to dry-run repairs, while 5.1% did not heal effectively.
• Timeliness: The measurement timing revealed a median linting time of 15,458 ms per asset, emphasizing efficiency despite the complexity of the analysis.

Important Considerations

Several critical notes are provided to guide users in interpreting the benchmark results:

• The dataset specifically reflects assets from the 3D Arena and does not represent all AI-generated 3D models, which may have varying characteristics.
• Texture quality assessments are not included, as texture defects do not fall within the scope of this analysis.
• An additional vision-backed sample offer insights into a separate scope, adhering to rigorous verification processes.

Conclusion

The 3DQA Benchmark serves as an essential tool for developers and researchers interested in understanding the landscape of AI-generated 3D assets. Its findings contribute to ongoing improvements in generative modeling and offer a foundation for further exploration in quality assurance within the realm of 3D graphics.