3dqa-benchmark

The 3DQA Benchmark repository offers an insightful look into the state of AI-generated 3D assets, featuring a comprehensive analysis based on a census of 2,307 assets gathered from the 3D Arena dataset hosted on Hugging Face. This project focuses on measuring the quality of generative AI exports without hand-typed or extrapolated data, ensuring that all metrics are generated from actual files.

Key Insights

The benchmark is built upon robust methodologies, employing tools such as batch_lint.py and batch_lint_parallel.py to assess the generated assets for various defects. The findings reveal critical insights into the prevalence of common issues in the generated assets:

Generator	N	Not Watertight	Non-Manifold	UVs Missing	Inverted Normals	Inconsistent Normals	Floating Debris	Over Poly Budget	Vertex-Colour-Only
ALL	2307	40.1%	9.7%	39.7%	0.0%	1.8%	4.7%	61.9%	14.9%
3DTopia-XL	101	36.6%	0.0%	0.0%	0.0%	1.0%	21.8%	48.5%	0.0%
IM-MA	101	84.2%	53.5%	100.0%	0.0%	4.0%	16.8%	0.0%	100.0%
Real3D	101	100.0%	44.6%	100.0%	0.0%	0.0%	2.0%	99.0%	100.0%
...	...	...	...	...	...	...	...	...	...

Note that the above table highlights the defect percentages by class and generator, offering a comparative view of asset quality across different AI generators.

Asset Quality Assessment

A notable portion of the analyzed assets presents various defects such as:

1. Not Watertight surfaces that could lead to rendering issues.
2. Non-manifold geometries, which complicate mesh processing operations.
3. Missing UV maps that are critical for texturing.

Performance Metrics

Performance benchmarking showed the median face count for assets is significantly higher than the typical budget for mobile applications, indicating a potential area for optimization. The median time taken for lint checks per asset is approximately 15.5 seconds.

Vision-Backed Sample

A separate vision-backed sampling has also been conducted, providing a more thorough quality check that includes healing, rendering, and visual verification. Detailed results of this sample can be found in the report linked within the repository.

Conclusion

This repository contributes to a better understanding of AI-generated 3D assets and their quality, providing valuable data for developers and researchers in the field of generative design. It demonstrates the importance of quality assurance in AI outputs, advocating for continued improvements in generative methodologies.