factlabel

factlabel: Ensuring Accuracy in AI-Written Content

factlabel serves as an essential tool that audits AI-generated content by validating claims against underlying data. This innovative solution not only identifies inaccuracies but also explains the basis for any discrepancies, providing transparency to readers.

The Challenge

In an era where AI agents craft important communications such as fundraising reports, financial summaries, and dashboards, the integrity of the accompanying narratives often comes into question. Common failures include:

• Invented Figures: Claims that appear reasonable do not align with actual data (e.g., stating more than 2,000 new donors when only 9 exist).
• False Attribution: Misleading attributions that obscure the true source of funding (e.g., suggesting grassroots momentum while most funds come from a single corporate gift).
• Spin: Misrepresentation of data trends (e.g., stating an increase in individual giving when the actual figure is a decrease).

Relying on another AI model for verification falls short due to potential biases and slow processing speeds, making automated scrutiny necessary.

How factlabel Works

factlabel functions as an intermediary between AI-generated drafts and the reader, performing the following actions:

1. Draft Auditing: Each component, such as figures and trends, is rigorously checked against source records.
2. Publish Gates: Drafts are classified as pass, review, or block, ensuring that only verified content is published.
3. Annotated Findings: Each draft is annotated with numbered findings, explanations for flags raised, and a Trust Facts badge summarizing overall content quality.
4. Content Correction (Optional): Through an integrated revision loop, factlabel can correct inaccuracies, reauditing until the draft meets passing criteria or necessitates human intervention.

Trust Facts

The Trust Facts badge provides a comprehensive overview of the draft, summarizing findings about numbers, sourcing, attribution, framing, and completeness. Each finding corresponds to an explanation, enhancing the trustworthiness of the content.

Why Choose factlabel?

factlabel leverages a unique three-tiered audit system:

[agent draft + source data]  
            │  
            ▼  
┌──────────────────────────────────────────────────────┐  
│ Tier 1: Deterministic Audit (Code-Only)              │  
│ - Metric recomputation  
│ - Figure extraction and validation  
│ - Anomaly detection of material facts  
└──────────────────────────────────────────────────────┘  
            │  
            ▼  
┌──────────────────────────────────────────────────────┐  
│ Tier 2: Parallel Jev Audit Matrix                     │  
│ - Blinded and grounded analyses to assess claims     │  
└──────────────────────────────────────────────────────┘  
            │  
            ▼  
┌──────────────────────────────────────────────────────┐  
│ Tier 3: Supervisory Mediation (Code)                 │  
│ - Confidence-based rule applications                  │  
└──────────────────────────────────────────────────────┘  

The system's sophistication lies in its reliance on code for comparisons, ensuring accuracy without the risk of editorial influence.

Benchmarking Performance

factlabel's auditing capability has been benchmarked against both misleading and honest drafts, demonstrating a 100% accuracy in identifying errors, verifying claims, and maintaining trustworthiness in presented data.

Key Features of factlabel

• Efficient Auditing: Rapidly audit drafts through a combination of automated checks, providing results in under one second.
• Transparent Processes: Each finding generates a clear path of inquiry, detailing how conclusions were reached.
• Scalability: enjoy the cost-effectiveness of stringent auditing with an approximate cost of $0.0006 per review, making it feasible to implement in continuous workflows.

How to Use factlabel

Here’s a quick usage example to run a basic audit:

factlabel audit examples/fundraising/spun_draft.json --html out/label.html  

This command validates a provided draft and produces an annotated output.

Factlabel is open-source and continuously evolving through community contributions. For those interested in improving model accuracy or exploring further complexities, active engagement is encouraged.

Emphasizing a commitment to factual integrity, factlabel equips users to navigate AI-generated content safely and transparently.