next-markdown-mirror

next-markdown-mirror is a self-hosted solution that enables the serving of clean Markdown content to AI agents, instead of traditional HTML. By processing requests to deliver Markdown, this tool significantly reduces the average token usage, ensuring that AI tools produce higher-quality responses without unnecessary overhead from HTML boilerplate.

The Challenge

AI agents often encounter excessive content when they parse HTML pages, which can include navigation menus, footers, and advertisements. This cumbersome parsing can inflate the token count by 2 to 5 times compared to serving clean Markdown. As a result, leveraging next-markdown-mirror allows for a more efficient and focused content delivery method.

Comparison with Cloudflare

Cloudflare provides a similar Markdown conversion service but at a cost of $20 per month for each domain. In contrast, next-markdown-mirror is completely free and open source, allowing users to self-host and maintain full control over their Markdown output without the financial burden. For multiple domains, this free solution can result in substantial savings:

Getting Started

To integrate next-markdown-mirror into a Next.js project, the following steps are required:

1. Configure a Proxy: Intercept and rewrite Markdown requests:

   // proxy.ts
   import { withMarkdownMirror } from 'next-markdown-mirror/nextjs';
   export const proxy = withMarkdownMirror();
   

2. Create a Route Handler: Fetch the HTML internally and convert it to Markdown:

   // app/md-mirror/[...path]/route.ts
   import { createMarkdownHandler } from 'next-markdown-mirror/nextjs';
   export const GET = createMarkdownHandler({
       baseUrl: process.env.NEXT_PUBLIC_SITE_URL!,
   });
   

3. Implement llms.txt: Generate an AI discovery file:

   // app/llms.txt/route.ts
   import { createLlmsTxtHandler } from 'next-markdown-mirror/nextjs';
   export const GET = createLlmsTxtHandler({
       siteName: 'My Site',
       baseUrl: process.env.NEXT_PUBLIC_SITE_URL!,
       pages: [
           { url: '/', title: 'Home', description: 'Welcome page' },
           { url: '/about', title: 'About' },
       ],
   });
   

Key Features

• Structured Data Handling: Automatic conversion of JSON-LD to YAML frontmatter enhances content structure.
• Extended Markdown Support: Full support for GitHub Flavored Markdown (GFM), including tables, task lists, and more.
• Token Management: Configuration options to measure token utilization efficiently.
• Content Filtering: Intelligent filtering capabilities to exclude unwanted HTML elements like nav bars and ads.
• Content-Signal Header: Control how AI agents can utilize the content provided.

Core API

next-markdown-mirror is designed to work independently of Next.js, providing a standalone converter for HTML to Markdown:

import { HtmlToMarkdown } from 'next-markdown-mirror';

const converter = new HtmlToMarkdown({
    baseUrl: 'https://example.com',
    extractJsonLd: true,
    contentSignal: 'ai-input',
});

const result = converter.convert(html);
// result.markdown    — output Markdown including YAML frontmatter
// result.tokenCount  — estimated token count
// result.jsonLd      — extracted JSON-LD data
// result.title       — page title

For complete configuration options and guidance, refer to the full documentation. Utilizing next-markdown-mirror can greatly enhance efficiency for content-driven applications, especially those leveraging AI technologies.