docexp

docexp is a streamlined documentation tool designed for expanding code blocks in Markdown files. It serves as a simplified alternative to Jupyter code cells, eliminating the need for complex file formats or intricate execution environments.

Features

Code Block Expansion

The core functionality of docexp focuses on parsing and expanding code blocks. For example, given a Markdown file named docs.md containing the following code block:

### Database access

To inspect the database, run a sample query:

```sql
SELECT * FROM user LIMIT 3
```

The docexp tool can interpret this block with a dry-run command as shown below:

$ docexp dry-run docs.md
{"content":"SELECT * FROM user LIMIT 3\n","id":1,"metadata":{"column":0,"filename":"docs.md","headings":{"3":"Database access"},"lang":"sql","line":7}}

The output reveals a structured JSON object containing the SQL query, its respective file position, and the contextual Markdown headings. This allows users to easily access the executed code.

Running Expansions

To execute and display the output from a code block, users must create an expander script. This script processes the JSON lines produced by the dry-run command. An example implementation in Python could resemble the following:

#!/usr/bin/env python

import sys, json, subprocess

for line in sys.stdin:
    block = json.loads(line)
    match block.get("lang"):
        case "sql":
            res = subprocess.run(
                ["psql", "-d", "db_dev01", "-c", block["content"]],
                capture_output=True,
                text=True,
                check=True,
            )
            out = {"id": block["id"], "exp": res.stdout}
            print(json.dumps(out))

After implementing the expander script, executing the following command generates the expanded output:

$ docexp run --expander exp.py docs.md

The original docs.md file is then updated to include the output:

## Database access

You can also inspect the database content directly:

```sql
SELECT * FROM user LIMIT 3
```

<!-- <docexp> -->
```
 id  |  name  |       email        
-----+--------+--------------------  
 123 | adam   | adam@example.com  
 124 | eva    | eva@example.com  
 125 | john   | john@example.com  
(3 rows)
<!-- </docexp> -->

Expander Script Management

The tool delegates much of the complexity to the expander script, allowing for various execution methodologies:

• Executing code blocks in a launched Docker container for isolated command execution.
• Using a persistent Docker container to mimic Jupyter kernel behaviors.
• Supporting different containers based on the programming language or Markdown headings.
• Implementing security checks and executing in sandboxed environments (e.g., with read-only database access).

Future iterations of docexp may introduce additional helpers to assist in the creation of expander scripts.

Advantages

Using docexp streamlines the documentation process and simplifies interaction with code. Unlike more complex content management systems like Jupyter Notebooks or Sphinx, which come with added complexities and custom file formats, docexp allows for simple Markdown with inline comment tags. It provides clear, traceable code execution capabilities without altering the original content, only adding necessary output.

If the primary requirement is the expansion of code blocks within Markdown, docexp is an optimal choice. This project is developed using Golang and reflects an initiative to enhance programming skills while providing a straightforward documentation expansion solution.