Wool

Wool is a powerful distributed Python runtime designed for executing tasks across a scalable pool of worker processes without relying on a centralized scheduler. By leveraging a decentralized peer-to-peer network, Wool efficiently dispatches routines directly to workers, allowing applications to manage cluster lifecycle and node orchestration using existing tools, such as Kubernetes, while Wool focuses exclusively on distributed execution.

Key Features

• Decentralized Execution: Wool allows for the execution of any async function or generator remotely by simply decorating it, maintaining the original async semantics of the function. This enables consistent behavior in terms of return types, streaming, cancellation, and exception handling.

• Flexible Worker Pools: The main entry point for executing routines is the WorkerPool, which orchestrates worker subprocess lifecycles and handles load-balanced dispatch. It supports various configurations:

  • Default Mode: Automatically spawns local workers based on CPU count.
  • Ephemeral Mode: Allows a defined number of local workers.
  • Durable Mode: Connects to existing workers across the network.
  • Hybrid Mode: Combines local worker spawning with remote discovery.

Easy to Use

Wool simplifies the process of making async routines executable in a distributed manner. Here’s a quick example of how to define and use a routine:

import asyncio
import wool

@wool.routine
async def add(x: int, y: int) -> int:
    return x + y

async def main():
    async with wool.WorkerPool(size=4):
        result = await add(1, 2)
        print(result)  # Outputs: 3

asyncio.run(main())

Discovery and Load Balancing

Wool includes built-in discovery protocols for efficient management of workers:

• LocalDiscovery is designed for single machines.
• LanDiscovery utilizes Zeroconf DNS-SD for network-wide discovery, eliminating the need for a central coordinator.

The load balancing mechanism decides which worker will handle each task. Wool defaults to a round-robin strategy but allows for the implementation of custom load balancers.

Both of these components can be easily extended to support custom use cases.

Error Handling and Security

Wool captures exceptions thrown during routine execution, providing comprehensive error handling while preserving the original exception types and tracebacks. It also supports secure communication with WorkerCredentials to manage TLS connections.

Visualization of Worker Lifecycle

The architecture diagram below illustrates the complete lifecycle of a Wool worker pool, outlining processes from startup and discovery to task dispatch and teardown:

sequenceDiagram
    participant Client
    participant Routine
    participant Pool
    participant Discovery
    participant Loadbalancer
    participant Worker

    %% ── 1. Pool startup ────────────────────────────────
    rect rgb(0, 0, 0, 0)
        Note over Client, Discovery: Worker pool startup
        Client ->> Pool: create pool (size, discovery, loadbalancer)
        activate Client
        Pool ->> Pool: resolve mode from size and discovery
        opt If size specified, spawn ephemeral workers
            loop Per worker
                Pool ->> Worker: spawn worker
                Worker ->> Worker: start process, bind gRPC server
                Worker -->> Pool: worker metadata (host, port, tags)
                Pool ->> Discovery: publish "worker added"
            end
        end
        Pool ->> Pool: create proxy (discovery subscriber, loadbalancer)
        Pool -->> Client: pool ready
        deactivate Client
    end

    %% ── 2. Task dispatch ─────────────────────────────────
    ...

Wool is uniquely positioned to facilitate distributed task execution, thereby improving the scalability and efficiency of Python applications in various environments. For more information and examples, visit the Wool repository.