swath

swath is an innovative command-line tool designed to efficiently list very large Amazon S3 buckets. It operates in parallel to intelligently split the keyspace, ensuring optimal performance while listing. With its advanced features, swath excels in handling large datasets with ease and precision.

Key Features

• Crash-Safe Resume: The tool allows users to interrupt listings and resume from checkpoints, ensuring that progress is not lost even in the event of a failure. Real-time interruptions can be handled with the ability to re-query the output efficiently.
• Parallel Processing: By partitioning the keyspace into disjoint ranges, swath maximizes throughput. Workers are dynamically adjusted to balance the workload as actual keys are retrieved, making it effective for varied key distributions.
• Output Formats: swath generates globally sorted Parquet outputs, as well as JSONL and TSV formats. This makes it versatile for data querying and analytics.

Usage Example

To list objects from an S3 bucket, use the following command:

docker run --rm -v "$PWD/out:/out" ghcr.io/varveio/swath:latest \
  list s3://my-bucket/prefix/ --no-sign-request --format parquet -o /out/data

Or from source:

./gradlew :swath-cli:installDist
export PATH="$PWD/swath-cli/build/install/swath/bin:$PATH"
swath list s3://my-bucket/prefix/ --no-sign-request --format parquet -o out/

Behavior and Limits

• Handling Key Distributions: Designed specifically for very large listings, swath effectively manages a wide range of key distributions without requiring manual intervention for partitioning.
• Cost Efficiency: swath incurs costs based on the number of LIST requests made, with transparency in reporting actual API call costs.
• Replay Server: The swath-replay-server component allows captured listings to be served as an S3-compatible endpoint. This enables deterministic access to previously listed bucket shapes without incurring additional S3 charges.

When to Use swath

swath is ideal for scenarios where precomputed listings (such as AWS S3 Inventory) are not available or practical. It provides a solution for users needing real-time insight into their S3 bucket's contents without direct reliance on object data.

Documentation and Contributions

Extensive documentation is provided to assist users with installation, usage, and troubleshooting. The architecture and internal workings of swath can also be explored through dedicated documentation sections.

For those interested in contributing, guidelines are outlined in the contribution section of the repository.