Firewalk

Firewalk: A Lightweight Collection Traversal Library for Firestore and Node.js

Firewalk is a lightweight, fast, and memory-efficient library designed for traversing Firestore collections in Node.js applications. It addresses the challenges posed by large document collections, enabling developers to efficiently access, modify, and manage data without overwhelming system memory.

Key Features

• Memory Efficiency: Firewalk's traverser objects allow for batch processing with concurrency control, ensuring that memory usage remains low even when handling millions of documents.
• Simplicity: The library offers a straightforward API that simplifies the process of data access in Firestore while maintaining high performance.
• Flexibility: Ideal for various tasks including database migrations, data retrieval scripts, and scheduled functions that require iterating over large collections.

Usage Example

Here’s a quick example demonstrating how to send emails to users from a Firestore collection using Firewalk:

import { firestore } from 'firebase-admin';
import { createTraverser } from 'firewalk';

const usersCollection = firestore().collection('users');
const traverser = createTraverser(usersCollection);

const { batchCount, docCount } = await traverser.traverse(async (batchDocs, batchIndex) => {
  const batchSize = batchDocs.length;
  await Promise.all(
    batchDocs.map(async (doc) => {
      const { email, firstName } = doc.data();
      await sendEmail({ to: email, content: `Hello ${firstName}!` });
    })
  );
  console.log(`Batch ${batchIndex} done! We emailed ${batchSize} users in this batch.`);
});

console.log(`Traversal done! We emailed ${docCount} users in ${batchCount} batches!`);

Quick Start

Firewalk features a simple interface with only two primary components:

• Traverser: Facilitates traversing a collection of documents.
• Migrator: A utility for performing migrations within a specific traversable, allowing custom migration logic.

For further exploration, the library's documentation provides comprehensive guidelines on setup, configuration, and advanced use cases such as concurrency adjustment and data field updates.

Conclusion

Firewalk is an essential tool for developers who work with large Firestore collections and require efficient data processing strategies. It streamlines complex tasks with practical solutions, making data management simple and effective.