noaa-radar-pipeline

The NOAA Radar Pipeline is a self-hosted solution that leverages free data from the NOAA to replace costly commercial radar tile APIs, typically priced at around $1,000 per month, with a remarkably economical setup costing only approximately $4 per month using an EC2 instance.

Key Features

• MRMS Live Radar Data: This pipeline downloads and processes NOAA's Multi-Radar Multi-Sensor (MRMS) GRIB2 data every 5 minutes, transforming it into Web Mercator projection. It then applies distinctive color ramps for rain, snow, and hail, generating XYZ PNG tiles that are uploaded to Cloudflare R2.

• HRRR 24-Hour Forecasts: The pipeline fetches the High-Resolution Rapid Refresh (HRRR) extended forecast data from AWS Open Data. By utilizing byte-range requests, only essential data (~5 MB) is downloaded instead of the entire GRIB2 file (~700 MB). It then computes 24-hour accumulated precipitation categorized by type (rain, snow, ice) and generates XYZ tiles, also uploaded to Cloudflare R2.

Both the MRMS and HRRR pipelines deliver tiles at a high resolution of 0.01° (~1 km) at zoom levels 3–8, offering a vital feature of precipitation type differentiation that is often lacking in commercial alternatives.

Demo Application

A practical implementation of this pipeline can be seen in the iOS weather application LucidSky, which utilizes the radar data for live updates and 24-hour precipitation forecasts.

Usage Instructions

Tile Generation and Upload

Run the full MRMS or HRRR pipeline using npm commands:

# For MRMS
npm run mrms

# For HRRR
npm run hrrr

To run steps individually, use:

npm run mrms:generate
npm run mrms:upload
npm run hrrr:generate
npm run hrrr:upload

Control the retention of frames with:

npx tsx scripts/upload-mrms-tiles.ts --keep 20  # keep 20 recent frames
npx tsx scripts/upload-hrrr-tiles.ts --keep 2   # keep 2 recent HRRR runs

Serving Tiles

Tiles are served directly from the R2 CDN with the following patterns:

{R2_RADAR_TILES_URL}/{timestamp}/{z}/{x}/{y}.png
{R2_FORECAST_TILES_URL}/{run}/{z}/{x}/{y}.png

Infrastructure Setup

The repository includes scripts for deploying the pipeline on an AWS EC2 instance, with the recommended instance type being t4g.small in the us-east-1 region to leverage zero egress costs from NOAA S3. The setup process involves:

1. Launching an EC2 instance with an initial configuration script.
2. Provisioning the instance to install necessary dependencies and initiate the pipeline, which is scheduled to run MRMS every 5 minutes and HRRR hourly without overlapping execution.

Cost Efficiency

This setup stands out for its cost-effectiveness:

By utilizing the NOAA Radar Pipeline, users benefit from access to high-resolution radar data without the burden of high costs, making it a valuable solution for developers and applications reliant on accurate, up-to-date weather data.