Austin Garrod
View resume

Project case study

Charity Data Scraper

A full-stack scraper for public Canadian charity records with queue-based processing, live progress, retries, and MongoDB storage.

All projects

Case study

Resumable Canadian charity data pipeline

Problem

  • Large public-record collection needed to tolerate long runtimes, retries, and interruptions.
  • Parsing logic needed to evolve without forcing every source record to be fetched again.

Approach

  • Use queue-backed processing with pause/resume controls, retry handling, and rate limiting.
  • Persist raw source responses separately from processed records so transformation logic can be replayed.

System Shape

  • Node.js and TypeScript coordinate collection, Redis-backed job state, MongoDB storage, and Docker-based local services.
  • A React dashboard exposes live progress, queue visibility, search views, and controls for long-running sessions.

Current State

  • The private application is implemented with resumable discovery, configurable workers, atomic offset tracking, retries, and source-aware rate controls.
  • Only public Canadian charity records are in scope; collection settings are designed to respect source limits, and deployments remain responsible for applicable access terms.