← Back to all projects

Greenwood Aero

Greenwood Aero Scraper Platform

Distributed supplier scraping and government award data pipelines for aviation parts procurement.

Role
Full Stack Developer
Timeframe
Nov 2022 — Jan 2023
Team Size
2
Domain
Aviation parts procurement

Greenwood Aero

Greenwood Aero Scraper Platform

Overview

The Greenwood Aero platform is backend tooling for an aviation-parts procurement business, aggregating availability, pricing, and catalog data from across the aerospace supplier landscape. I owned end-to-end delivery of the scrapers, crawlers, and data pipelines.

At its heart is a distributed WebSocket architecture that routes part-number searches from a central aggregator to supplier-specific scraper nodes running on separate VPS instances, scaling browser automation with Selenium Grid. A companion ETL module ingests U.S. government prime- and sub-award contract data from USAspending.gov into PostgreSQL, and fuzzy-matching search integrations help discover parts and pricing across the web.

My Role

Full Stack Developer

  • Owned end-to-end delivery of multi-vendor aviation parts scrapers and crawlers
  • Designed and built a distributed WebSocket scraper architecture across VPS-hosted supplier nodes
  • Implemented USAspending.gov ETL pipeline to ingest prime and sub-award data into PostgreSQL
  • Built Google Search and SerpAPI integrations with fuzzy matching for part discovery
  • Integrated Selenium Grid, CAPTCHA solving, and FedEx tracking utilities into the scraping workflow

Key Features

  • Shared Scraper abstract base class for consistent multi-vendor part search behavior
  • Distributed WebSocket server fleet with per-supplier nodes and central aggregator
  • Selenium Grid Docker Compose setup with multiple Firefox nodes for parallel browser sessions
  • USAspending.gov ETL pipeline for prime awards, sub-awards, and company records in PostgreSQL
  • Bulk catalog crawlers for Monroe Aerospace, Pelican Sales Supply, and Chief Aircraft
  • SerpAPI and Google Custom Search part discovery with fuzzy result ranking
  • CAPTCHA bypass integration for authenticated supplier portals
  • Vanilla JavaScript WebSocket frontend for live multi-supplier search demo

Challenges & Solutions

Problem

Selenium WebDriver instances consume 400–600 MB RAM each, limiting concurrent requests to roughly 15–25 per server

Solution

Used headless Firefox with reduced rendering, async HTTP + BeautifulSoup where sites allow it (supporting far higher concurrency), and Selenium Grid with multiple Firefox nodes

Problem

Synchronous scrapers blocked the async WebSocket event loop under concurrent load

Solution

Ran each scraper search in a thread pool executor via asyncio.run_in_executor while keeping the WebSocket server async

Problem

Protected supplier sites require login sessions and CAPTCHA solving

Solution

Implemented authenticated scraper flows with session persistence and DeathByCaptcha integration for robot-check pages

Architecture & Tech

Languages

PythonJavaScript

Frameworks

SeleniumSQLAlchemyPrefect

Databases

PostgreSQL

Tools

DockerSelenium GridWebSocketsSSHTunnelBeautifulSoup4httpxrapidfuzz

Infrastructure

Deployment
VPS (Ubuntu 22.04)
Containers
Docker, Selenium Grid
Other
Multi-server WebSocket fleet, SSH tunnel to PostgreSQL

Integrations

USAspending.gov API

Government prime and sub-award contract data ingestion

SerpAPI

Google search results for part discovery and pricing

Google Custom Search API

Alternative Google search integration for part lookup

DeathByCaptcha

CAPTCHA solving for protected supplier sites

FedEx Tracking API

Shipment status lookup by tracking number

AeroFastener

Supplier part search and pricing

Boeing Distribution

Supplier part search and pricing

Incora / Wesco Air

Supplier part search and pricing

Chief Aircraft

Supplier catalog and part search

PilotJohn

Supplier part search with authenticated sessions

Mid-Continent Instruments

Supplier part search

Monroe Aerospace

Catalog crawling and part data extraction

Pelican Sales Supply

Catalog crawling and part data extraction

Previous project

Portal V2 Backend (Arcadia)

New Hotels Development (Arcadia Resorts Club)

View case study

Have a project in mind?

Let’s talk about what you’re building and how Montgomery Labs can help ship it.

Start a project