The problem
E-commerce and market intelligence teams need accurate, real-time pricing data from competitor stores. Doing this manually involves copy-pasting tables from dynamic Single Page Applications (SPAs) that require JavaScript execution, authentication, and pagination handling. It is tedious, error-prone, and impossible to scale manually.
What I built
A resilient automated crawler using Puppeteer and Playwright configured for stealth and reliability:
- Headless browser workers that handle client-side rendering, cookie consents, dynamic scroll loading, and pagination.
- Data extraction pipeline normalizing pricing, currency conversions, availability, and SKU attributes into structured JSON.
- Automated diff detection that only records modified records and alerts stakeholders when price changes exceed set thresholds.
Outcome
- [[TODO: Samuel to add metric — e.g. Replaced X hours of daily manual data collection]]
- Clean, structured datasets delivered on an automated daily schedule.
- Resilient retry logic to handle rate-limiting and layout adjustments without breaking the worker.