Case study · Data Engineering
Automated Market Intelligence
A system that collects, normalizes and monitors data across hundreds of public websites.
- Discipline
- Data Engineering
- Scope
- Extraction, pipeline, storage, monitoring
- Stack
- Python, Playwright, PostgreSQL, Docker, AWS
The problem
The problem
Pricing, availability and product changes were tracked across a large set of public websites by hand. Every source had a different structure, the process took hours each week, and updates were still missed.
The approach
How we built it.
- 01Audited every source and grouped them by structure and rendering method.
- 02Defined one normalized schema for products, prices and availability.
- 03Built resilient extractors per source group, using Playwright and plain HTTP where possible.
- 04Scheduled collection with validation, change detection and alerts on failure.
System components
- Source registry
- Extraction workers
- Normalization
- PostgreSQL
- Change detection
- Reporting views
The outcome
Automated data collection and centralized reporting.
- Data collection runs automatically on schedule.
- Every source feeds one normalized dataset.
- Reporting is centralized instead of assembled by hand.
- Broken sources raise alerts instead of failing silently.
Have a processthat shouldrun itself?
Tell us what is slowing you down. We’ll figure out what can be automated, engineered or rebuilt.