Client project · Data pipeline · Automation · Running daily
Competitor intelligence at scale
Daily data from a platform three other contractors said couldn’t be done.
A Playwright pipeline pulling structured competitor data from a heavily bot-protected platform, running daily without anyone babysitting it.
- Type
- Data pipeline
- Client
- Under NDA
- Stack
- Playwright · Python · PostgreSQL
- Status
- Running daily
2M+
records processed
5+ mo
running daily
0
manual interventions
The problem
The client needed competitor data from a platform protected by sophisticated bot detection. Three other contractors had already told them extraction wasn’t possible.
What we built
We built a Playwright-based pipeline, scheduled it to run daily, and stored clean, structured records in PostgreSQL. It was deployed like production software rather than run as a one-off script.
How it works
Runs itself
Scheduled daily runs, with no one needed to start, watch or restart them.
Clean output
Normalised, structured records in PostgreSQL, ready for analysis.
Built for a hard target
Engineered for a platform that actively resists automation.
The stack
- Automation
- Playwright · Python
- Data
- PostgreSQL
- Operations
- Scheduler
“They automated a competitor data pipeline that three other contractors said wasn't possible due to the site's bot protection. Running daily for five months now. I stopped asking how it works and just watch the data come in.”
Tell us what you’re working on
A senior engineer reads every message and replies within 24 hours, with questions or first thoughts.