Pharma and life-science job data collection
Scheduled scrapers that collect complete job listings from four pharma and life-science sources into one database, including 52,000+ records from one source.
- Year
- 2026
- Service
- AI Agents & Automation
The challenge
The client needed complete, up-to-date pharma and life-science job data from several sites, each built differently, collected every day without anyone copying and pasting.
What we built
Four sources
Dedicated scrapers for PharmIWeb, HireLifeScience, ProClinical and VisibleLegacy.
Full detail, not just titles
Each listing's full description and every metadata field, from location to salary and job type.
One clean database
Results are added or updated in MongoDB, so re-runs never create duplicates.
Resumes where it stopped
Progress is saved as it goes, so a long run can pick up after an interruption.
Runs on a schedule
All scrapers run in sequence on a schedule, each independent so one failure never stops the rest.
How it works
- ScheduleThe scheduler starts the run.
- CollectEach scraper works through listings and detail pages.
- StoreRecords are saved or updated in MongoDB.
- RepeatThe next run picks up new and changed listings.
Technology
- Python
- Playwright
- BeautifulSoup
- Requests
- MongoDB
- Cronicle
