Case study · AI in production
Large-Scale AI-Powered Data Crawling and ETL Platform
A large-scale .NET and AI-powered crawling platform that collects data from 3,500+ sources, parses 120,000+ records every day with a mix of AI and rule-based extractors, and has built a structured database of 122M+ records that powers products used by 5M+ people.
AIThe client
Market intelligence and marketplace products depend on fresh, complete data: listings, prices, auction lots, specifications and catalogue details spread across thousands of websites. Every site has its own layout and changes it without warning, and the raw data arrives messy, duplicated and inconsistent. The products needed a pipeline that keeps that data flowing into one clean, structured database, every day.
The challenge
- Crawl 3,500+ sources and many product categories reliably, at scale and on schedule, every day.
- Keep extraction working when page layouts change, without rewriting a crawler each time.
- Turn messy, inconsistent source data into one standard structure.
- Remove duplicates and match the same item across different sources.
- Grow to more than 100 million records in structured databases that live products query directly.
What we did
We built and run the crawling and ETL platform on .NET. Crawlers are scheduled and scaled through job queues, with headless browsers for sites that render content in the browser. Extraction is hybrid: XPath and regular expressions handle pages with a stable structure, and AI-based extraction reads pages whose layouts are irregular or change often, returning structured fields where fixed rules would break. Extracted data then goes through a transformation stage that cleans, normalises, deduplicates and matches records, before loading into structured databases that feed the client's products. Monitoring flags failing sources so the team can fix them quickly.
What it does
Large-scale crawling
Scheduled crawlers across 3,500+ sources and many categories, scaled out through job queues to parse 120,000+ records a day.
Smart AI extraction
AI models read unstructured or changing page layouts and return clean, structured fields.
Rule-based extraction
XPath and regex extractors for stable, high-volume sources where speed matters.
Transformation and cleansing
Normalises units, currencies, names and formats into one standard schema.
Deduplication and matching
Detects duplicates and links the same item across different sources.
Structured loading
Loads every record into relational databases, now 122M+ records, ready for search, analytics and product features.
Monitoring and recovery
Tracks source health and extraction quality, and flags crawlers that need attention.
How it is built
- Extract.NET crawlers with headless-browser support, scheduled and distributed through job queues.
- TransformHybrid AI and rule-based extraction, then cleansing, normalisation, deduplication and entity matching.
- LoadStructured SQL databases that downstream products query directly.
- Each source can switch between AI and rule-based extraction without rebuilding the pipeline.
Results
- 122M+ structured records delivered to live intelligence and marketplace products used by 5M+ people.
- AI-based extraction keeps data flowing when site layouts change and fixed rules would fail.
- 120,000+ records parsed every day from 3,500+ sources, clean, deduplicated and in one consistent structure.
AI extraction that keeps
data flowing when websites change
Rule-based scrapers break whenever a site changes its layout. We combined AI-based extraction with fast rule-based extractors, so millions of records keep arriving clean, structured and on schedule.
LLM-based extraction returns structured fields from pages whose layouts are irregular or change often.
Each source switches between AI and rule-based extraction without rebuilding the pipeline, balancing cost and speed.
Cleansing, normalisation, deduplication and entity matching turn raw pages into one standard structure.
Source health and extraction quality are tracked so failing crawlers are fixed fast.