AI Data Extraction & Web Scraping

Description

We track pricing and availability across roughly 40 competitor sites for a retail analytics product. Our current scrapers are a pile of scripts that break weekly and fail silently, which is worse than failing loudly — we have shipped reports built on three-week-old data. We want the collection layer rebuilt properly: scheduled runs, change detection when a site's markup shifts, alerting when a source goes stale, and a clear record of when each field was last successfully collected. Please only apply if you are comfortable discussing robots.txt, rate limiting and terms of service. We collect publicly listed prices, we identify our crawler, and we are not interested in anything that requires evading detection.

• Rebuild collection for the existing sources with scheduled, monitored runs
• Detect markup changes and alert rather than silently returning empty or wrong fields
• Record per-field collection timestamps so downstream reports can refuse stale data
• Respect robots.txt, rate limits and a clearly identified user agent
• Provide a documented path for us to add a new source without your involvement

• Collection service covering the current sources, deployed and scheduled
• Monitoring and alerting for stale or broken sources
• Data freshness view showing last successful collection per source and field
• Guide for adding a new source, written for our own engineers