A Price Scraper That Survives The Site Changing Its Markup

The Problem

Someone checks four competitor sites every Monday and pastes prices into a spreadsheet. The obvious fix is a scraper, and the obvious scraper works for three weeks, then a site ships a redesign and it starts writing nulls that nobody notices until the sheet is useless.

Prerequisites & Tool Stack

Everything this blueprint calls. Check you have them before you start - the usual reason a build stalls halfway is a key that takes a day to get approved.

Some links above are partner links and we earn a commission if you sign up through them. It costs you nothing extra, and it does not decide what goes in a blueprint - the self-hosted option is recommended wherever it is genuinely the better call.

What It Costs To Run

Monthly, at the volumes this blueprint was tested against. Worth comparing against what the manual version of this work costs you in hours.

Technical Breakdown

The decisions that matter, in the order you will meet them. Everything here is a thing that broke in testing before it was a rule.

from decimal import Decimal
from pydantic import BaseModel, field_validator

class Observation(BaseModel):
    sku: str
    price: Decimal
    currency: str
    in_stock: bool
    source_url: str

    @field_validator("price")
    @classmethod
    def sane_price(cls, v: Decimal) -> Decimal:
        # A selector that stops matching returns 0, not an exception.
        # Without this the pipeline writes zeroes for a week in silence.
        if v <= 0 or v > Decimal("1000000"):
            raise ValueError(f"implausible price: {v}")
        return v

ratio = extracted / max(attempted, 1)
if ratio < 0.9 * trailing_average(target, days=7):
    alert(
        f"{target}: extraction dropped to {ratio:.0%} "
        f"({extracted}/{attempted}) - selectors likely stale"
    )

When It Goes Wrong

The difference between a demo and something you can leave running is entirely in this table. Every row is a failure the blueprint handles explicitly rather than hoping about.

Payload Example

One row written per observation.

{
  "target": "competitor-a",
  "sku": "NW-4410-BLK",
  "price": "289.00",
  "currency": "USD",
  "in_stock": true,
  "source_url": "https://example.com/p/nw-4410",
  "observed_at": "2026-07-30T06:14:02Z",
  "extraction_ratio": 1.0,
  "via": "scrapingbee:us"
}