Data ToolyScrape Product Prices with Python from JSON-LD
Scrape product prices with Python from schema.org JSON-LD: Offer, offer lists, AggregateOffer, @graph and variants, tested on live stores, and where it breaks.
Published · Updated · By Omar Eldeeb
To scrape product prices with Python, fetch the page with requests, parse every <script type="application/ld+json"> tag with json.loads, find the schema.org Product nodes, and read offers: price, priceCurrency and availability. Handle the real-world shapes: @graph arrays, lists of offers, AggregateOffer (lowPrice/highPrice), prices inside priceSpecification, and ProductGroup variants.
Below is a script that handles all of these, its output on live stores on 2026-10-20, the stores where the same approach returned nothing that day, and how E-commerce Product Scraper Pro handles the cases plain JSON-LD can't.
Why JSON-LD
Many stores embed schema.org product data in their HTML for search engines, usually as JSON-LD. It's machine-readable data the store publishes on purpose, so it changes less often than the page's CSS classes. The fields you need are defined by schema.org:
Offer:price,priceCurrency,availability(a URL such ashttps://schema.org/InStock), oftenskuandurl.AggregateOffer: used "when a single product is associated with multiple offers", withlowPrice,highPriceandofferCount.ProductGroup: "a group of Products that vary only in certain well-described ways, such as by size, color", with the variants inhasVariant.
The shapes you'll actually meet
Every row below is a live page I parsed on 2026-10-20:
| Store (platform) | Where the price was |
|---|---|
| IKEA US | A plain Offer with "price": "79" as a string, priceCurrency alongside |
| Marks & Spencer | No top-level price: it's in offers.priceSpecification (a UnitPriceSpecification with price, minPrice, maxPrice, priceCurrency) |
| Nalgene (WooCommerce) | Inside @graph; a ProductGroup whose hasVariant items each have an offer; priceSpecification serialised as an object with a "0" key holding the currency |
| Death Wish Coffee (Shopify) | One Product with a list of Offers, one per variant, each with its own sku and price |
| Skullcandy (Shopify) | A ProductGroup with hasVariant, plus a separate Product for the default variant |
A parser that only reads data["offers"]["price"] gets two of these five. The script walks the whole document instead.
The script
"""Extract product prices from schema.org JSON-LD. pip install requests beautifulsoup4"""
import json
import sys
import requests
from bs4 import BeautifulSoup
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def walk(node):
"""Yield every dict in a JSON-LD document: top-level lists, @graph, nested values."""
if isinstance(node, list):
for item in node:
yield from walk(item)
elif isinstance(node, dict):
yield node
for value in node.values():
if isinstance(value, (dict, list)):
yield from walk(value)
def has_type(node, name):
t = node.get("@type")
return any(str(x).rsplit("/", 1)[-1] == name for x in (t if isinstance(t, list) else [t]) if x)
def number(value):
try:
return float(str(value).replace(",", "")) if value not in (None, "") else None
except ValueError:
return None
def price_spec(spec):
"""priceSpecification can be a dict, a list, or (WooCommerce) a dict with a "0" key."""
specs = spec if isinstance(spec, list) else [spec]
for s in specs:
if not isinstance(s, dict):
continue
nested = s.get("0")
price = s.get("price", nested.get("price") if isinstance(nested, dict) else None)
currency = s.get("priceCurrency") or (nested.get("priceCurrency") if isinstance(nested, dict) else None)
if price is not None:
return number(price), currency
return None, None
def offers(product):
"""One dict per offer: handles Offer, a list of Offers and AggregateOffer."""
raw = product.get("offers") or []
out = []
for o in raw if isinstance(raw, list) else [raw]:
if not isinstance(o, dict):
continue
if has_type(o, "AggregateOffer") and o.get("offers"):
out += offers({"offers": o["offers"]}) # real offers nested inside
continue
price, currency = number(o.get("price")), o.get("priceCurrency")
if has_type(o, "AggregateOffer"):
price = number(o.get("lowPrice"))
if price is None and "priceSpecification" in o:
price, spec_currency = price_spec(o["priceSpecification"])
currency = currency or spec_currency
out.append({
"price": price,
"high_price": number(o.get("highPrice")), # AggregateOffer only
"currency": currency,
"availability": str(o.get("availability") or "").rsplit("/", 1)[-1] or None,
"sku": o.get("sku") or product.get("sku"),
"url": o.get("url"),
})
return out
def extract(html):
"""All (product name, offer) pairs in the page's JSON-LD, de-duplicated."""
soup = BeautifulSoup(html, "html.parser")
rows, seen = [], set()
for tag in soup.find_all("script", type="application/ld+json"):
try:
doc = json.loads(tag.string or "")
except json.JSONDecodeError:
continue # malformed JSON-LD is common; skip it
for node in walk(doc): # also reaches ProductGroup.hasVariant
if not has_type(node, "Product"):
continue
for o in offers(node):
key = (o["sku"], o["price"], o["url"])
if o["price"] is not None and key not in seen:
seen.add(key)
rows.append({"name": node.get("name"), **o})
return rows
def fetch(url):
try:
r = requests.get(url, headers=HEADERS, timeout=20)
except requests.RequestException as e:
return None, f"request failed: {type(e).__name__}"
return r, f"HTTP {r.status_code}"
if __name__ == "__main__":
for url in sys.argv[1:]:
r, status = fetch(url)
rows = extract(r.text) if r is not None and r.ok else []
print(f"\n{url}\n {status}, {len(rows)} offer(s) in JSON-LD")
for row in rows[:3]:
print(f" {row['name'][:45]!r} | {row['sku']} | {row['price']} {row['currency']} | {row['availability']}")Design notes:
walk()visits every dict, so it findsProductnodes at the top level, in a list, inside@graph, and insideProductGroup.hasVariantwithout special cases. Theseenset removes the duplicates that this produces when a page describes the same variant twice (Skullcandy does).has_type()accepts lists and full URLs, because@typecan be["Product", "Thing"]or"http://schema.org/Product".number()handles strings, because"79"and79are both common.- Bad JSON is skipped, not fatal. Some stores publish invalid JSON-LD.
- A browser-like
User-Agentis sent because several stores answer the defaultpython-requestsagent differently. It doesn't get you past real bot protection (see below).
Run it with Python 3.12 (pip install requests beautifulsoup4):
python product_prices.py https://www.ikea.com/us/en/p/billy-bookcase-white-20522046/ https://www.marksandspencer.com/fair-isle-crew-neck-relaxed-jumper-with-wool/p/clp61248839 https://nalgene.com/product/32oz-wide-mouth-surrealist-print-bottles/ https://www.deathwishcoffee.com/products/power-surge-coffee https://www.skullcandy.com/products/crusher-540-activeOutput on 2026-10-20:
https://www.ikea.com/us/en/p/billy-bookcase-white-20522046/
HTTP 200, 1 offer(s) in JSON-LD
'BILLY Bookcase - white 31 1/2x11x79 1/2 "' | 205.220.46 | 79.0 USD | InStock
https://www.marksandspencer.com/fair-isle-crew-neck-relaxed-jumper-with-wool/p/clp61248839
HTTP 200, 1 offer(s) in JSON-LD
'Fair Isle Crew Neck Relaxed Jumper' | P61248839 | 34.0 GBP | InStock
https://nalgene.com/product/32oz-wide-mouth-surrealist-print-bottles/
HTTP 200, 3 offer(s) in JSON-LD
'32oz Wide Mouth Surrealist Bottles - landscap' | 682024-0052 | 18.99 USD | None
'32oz Wide Mouth Surrealist Bottles - hummingb' | 682024-0051 | 18.99 USD | None
'32oz Wide Mouth Surrealist Bottles - astro-mo' | 682024-0050 | 18.99 USD | None
https://www.deathwishcoffee.com/products/power-surge-coffee
HTTP 200, 8 offer(s) in JSON-LD
'Power Surge Coffee' | LGPWSR16 | 19.99 USD | InStock
'Power Surge Coffee' | LGPWSR32K | 39.98 USD | InStock
'Power Surge Coffee' | LGPWSR48K | 59.97 USD | InStock
https://www.skullcandy.com/products/crusher-540-active
HTTP 200, 6 offer(s) in JSON-LD
'Crusher® 540 Active - Skullcandy X Baker Skat' | S6EVW-U054 | 229.99 USD | InStock
'Crusher® 540 Active - Smoke' | S6EVW-T013 | 209.99 USD | InStock
'Crusher® 540 Active - Forest' | S6EVW-T016 | 209.99 USD | InStockFive stores, five shapes, one parser. Nalgene's variant offers carry no availability, so it prints None. That is "unknown", not "out of stock".
Testing the AggregateOffer branch
None of the pages I tried on 2026-10-20 used AggregateOffer, so the branch is tested on a small inline document shaped like schema.org's definition, together with a broken tag:
from product_prices import extract
html = """
<script type="application/ld+json">
{"@context": "https://schema.org", "@type": "Product", "name": "Demo kettle", "sku": "K-1",
"offers": {"@type": "AggregateOffer", "lowPrice": "39.99", "highPrice": "54.00",
"offerCount": 4, "priceCurrency": "USD"}}
</script>
<script type="application/ld+json">{"@type": "Product", "name": "Broken", </script>
"""
for row in extract(html):
print(row){'name': 'Demo kettle', 'price': 39.99, 'high_price': 54.0, 'currency': 'USD', 'availability': None, 'sku': 'K-1', 'url': None}An AggregateOffer is a price range, not a price. Store lowPrice and highPrice separately and don't compare a range with a single price in a price-change alert.
Where it breaks
The same script, same day, same connection (one home connection outside the US; results depend on where and when you run it):
https://www.target.com/p/lego-marvel-spider-man-vs-anti-venom-heist-30725/-/A-94828796
HTTP 200, 0 offer(s) in JSON-LD
https://www.amazon.com/dp/B088NRLMPV
HTTP 200, 0 offer(s) in JSON-LD
https://www.sephora.com/product/peptide-glazing-fluid-P517531
HTTP 403, 0 offer(s) in JSON-LD
https://www.zalando.de/puma-club-ii-era-unisex-sneaker-low-whiteblackvapor-gray-pu115o0sb-a11.html
HTTP 403, 0 offer(s) in JSON-LDEach failure is a different problem:
- The data is in the app's state, not JSON-LD. Target's and Walmart's product pages returned HTTP 200 with the product title, but no Product JSON-LD in what
requestsreceived. Both pages contained a__NEXT_DATA__script (Target also__TGT_DATA__): the page is built from a JavaScript state object. You'd parse that object per site, or render the page. - The store doesn't use JSON-LD for the price. Amazon's page returned 200 with the price in its HTML (
a-priceelements), not in JSON-LD. That needs site-specific parsing. - Bot protection. Sephora and Zalando answered 403; John Lewis and Waitrose didn't answer at all within 20 seconds (
ReadTimeout) in earlier tries. A different User-Agent doesn't fix this; it needs other network routes or a real browser, and respect for the site's terms. - Variants. Some pages describe only the selected variant. Nalgene and Death Wish publish every variant; other stores don't, and then sizes and colours need the store's own product data.
- Pagination and discovery. JSON-LD describes one product page. To cover a category you need its product links first, through page parameters, "load more" APIs or the store's search, and a rule for when you've seen everything.
Scaling up: the same pages through the actor
The actor starts with the same structured data and adds what the script lacks: per-site adapters (Shopify and WooCommerce product APIs, Walmart's page data, Amazon's HTML), a route ladder that moves from datacenter to a crawler User-Agent, residential proxies or a browser only when a page is blocked, variants as variants[], pagination up to 200 pages per listing where the site supports it, and one status row per input explaining any failure. The README's supported-sites matrix says what works where; Sephora product pages are best-effort through a browser, and eBay, Best Buy, Home Depot, Costco and Etsy are unsupported.
"""Same pages through the Apify actor. pip install apify-client"""
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(
run_input={
"detailsUrls": [
{"url": "https://www.marksandspencer.com/fair-isle-crew-neck-relaxed-jumper-with-wool/p/clp61248839"},
{"url": "https://nalgene.com/product/32oz-wide-mouth-surrealist-print-bottles/"},
{"url": "https://www.target.com/p/lego-marvel-spider-man-vs-anti-venom-heist-30725/-/A-94828796"},
{"url": "https://www.amazon.com/dp/B088NRLMPV"},
],
"maxProductResults": 4,
},
max_total_charge_usd=0.10, # hard cap on what this run may cost
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
print(f"{row['site']:<16} {row['price']:>7} {row['currency']} inStock={row['inStock']}"
f" variants={row['variantCount']:<2} tier={row['tierUsed']:<11} from={'+'.join(row['dataSources'])}")Output on 2026-10-20 (the client also streams the run's log while it waits):
marksandspencer 34 GBP inStock=True variants=0 tier=datacenter from=jsonld+meta
target 4.99 USD inStock=True variants=0 tier=crawler-ua from=jsonld+meta
amazon-us 9.99 USD inStock=True variants=12 tier=datacenter from=amazon-html
nalgene.com 18.99 USD inStock=True variants=3 tier=datacenter from=jsonld+woo-apidataSources shows where each row came from. Target delivered through the crawler User-Agent route, where its page includes JSON-LD. Amazon came from its HTML. Nalgene combined JSON-LD with the WooCommerce Store API, and its row has a stock flag (inStock=True) where the page's JSON-LD had none.
Cost: the actor is pay per event, and these four products are one start plus four product rows on datacenter routes (no traffic fee): 4 × $0.0048 + $0.005 = $0.0242 on Apify's FREE tier, 4 × $0.0008 + $0.005 = $0.0082 on GOLD, from the README's pricing table. Residential and browser sites add per-MB and per-render fees. Apify's free plan gives $5 of usage a month with no credit card required; there's no in-actor free trial.
During Black Friday and the holiday season
The script is enough to start a baseline for a handful of stores that serve JSON-LD: run it once a day from cron or Task Scheduler and append the rows to a CSV, beginning now, about five weeks before Black Friday (Friday 27 November 2026). In the sale window, prices change several times a day, so increase the frequency for the few products you care about and keep the daily readings for medians. Compare the sale price with your own 30-day median, not with the store's "was" price: how to spot fake Black Friday discounts shows the method, and the holiday sales calendar lists when each event's baseline should start.
Related: scraping product variants, Shopify products.json pagination, competitor price monitoring, and the e-commerce price tracker query builder. Respect each site's terms and robots rules, and keep request rates low.
Disclosure: I built the actor and this site. The JSON-LD script above works on its own, without either.
Sources: schema.org Offer, schema.org AggregateOffer, schema.org ProductGroup, E-commerce Product Scraper Pro README (supported sites, pricing), and live requests to the listed product pages on 2026-10-20.
Quick answers
What is the easiest way to scrape a product price in Python?
Fetch the product page with requests and read the schema.org Product data in its <script type="application/ld+json"> tags: the price is in offers.price with offers.priceCurrency. It's structured data the store publishes on purpose, so it's more stable than CSS selectors. It only works on pages that include it in the HTML they send.
Why does my Python scraper return no price?
The usual causes: the page builds its content with JavaScript, so the price isn't in the HTML requests receives; the site blocked the request (HTTP 403, a captcha page or a hang); or the price sits in a shape your code doesn't handle, such as priceSpecification, an AggregateOffer or a @graph array. Print the status code and the JSON-LD types first.
Should I use BeautifulSoup or a regex for JSON-LD?
Use an HTML parser such as BeautifulSoup to find the script tags, then json.loads on each one's text. A regex over raw HTML breaks on attribute order, whitespace and multiple tags. Wrap json.loads in try/except, because some stores publish invalid JSON-LD.
How do I get the price of each size or colour?
Look for one Offer per variant in offers (common on Shopify) or for a ProductGroup whose hasVariant lists one Product per variant, each with its own offer and SKU. If the JSON-LD only describes the selected variant, you need the store's own product data (for example Shopify's product JSON) or a rendered page.