Data Tooly
Guides · 74
Guides to public data APIs and scraping
Each guide explains one public data source in depth: whether an official API exists, what it limits or blocks, and a runnable way to get the data out. Every guide links to the free tool on this site that builds the matching query.
Updated · By Omar Eldeeb
All guides
Amazon Price History Without Keepa: Build Your OwnSee Amazon price history with Keepa or camelcamelcamel, or build your own: scheduled scrapes appended to SQLite, daily min and median, and a chart. Tested code.Amazon Price Scraper: Prices by ASIN on Any MarketplaceScrape Amazon prices by ASIN on amazon.com, .co.uk and .de: the URL pattern, the fields you get, residential cost math and the 'cannot be dispatched' trap.Back in Stock Alert Scraper: Products, Sizes, ColoursBuild back in stock alerts with a scraper: the stock fields to read, size and colour restocks from variants, null vs out of stock, check cadence, and cost.Back to School Price Tracker: Tax Holidays & LaptopsTrack back-to-school prices on Target, Walmart, Amazon and UK retailers from July, and time purchases around state sales-tax holidays (2026 dates listed).Compare Black Friday Prices Across Stores (Google Shopping)Compare Black Friday prices across stores with Google Shopping: every seller's offer for one product, condition and shipping, automated per keyword.Black Friday Price Tracker: Watch 2026 Prices DailyBuild a Black Friday 2026 price tracker for Amazon, Walmart and Target: a product watchlist, an October baseline, a schedule to Cyber Monday, and export.Boxing Day Sales Tracker UK 2026: Prices in GBPTrack UK Boxing Day and January sales in GBP: which retailers can be scraped, when to start, and how to compare an item's Black Friday and Boxing Day price.Christmas Price Tracker: When to Buy Gifts CheapestTrack Christmas gift prices from November, compare each price with its own season low, and buy before the 2026 shipping cut-offs. Scripts and schedule included.Competitor Price Monitoring: Scraper Setup and CostSet up competitor price monitoring with a scraper: a URL watchlist, a daily schedule, change-only rows, GTIN matching across stores, and cost per 1,000 checks.Cyber Monday Price Tracker: Hourly Checks for Nov 30Track Cyber Monday 2026 prices hour by hour from Sunday evening, compare each item with its Black Friday price, and see what hourly checks cost per run.Grocery Price Tracker: Build a Basket Index in PythonBuild a grocery price tracker: a fixed basket of Waitrose, Aldi or Walmart products, scraped on a schedule, with unit prices per litre and a basket index.Retail Sales Calendar 2026-2027: When to Track PricesEvery major sale from 11.11 to Prime Big Deal Days, Nov 2026 to Oct 2027, for the US, UK and Gulf, with when to start a price baseline and how often to check.MAP Monitoring: Find Minimum Advertised Price ViolationsHow to monitor MAP: list every seller of your products from Google Shopping and retailer pages, flag advertised prices below MAP, and keep timestamped evidence.Price Drop Alerts to Slack or Email with a ScraperGet price drop alerts in Slack or email: a scheduled change-only scrape, Apify's Slack and Gmail integrations, and a tested webhook handler with a threshold.Prime Day Price Tracker: ASINs, Big Deal Days, 2027Track Amazon prices around Prime Day and Prime Big Deal Days: past dates, how 2027 dates get announced, multi-country ASINs and what a scraper can see.Ramadan Sale Price Tracker 2027: Noon & Amazon UAETrack Ramadan and Eid sale prices on noon and Amazon UAE / Saudi: approximate 2027 dates, why sales start weeks before Ramadan, a schedule and a script.Scrape Product Prices with Python from JSON-LDScrape product prices with Python from schema.org JSON-LD: Offer, offer lists, AggregateOffer, @graph and variants, tested on live stores, and where it breaks.Scrape Product Reviews from Walmart, Target and AmazonScrape product reviews from Walmart, Target, Amazon and Shopify stores: inputs, sort order, real review fields, site limits and a Python complaint summary.Scrape Product Variants: Sizes, Colors and Price per SKUHow to scrape product variants (size, color, pack) with each variant's SKU, price and stock: the Shopify .js endpoint, a Python API run, and the traps.Fake Black Friday Deals: Check the Real DiscountAre Black Friday deals real? Compare the sale price with your own 30-day median and 90-day low, not the was price. Research findings and a Python checker.Apify E-commerce Scraping Tool Alternative: Same InputAn alternative to Apify's E-commerce Scraping Tool that takes the same input JSON: what stays, what's added, how sites and pricing differ, and a checklist.When Do Black Friday Prices Drop? The 2026 TimelineWhen do Black Friday prices drop? 2025's timeline from mid-November events to Cyber Monday, the 2026 calendar, and a script to find each item's low day.Walmart and Target Price Scraper: Search and Product PagesScrape Walmart and Target prices from search, category and product pages: real output, GTIN matching between the two, the was-price, stock, cost and caveats.White Friday & 11.11 Tracker: Noon and Amazon UAE PricesTrack 11.11, White Friday and Yellow Friday prices on Noon and Amazon UAE and Saudi Arabia in AED and SAR: 2025 sale windows, what works, real output and costs.AI Share of Voice: Measure It in ChatGPT and PerplexityWhat AI share of voice means, the formula, mentions vs citations, how many answers you need, and a free script to score your ChatGPT and Perplexity runs.How to Check if ChatGPT Mentions Your Brand (Free)A free, step-by-step way to check if ChatGPT and other AI assistants mention your brand: buyer prompts, clean chats, repeat runs, a simple log, and scoring.Get Completed Clinical Trial Results with the APIFind completed trials with posted results on ClinicalTrials.gov and read outcomes, p-values and adverse events from the v2 API resultsSection. Python included.ClinicalTrials.gov API: Look Up a Study by NCT NumberFetch one trial by NCT number with the ClinicalTrials.gov v2 API: the /studies/{nctId} endpoint, NCT format, fields, 404 vs 400, aliases and batch lookups.ClinicalTrials.gov API v2 Query Parameters: Full ReferenceEvery ClinicalTrials.gov API v2 /studies parameter: query.cond, query.term, filter.advanced AREA[] syntax, filter.geo, aggFilters, sort, fields, paging. Tested.ClinicalTrials.gov Classic API to v2: Migration GuideThe classic ClinicalTrials.gov API was retired on June 25, 2024. Map full_studies, study_fields, expr, min_rnk and fmt to API v2, or use the legacy XML shims.FPL API Endpoints: Complete Reference for 2026/27Every public Fantasy Premier League API endpoint, checked live in 2026/27: bootstrap-static, fixtures, live, entry, picks, leagues, with key fields and errors.FPL Fixture Difficulty (FDR) from the API in PythonRead FPL fixture difficulty from the free API: team_h_difficulty, team_a_difficulty, how they map to team strength, and a Python table of the next 5 gameweeks.FPL Live Gameweek Points from the API: Picks MathCompute any FPL team's gameweek score from event/{gw}/live and picks: multipliers, captaincy, chips, hits and auto-subs. Python checked against 300 teams.Find Your FPL Manager ID and Pull Your Team HistoryWhere to find your Fantasy Premier League manager (team) ID, then pull every gameweek, chip, transfer and past season for it from the free FPL API in Python.HN Algolia API Reference: Search, Tags, Filters, LimitsThe Hacker News Algolia API in one page: /search vs /search_by_date, every tag, numericFilters, hitsPerPage, the 1,000-hit cap, /items and /users. Tested live.Get All Comments for a Hacker News Story with the APIThree ways to download every comment on a Hacker News thread: the Algolia item tree, the search index and the Firebase API. Tested Python, and what each misses.Hacker News Firebase API vs Algolia: Which One to UseHN's official Firebase API (topstories, item, user) vs the Algolia search API: endpoints, rate limits, freshness and speed, tested side by side in Python.How AI Assistants Choose Which Brands and Sources to CiteWhat OpenAI, Perplexity, Anthropic and Google document about how their assistants pick sources: web search, crawlers, robots.txt. Plus a crawler check script.Search Hacker News by Date Range or Points (API)Find Hacker News stories between two dates or above a points threshold with the free Algolia API, and get past the 1,000-result cap. Tested Python and JS.SEC Company Facts API: companyfacts, concept & framesPull XBRL financials from data.sec.gov: companyfacts, companyconcept and frames URLs, the JSON shape, the fy and frame traps, and a CSV export. Tested Python.EDGAR Full-Text Search API: efts.sec.gov ParametersHow efts.sec.gov/LATEST/search-index works: q syntax, forms, ciks, startdt/enddt, sort, paging with from, the JSON response and its traps. Tested Python code.SEC EDGAR Rate Limit and User-Agent: Fair Access RulesSEC fair access in practice: the User-Agent format, the 10 requests/second cap across all machines, what the 403 means, how long a block lasts. Tested client.SEC EDGAR API: Submissions JSON and Ticker to CIKYes, EDGAR has a free API with no key: map a ticker to a CIK with company_tickers.json, then list every filing from data.sec.gov/submissions. Tested Python.How to Tell if a Token's Pool Is Getting Sandwiched Before You SwapBefore you swap, check whether a DEX pool is being sandwiched: read its trade feed, check its 24h sandwich rate and MEV fee share, then route your trade.Track AI Brand Mentions Over Time: Schedules and DriftMonitor how often ChatGPT and other AI assistants mention your brand over time: what to hold fixed, how often to run, what causes drift, how to compare runs.What Slippage Should I Set to Avoid Sandwich Attacks?Pick a slippage tolerance from pool depth and the pool's MEV risk: why loose settings invite sandwich bots, a rule-of-thumb table, and what to check first.ClinicalTrials.gov API v2 in Python: query.cond, Phase, CSVQuery the free ClinicalTrials.gov v2 API with query.cond, filter by phase (filter.phase returns HTTP 400; use filter.advanced) and export CSV. No key needed.Cointelegraph API: Get Crypto News as JSON Without a KeyCointelegraph has no public news API, but its site runs on a keyless GraphQL endpoint and RSS feeds. Pull articles, search results and full text as JSON.Price per sqm by Compound in New Cairo: Compute It with PythonStop trusting broker blogs: compute New Cairo apartment price per square meter by compound from live Property Finder Egypt listings, split ready vs off-plan.FPL API Price Change Data in Python: The Official 2026/27 FieldsRead FPL's own price-change predictions from the free bootstrap-static API in Python: price_change_percent, projections, likelihood, and the ±100 rule.LinkedIn X-Ray Search for Recently Updated Profiles (Past Week)Add Google's past-week filter to a LinkedIn X-ray search to surface recently re-indexed profiles, then dedupe by slug so each run shows only new people.Measure AI Share of Voice in Python: ChatGPT, Gemini, PerplexityMeasure your brand's AI share of voice in Python: one prompt set, several LLMs, mention counting that ignores 'I'm not familiar with X' answers.How to Check a DEX Pool's Sandwich Attack Rate (Python + API)Measure how often a Uniswap, Raydium or Aerodrome pool gets sandwiched: pull per-hour sandwichRate and MEV fee data from Codex getBars in Python.Track Peptide & GLP-1 Mentions on YouTube with Python (No API Key)Count YouTube videos mentioning semaglutide, tirzepatide or BPC-157 without an API key: parse ytInitialData, map brand names to compounds, export to CSV.Scrape Monthly vs Annual SaaS Prices From Pricing-Page JSON-LDGet both sides of a SaaS pricing page's monthly/annual toggle without a browser: read schema.org Offer JSON-LD in Python, then diff it on a schedule.Shopify products.json Pagination: 250 Limit and the 25,000 CapPaginate any Shopify store's public /products.json: limit=250, page numbers, the empty-page stop signal, and the HTTP 400 you hit at 25,000 products.threads_keyword_search: App Review, 2,200 Limit, WorkaroundWhat Meta's threads_keyword_search permission unlocks, why unapproved apps only search your own posts, the 2,200 queries/24h cap, and a Python monitor.PERM Disclosure Data FY2026 in Python: Green Card SponsorsFind green card sponsors by employer, city and job in the DOL's free PERM disclosure Excel files with Python: where the FY2026 files moved, key columns, traps.Scrape YouTube Shorts Data: Exact Views, Likes & CommentsScrape YouTube Shorts data with exact view, like and comment counts: where the official Data API falls short, how the internal API works, and runnable code.How to Extract Dubai Used Car Listings Data from Gulf MarketplacesA practical, honest guide to extracting Dubai used car listings data from DubiCars, YallaMotor, and Dubizzle Motors using JSON-LD and embedded __NEXT_DATA__.How to Build an H-1B Salary Database by Employer (with Python)A practical guide to querying H-1B salary data by employer and role, using the authoritative DOL OFLC LCA disclosure files with runnable Python.How to Extract Saudi Arabia Property Data from 4 Listing PortalsA practical guide to collecting Saudi Arabia property data from the major portals, including the REGA license trick that dedupes listings across platforms.How to Scrape a Telegram Channel Without Login (No API Key)A verified, runnable guide to scraping public Telegram channels without login, an API key or a phone number, using the t.me/s/ preview and plain HTTP.Understat xG Data Export: Pull Expected Goals with Python + CSVA practical guide to Understat xG data export: how the data is embedded in the page, a Python scraper that decodes it, and league and player xG to CSV.Facebook Ad Library Scraper: API Limits and the Real ApproachWhy the official Meta Ad Library API only covers political ads, and how to build a Facebook ad library scraper for commercial competitor ads instead.How to Build a LinkedIn Profile Scraper: The Honest Technical GuideA practical, honest guide to building a LinkedIn profile scraper: why there's no public API, how public pages embed JSON-LD, and the legal reality.SEC EDGAR API in Python: User-Agent Format & Rate LimitsUse the free SEC EDGAR API in Python: the User-Agent format the SEC asks for, the 10 requests/second limit, ticker-to-CIK and XBRL financials. Code included.How to Build a Threads Scraper for Meta Profiles and PostsA practical, honest guide to building a Threads scraper for Meta's threads.com — what loads cookie-free, what doesn't, and a runnable example.TikTok Ad Library API: Who Gets Access + a Working RequestHow to query the TikTok Ad Library: what the EU DSA library covers, who qualifies for the official Commercial Content API, and a working public request.App Store Top Charts API: Free JSON by Country, No KeyGet the App Store top free, paid and grossing charts as JSON for any country from the iTunes RSS feed. No key, works in the browser, with fetch() and curl code.The Hacker News Search API: Free, No-Key, and Surprisingly PowerfulA practical guide to the free, no-key Hacker News search API (hn.algolia.com): tags, numeric filters, relevance vs. date sorting, and runnable code.Read Company Hiring Signals From Public Job Board APIs (with code)Open job postings leak strategy. Learn to read company hiring signals from the public, no-auth Greenhouse API with a runnable JS classifier.How to Export Google Patents to CSV (Honest Guide to Every Real Path)A precise, no-hype guide to exporting Google Patents search results to CSV — the built-in 1,000-row cap, the BigQuery bulk path, and why browser scraping fails.How to Scrape Reddit Without the API (After the 2023 Price Changes)A precise, honest guide to scraping Reddit without the API: what still works in 2026, the login-wall and 403/CORS traps, and old.reddit.com.