API Overview
ScrapeGuys exposes real-time web data through a small set of REST endpoints. This page covers the concepts shared by every scraper — authentication, base URLs, the async job workflow, rate limits and errors. Each scraper then has its own reference page.
Flights
Google Flights + MakeMyTrip, each its own standalone guide. Structured, priced offers. Async jobs.
Hotels
Google Hotels, Agoda, Booking.com, MakeMyTrip, Expedia, Goibibo, Destination Search, Rate Shop, Cheapest Room and Hotel Details — ten standalone guides. Async jobs.
Public profile posts and single posts with comments, read logged out. Cursor pagination. Async jobs.
Base URLs
The scrapers are served by two independent API surfaces. Each has its own API keys (see Authentication).
| Surface | Base URL | Covers |
|---|---|---|
| Hotel API | https://api.scrapeguys.com/api/v1/scraper | Meta (Google Hotels) + OTA (Agoda, Booking.com, MakeMyTrip, Expedia, Goibibo) |
| Social Scraper API | https://api.scrapeguys.com/api/v1/social-scraper | Instagram (profile posts, post + comments) — same gsk_… keys, own scopes |
| Flight API | https://api.scrapeguys.com/api/v1/flight-scraper | Meta (Google Flights) + OTA (MakeMyTrip) |
api.scrapeguys.com with the host provided to you during onboarding if it differs. All examples on these docs use the host above.Authentication
Every request must include your API key in the X-API-Key header. Keys are 68-character strings; the Hotel API issues gsk_… keys and the Flight API issues fsk_… keys. The two are independent — a Hotel key does not authenticate the Flight API and vice-versa.
Scopes. Both key types carry an explicit scope list controlling which sources they may call. A gsk_… key can hold: gmaps (Google Hotels), ota:agoda (Agoda), ota:booking (Booking.com), ota:makemytrip (MakeMyTrip), ota:expedia (Expedia), ota:goibibo (Goibibo), destination:booking / destination:makemytrip (Destination Search), details:makemytrip / details:booking (Hotel Details), and instagram:profile_posts / instagram:post (Instagram). A fsk_… key can hold: google_flights (Google Flights) and/or makemytrip (MakeMyTrip Flights, both its app and desktop platforms), cleartrip (Cleartrip Flights) and/or paytm (Paytm Flights), per source. Which of these a given key actually carries depends on when it was issued and what was requested for it — see below. Calling an endpoint or source outside your key's scopes returns 403 with {"detail": "This API key does not have the '<scope>' scope."}. There is no wildcard scope — a source added later is never automatically granted to existing keys. See the OTA Directory (GET /otas/) for a live, per-key readout of which OTA scopes your gsk_… key actually holds.
gmaps, ota:agoda, ota:booking and ota:makemytrip; migrated Flight keys carry google_flights and makemytrip. ota:expedia, ota:goibibo and destination:booking came later and have to be added to a migrated key explicitly. The same goes for cleartrip (Cleartrip Flights) and paytm (Paytm Flights). Scoped (narrower) keys are opt-in from here: ask your ScrapeGuys contact if you want a new key limited to specific sources.destination:booking is not on any key yet, including full-access ones. It's a separate scope from ota:booking, not implied by it — pricing one hotel and enumerating an entire city are different products with unit costs orders of magnitude apart. Since it post-dates every key issued so far, it is never granted implicitly by the "existing keys keep working" migration above; ask your ScrapeGuys contact to add it explicitly before your first call to Destination Search.ota:goibibo is our newest scope and ships gated per environment. The same "existing keys keep working" migration does not reach it — it post-dates every key issued so far, same as destination:booking above. Ask your ScrapeGuys contact to add it, and see the Goibibo guide for the separate platform-wide gate that can still return 400 even once your key holds the scope.- 401 Unauthorized — missing, invalid, or deactivated key.
- 403 Forbidden — key is valid but lacks the required scope.
Request free-tier keys (1,000 API calls, no card needed) to get started.
The job workflow
Every scraping endpoint is asynchronous. You submit a job, receive a job_id and a poll_url, then poll until the job reaches a terminal state:
- Submit —
POST …/scrape/(orPOST …/ota-price/) returns202with{ job_id, status: "pending", poll_url }. - Poll —
GETthepoll_urlevery ~2 seconds.statusmovespending → processing → completed(orfailed). - Read — when
statusiscompleted, theresultobject holds the data.
/scrape/{job_id}/, Hotels → OTA jobs at /ota-price/{job_id}/, Hotels → Destination Search jobs at /destination-search/{job_id}/, Hotels → Rate Shop jobs at /rate-shop/{job_id}/, Hotels → Cheapest Room jobs at /cheapest-room/{job_id}/ (an OTA job can be read at either /ota-price/{job_id}/ or /cheapest-room/{job_id}/), Flights jobs (either source) at /scrape/{job_id}/ on the Flight API. Polling a job on another family's endpoint returns 404.Rate limits & quotas
Limits are enforced per API client and are configurable per plan. Defaults:
| Limit | Default | Window |
|---|---|---|
| Rate limit | 10 requests | per minute |
| Daily quota | 1,000 requests | per day (UTC) |
| Monthly quota | 20,000 requests | per billing cycle |
"Per billing cycle" is not always the calendar month: each client has a billing_cycle_day (1–28) the monthly quota resets on. New clients are anchored to their sign-up day; clients provisioned before billing cycles existed run on day 1 — the calendar month, unchanged. GET /usage/'s billing_period block reports your account's actual window.
Exceeding any limit returns HTTP 429 with a descriptive detail, e.g. "Rate limit exceeded (10/min). Try again shortly."
429 saying how many it needed and how many remain (e.g. "This batch needs 20 requests but only 10 remain in this minute (10/min). Nothing was submitted; retry next minute or split the batch."), no job is created and nothing is billed or counted. At the default 10 requests/minute, a batch of more than 10 items is always refused: split it.hotels × otas × dates. A 10-hotel, 3-OTA, 30-date shop is 900 units. The whole matrix is priced against your quota before any work starts and refused as a unit if it won't fit, with the arithmetic in the error — never billed for a partial matrix./usage/ and /otas/ are never counted.Errors
| Status | Meaning |
|---|---|
400 | Invalid input or a refused request — validation errors are reported per field; other refusals return {"error": "…"}. |
401 | Missing, invalid, or deactivated API key. |
403 | Key is valid but lacks the scope for the requested source/scraper. |
404 | Job not found (or not owned by your account). |
429 | Rate limit or quota exceeded. |
500 | Unexpected server error. |
200: a job ends completed with an empty result (e.g. no_results, sold_out: true) or failed with a human-readable error_message. Read each scraper's page for specifics.Usage endpoint
Check your consumption against your quotas at any time. Available on both surfaces:
/api/v1/scraper/usage/?days=30/api/v1/flight-scraper/usage/?days=30days is clamped into the 1–90 range (out-of-range values are adjusted, not rejected); a non-integer returns 400 with a message saying days must be an integer from 1 to 90.
billing_period is the window "monthly" means for your account — the monthly quota is summed over your billing cycle, which starts on cycle_day of each month and resets on resets_on, the date your monthly quota springs back to the full amount. start and end are both inclusive. Identical shape on both APIs, so a customer of both reads one contract.
in_flight (Hotel API only) is the work you have already submitted that has not finished and been billed yet. units is that count; available_today and available_this_month are your remaining quota after subtracting it, so they are the numbers that predict a Rate Shop 429. Submits are checked against billed usage alone, not in_flight: a single job is refused once billed usage reaches the quota, and a batch is refused unless billed usage plus the number of items fits within it. The block is left out if it cannot be computed.
lifetime.total_bandwidth_kb, per-day total_bandwidth_kb) exist on the Hotel API only. The Flight API usage response reports requests, successes, failures and durations — no bandwidth accounting. Bandwidth is the measured, compressed traffic the scrape moved through our proxies (response headers excluded); failed scrapes count what they spent before failing.Client dashboard
A browser dashboard of your own scraping activity — submitted / succeeded / Completed - No Data / in-flight counts as KPI cards, a time-series chart, and a PDF export. “Completed - No Data” counts the jobs whose status is failed — jobs that ended without a usable result. That can be a search the source site had no matching results for, or a transient error; poll the job and check its error_message, and retry if it is retryable. Searches that ran and simply found nothing end completed with an empty result and are counted as succeeded. Sign in with the portal username/email + password you were provisioned (ask your ScrapeGuys contact), or with any of your active Hotel API keys (gsk_…). Flight API keys (fsk_…) are not accepted here, and flight jobs are not counted. Either way the dashboard is scoped server-side to your data only, and access ends the moment your credentials are revoked on our side.
GET /usage/ above, so the two can disagree if you use Destination Search.https://api.scrapeguys.com/client/scrape-dashboard/For programmatic access, the same stats are available as JSON or PDF using just your API key — no sign-in needed:
/client/scrape-dashboard/data/?range=7d/client/scrape-dashboard/pdf/?range=7d| Param | Optional | Description |
|---|---|---|
range | yes | Quick window ending now, e.g. 10m, 30m, 7d. Wins over start/end when present. |
start, end | yes | YYYY-MM-DD or full ISO 8601. Ignored if range is set. |
granularity | yes | minute | hour | day | month. When omitted: auto-picked from a range window (minute ≤3h, hour ≤7d, day ≤90d, month beyond), and hour for a start/end window. |
scraper_type | yes | Narrows the counts to one engine: google, or an OTA slug — agoda, booking, makemytrip, expedia, goibibo (aliases such as mmt are accepted too; ota works as an alternate param name). per_scraper_type always covers every type regardless of this filter; series, totals and per_client are narrowed. |
With nothing supplied the window defaults to the last 7 days, every scraper type. Unrecognized values are ignored (not rejected) and reported back in a warnings array; the request still succeeds. A bad range, start or end (or start after end) falls back to a window sized by the granularity — the last 24 hours at the default hour.
per_client contains at most one row — your own — the dashboard endpoints never return another client's name, id or volume. per_scraper_type is your own data split by engine (Google vs each OTA you've used); share is each type's percentage of your total submissions in the window, and the split always covers every type you've used regardless of the scraper_type filter. A per-bucket breakdown of the same split is in each series row's by_scraper_type. bucket timestamps are UTC-offset ISO 8601; label is the same instant formatted for display in Asia/Kolkata.
/client/scrape-dashboard/pdf/ takes the same query params and streams a PDF (Content-Type: application/pdf) instead of JSON.
/usage/, the dashboard page, JSON and PDF endpoints are never counted against your rate limit or quota.Ready to dive in? Pick a guide: Google Flights · MakeMyTrip Flights · Cleartrip Flights · Paytm Flights · MakeMyTrip Bus · Google Hotels · Agoda · Booking.com · MakeMyTrip Hotels · Expedia · Goibibo · Destination Search · Rate Shop · Cheapest Room · Hotel Details · Instagram.