kestrel actors docs

kestrel actors › Guides

How to scrape Trustpilot reviews past the 200-review wall

Trustpilot gives an anonymous visitor ten pages of any filtered view — 200 reviews — and then redirects page eleven to the login screen. That is why so many Trustpilot datasets stop at exactly 200 rows and call a 125,000-review brand done. The wall is per view, which is the whole trick.

The short version

  1. Solve the challenge once, not per page. Trustpilot is behind an AWS WAF challenge in challenge mode: every path needs a JS-computed token. Solve it once in a browser, then fetch pages with a plain client on that token.
  2. Refresh the token before five minutes. The issued token stops being honoured at about five minutes. Refreshing every four keeps a long run on one browser solve.
  3. Split the request across the five star bands. Each star rating is its own filtered view with its own 200-review allowance, and the bands are disjoint — a review is one star or four, never both.
  4. Take each band in proportion to the histogram. Drawing from each band in the shape of the company's own rating histogram keeps the sample representative instead of over-weighting whichever band you read first.

The wall, stated plainly

An anonymous visitor to Trustpilot gets 200 reviews per filtered view — ten pages. Request page eleven of any view and the site redirects you to its login screen.

This is the number most Trustpilot listings leave out, and it explains a pattern you may have noticed in scraped datasets: a company with 125,000 reviews yields exactly 200 rows, with no error and no warning, and the pipeline downstream quietly treats that as the company's whole review history.

The part that gets you past it honestly

The wall is applied per filtered view, and the star rating is a filter that runs on Trustpilot's own servers via the stars= parameter.

There are five star ratings. They are disjoint — a review is one star or four, never both — and each is its own view with its own 200-review allowance. Read all five and you have up to 1,000 reviews per company per run, none of them duplicated, without a login and without pretending to be a signed-in user.

Two details make the difference between a clever trick and a usable sample:

Past 1,000, narrow the view: a date range or a single language gives every band a fresh allowance. Twelve monthly windows across five bands is a large corpus assembled legitimately, one view at a time.

The token that lives five minutes

Trustpilot's AWS WAF runs in challenge mode. Every URL tested returned the same 991-byte Verifying Connection interstitial with HTTP 403: the review page, ?page=2, the home page, /categories, the business-unit finder, and even the Next.js JSON data route. That was on three datacenter sessions, three residential sessions, over HTTP/1.1, with Chrome, Firefox and Safari TLS fingerprints, and direct from a home IP with no proxy at all. Only robots.txt is outside the WAF.

So the token has to be computed by a real JS runtime. The efficient shape is to solve it once — five to twenty seconds in a headless browser — then fetch every page with a plain HTTP client carrying that token, from the review page's own JSON data route, which is less than half the bytes of the HTML. The one thing to know: Trustpilot stops honouring the token at about five minutes, so refresh it every four. Get that wrong and a long run dies two hundred rows in, which looks exactly like the paging wall and is not.

If a run is very short, that one browser solve is a real share of its cost. Batch your companies rather than running one company per run.

Working code

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("kestrel/trustpilot-reviews-scraper").call(run_input={
    "companyDomains": ["www.booking.com"],
    "maxReviewsPerCompany": 1000,   # deep paging splits this across the five star bands
    "languages": "all",             # the public page shows English only
    "sort": "most_recent",
})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
company = next(r for r in rows if r["type"] == "company")
status = next(r for r in rows if r["type"] == "status")
print(company["trust_score"], company["review_count"], "lifetime reviews")
print(len([r for r in rows if r["type"] == "review"]), "delivered; capped:", status["capped"])

A daily complaints feed is the cheapest useful shape, because the star filter runs on Trustpilot's servers — nothing else is ever downloaded:

run = client.actor("kestrel/trustpilot-reviews-scraper").call(run_input={
    "companyDomains": ["www.monzo.com", "ryanair.com"],
    "minRating": 1, "maxRating": 2, "sinceDate": "2 days", "sort": "most_recent",
})

At $0.0007 per review — the cheapest row in the suite — watching a hundred brands for one- and two-star reviews costs less per month than one lunch.

The language default that changes your numbers

Trustpilot's public page shows English only. For a global brand that hides most of the corpus: one travel brand returned forty languages across 120,000 reviews. The free company row carries review_languages with a count per language, so you can see the distribution before committing to a run.

Set the language filter to all languages unless you have a reason not to, and be aware that a "Trustpilot average rating" computed from the English default is an average of the English-speaking subset.

Two dates, and the one you want

Every row carries review_date (when it was published) and experience_date (when the experience happened). They are often weeks apart. For cohort analysis — "how did customers who bought in March rate us" — the second one is the one you want, and it is the one most datasets do not carry.

The rest of the row: rating, title, text, language, reviewer name, country and lifetime review count, the verified badge and verification level, how the review was collected (organic, invitation, business-generated link), likes, the company's reply with its date, and a permalink.

The honest limitations

Run it

Trustpilot Reviews Scraper — $0.0007 per delivered review, with the company row, status rows, filtered reviews, empty profiles and unknown domains free. Related: which review sites can be scraped without a browser, and how to build a voice-of-customer dataset that puts Trustpilot next to Amazon and the app stores.

FAQ

Is the 200-review limit a rate limit?

No. It is a paging wall for anonymous visitors: page eleven of any filtered view redirects to the login screen. Waiting, slowing down or rotating IPs does not move it, because it is not about your traffic — it is about not being signed in.

How do I get more than 1,000 reviews from one company?

Narrow the view and run again. Each combination of filters has its own allowance, so a date range or a single language is a fresh 200 per star band. Twelve monthly windows across five bands is a legitimate route to a large corpus, one request set at a time.

Why does the run need a browser at all?

Trustpilot's AWS WAF is in challenge mode, unlike Booking's, which only fingerprints the TLS stack on page routes. Every URL tried — the review page, page 2, the home page, the categories index, even the Next.js data route — returned the same 991-byte interstitial, on datacenter, on residential, over HTTP/1.1, and from a clean home IP with no proxy at all. No header or pool change passes it; a JS-computed token is required.

Does the widget endpoint work instead?

Not for paging. widget.trustpilot.com sits outside the WAF but needs the business-unit id and a TrustBox template the business has enabled, and returns a handful of reviews per call. It is a widget feed, not a corpus.

Run Trustpilot Reviews Scraper on Apify →

More guides