How to build a voice-of-customer dataset from public reviews

A voice-of-customer dataset is a join, and the join is where these projects fail: four rating scales, a dozen languages, three definitions of "review date", and per-source ceilings that make one brand look ten times more reviewed than another for reasons that have nothing to do with customers.

The short version

Choose sources by where your customers actually write. Trustpilot for the brand, the app stores for the product, Amazon and AliExpress per SKU, the OTAs per property, Indeed for the employer side.
Pull with filters, not everything. Text required, a rating band and a date cut cost nothing and remove the rows a language model cannot use.
Normalise scale, date and language on the way in. Score out of five, one date column with a stated meaning, and the language the text was actually written in.
Record the ceiling with the rows. Store how many reviews the source claims and how many you could read, per source, per entity. Without it, every comparison you make later is between samples of unknown size.

Start from what each source is actually good for

A voice-of-customer dataset is not "all the reviews". It is the reviews that answer a question, and the sources differ in what they can answer:

Source	What it is good for	Depth per entity
Trustpilot	Brand-level service and support complaints, across every language a global brand attracts	About 1,000 per company per run
App Store / Google Play	Product quality by release, with the app version on the row	Deep — tens of thousands for big apps
Amazon	What a shopper reads before buying, per SKU, at catalogue scale	8-13 per product
AliExpress	Variant-level product quality, with follow-up reviews weeks after delivery	The whole corpus
Booking / Agoda / TripAdvisor / Trip.com	Guest experience per property, with sub-ratings and traveller type	The whole corpus
Indeed	The employee side of the same brand, with five sub-ratings	Essentially all of it, via the worldwide view

The mix matters more than the volume. Ten thousand five-star Amazon reviews say less about a product than four hundred one-star Trustpilot reviews mentioning refunds.

The four normalisations to do on the way in

1. Scale. Booking, Agoda, HRS and Trip.com rate 0-10. TripAdvisor, Airbnb, Amazon, AliExpress, Trustpilot, Indeed, the App Store and Google Play rate 1-5. Hostelworld rates 0-100. Zoover rates 1-10. Kurzurlaub rates 1-6 where 6 is best. Every non-five-point source here also emits the score out of five; store both, and never average raw scores across sources.

2. Date. "Review date" means three different things. Trustpilot distinguishes the publication date from the experience date, and they are often weeks apart — for cohort work you want the second. Booking gives you check-in and check-out; TripAdvisor gives you a stay month. Amazon gives you the date string as printed. Pick one column with a stated meaning and keep the source's own dates alongside.

3. Language. Store the language the text was written in, and whether what you have is a machine translation. Sources differ: Airbnb and AliExpress add their own translation in a second field; TripAdvisor serves translations as separate rows per language domain; Hostelworld only ever serves one machine-translated English set, which is why it offers no language filter rather than one that does nothing.

4. Ceiling. For each source and entity, store what the source claims and what you got. A TripAdvisor hotel row reports its all-language review count against what the language site listed; an Amazon product row reports its ratings count against the reviews visible on the page; a Trustpilot status row flags a walk that stopped at the wall rather than at the end. Without those numbers, a chart comparing "review volume by brand" is comparing your collection method with itself.

A working collector

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

SOURCES = [
    ("kestrel/trustpilot-reviews-scraper",
     {"companyDomains": ["www.example.com"], "maxReviewsPerCompany": 1000,
      "languages": "all", "requireText": True}),
    ("kestrel/app-store-reviews-scraper",
     {"appIds": ["310633997"], "countries": ["us", "gb", "de"],
      "maxReviewsPerApp": 500, "requireText": True}),
    ("kestrel/google-play-reviews-scraper",
     {"appIds": ["com.example.app"], "languages": ["en", "de"], "countries": ["us", "de"],
      "sort": "most_recent", "maxReviewsPerApp": 500, "requireText": True}),
    ("kestrel/amazon-reviews-scraper",
     {"asins": ["B0BSHF7WHW"], "requireText": True}),
]

rows, coverage = [], []
for actor, run_input in SOURCES:
    run = client.actor(actor).call(run_input=run_input)
    src = actor.split("/")[1]
    for r in client.dataset(run["defaultDatasetId"]).iterate_items():
        if r["type"] == "review":
            rows.append({
                "source": src,
                "entity": r.get("app_id") or r.get("asin") or r.get("company_domain"),
                "review_id": r["review_id"],
                "score_5": r.get("rating_5") or r["rating"],
                "native_score": r["rating"],
                "date": r.get("review_day") or r.get("review_date") or r.get("date_text"),
                "language": r.get("language"),
                "text": r.get("text"),
                "reply": r.get("reply_text") or r.get("developer_response") or r.get("response"),
            })
        elif r["type"] == "status":
            coverage.append({"source": src, "delivered": r.get("reviews"),
                             "claimed": r.get("total"), "capped": r.get("capped")})

seen, deduped = set(), []
for r in rows:
    key = (r["source"], r["review_id"])
    if key not in seen:
        seen.add(key); deduped.append(r)
print(len(deduped), "unique reviews;", coverage)

The coverage list is not an afterthought. It is the part that makes the dataset defensible six months later, when someone asks why one brand has 200 rows and another has 4,000.

What it costs to assemble

Per delivered row, and the row prices differ by an order of magnitude: Trustpilot $0.0007, Indeed and AliExpress $0.002, Hostelworld, HRS and Kurzurlaub $0.003, the app stores, Trip.com and Zoover $0.004, the OTAs and Amazon $0.005.

A realistic first corpus — 1,000 Trustpilot reviews, 1,500 app reviews across two stores and three storefronts, 13 Amazon reviews each on 200 SKUs — is roughly $0.70 + $6.00 + $13.00, so under $20. The text filter is what keeps it there: rating-only rows are worthless to a model and are dropped before billing.

Run it once to build the base, then run it daily with a date cut so you only pay for what is new. That is the difference between a corpus that ages and a living one.

Preparing it for a model

Three habits that save re-runs:

Keep the reply. Whether a complaint was answered is a feature, and most datasets throw it away. Booking, TripAdvisor, Agoda, Trip.com, HRS, Hostelworld, Zoover, Trustpilot, Indeed, the App Store and Google Play all publish it.
Keep the sub-ratings in their own columns. "Rooms 2, service 5" tells a topic model where to look before it reads a word. Do not average them into the overall score.
Tell the model what it is reading. Machine-translated text, score-only rows and a 0-10 scale flattened to five are all things a prompt should mention, because a model cannot see the provenance you dropped.

Run it

Cheapest starting point: Trustpilot Reviews at $0.0007 a row. Product side: App Store, Google Play, Amazon, AliExpress. Employer side: Indeed. Hospitality: the review scraper comparison lists all thirteen with their ceilings. Related: which review sites can be scraped without a browser before you plan a source you cannot reach.

FAQ

Which columns should the unified table have?

Source, entity id, review id, score out of five, native score and its scale, review date, experience or stay date where the source has one, language, whether the text is a machine translation, the text, the reply, and a fetched-at timestamp. Everything else stays in a per-source column set, because sub-ratings genuinely differ: six aspects on TripAdvisor hotels, four on TripAdvisor restaurants and Trip.com, seven on Hostelworld, five on Indeed.

How do I avoid double-counting a review?

Key on source plus the source's own review id, and be aware of the two ways a duplicate appears legitimately: translations (a TripAdvisor review read on two language domains arrives twice, once machine-translated) and locale overlap (Apple and Google both serve some reviews under more than one storefront or locale). Both are solvable at the id level, which is why every row here carries the source's native id rather than a hash.

Should I translate everything to English before analysis?

Keep the original and add a translation as a separate column; never overwrite. Some sources hand you their own translation (Airbnb, AliExpress), TripAdvisor serves machine translations as separate rows, and Hostelworld only ever serves one machine-translated English set. Mixing an original and a translation in one column makes language a hidden variable in every downstream model.

How large a corpus can I actually assemble?

It depends entirely on the source ceilings, which differ by two orders of magnitude: an entire Booking, Agoda, TripAdvisor or Trip.com property; about 1,000 per Trustpilot company per run; 8-13 per Amazon product; four comments per Despegar hotel. Plan the corpus around the ceilings, not around the entity count.

Run Trustpilot Reviews Scraper on Apify →

Other actors used on this page: apify.com/kestrel/amazon-reviews-scraper, apify.com/kestrel/app-store-reviews-scraper, apify.com/kestrel/google-play-reviews-scraper, apify.com/kestrel/indeed-company-reviews, apify.com/kestrel/booking-reviews-scraper, apify.com/kestrel/aliexpress-reviews-scraper.