Scraping Google Maps With Python: What Breaks and What Works

Type "scrape Google Maps with Python" into a search box and you get a hundred tutorials with roughly the same shape. Import requests, import BeautifulSoup, find the div, loop, write CSV. Copy the code, run it, and get an empty list.

The code is not wrong exactly. It was written for a web that Google Maps stopped being part of years ago. This post walks the three approaches that people actually try, explains where each one ends, and finishes with the version that keeps running after you stop looking at it.

Approach One: requests and BeautifulSoup

This is the one in most tutorials, and it fails immediately.

requests.get() downloads the HTML the server sent. Google Maps sends a small shell and then builds the entire result list in the browser with JavaScript. The businesses are not in the document you downloaded. They do not exist until a browser engine has run the page.

So BeautifulSoup parses a page with no listings in it, your loop runs zero times, and you get an empty CSV with correct headers. Nothing in the traceback tells you why, which is why people spend an evening adjusting selectors that were never going to match anything.

If a tutorial shows this working, check the date, then check whether the output in the screenshot is real.

Approach Two: Selenium or Playwright

This one works. That is the trap.

A headless browser renders the page properly, so the listings really are there and really can be read. You launch Chrome, search, scroll the result panel until it stops adding rows, and pull the fields out. On the first afternoon it feels like you solved it.

Here is what the next few months look like.

The selectors move. Google ships interface changes constantly, and the generated class names are not a public contract. Your script does not error when they change. It returns zero rows, or worse, it returns rows with one column silently empty.

Scrolling is the whole job. New entries load as you scroll the panel. Too slow and a large export takes an hour. Too fast and rows fail to load, which produces gaps rather than errors. Tuning that sleep value becomes a permanent part of your life.

The list runs out. Google stops feeding new results after a set number per search. That number is nowhere near the count of businesses in a real city, so covering a metro means slicing it into neighbourhoods and categories yourself and merging the files afterwards.

Automated traffic gets treated as automated traffic. Repeated, fast, headless requests from one address get slowed, challenged or blocked. Working around that is a second project with its own running cost, and it is the point where a data-collection script turns into something you should think carefully about.

Nobody maintains it. The script belongs to whoever wrote it. When it breaks in month four, that person is busy, and the list nobody has refreshed since March is quietly wrong.

None of that means a headless browser is the wrong tool in general. It means Google Maps at list-building scale is an unusually bad target for one.

Approach Three: the Places API

Google's own Places API is stable, documented and does not break when the interface changes. For a store locator or an address autocomplete inside your product, it is the correct answer and nothing else comes close.

Bulk prospect lists are a different job, and the pricing model reflects that. You pay per call, and the price tier is set by the most expensive field you ask for, so adding a phone number or a rating to a request can move the whole call into a higher tier. There are also storage terms limiting how long you may keep what comes back.

We worked through the field tiers and the arithmetic in Google Places API pricing, and the licensing question in Places API versus scraping. The short version: it is an API for showing places to a user, not for filling a CRM.

What Actually Stays Working

The version that survives is the one where you do not maintain the collection at all. You call an API that already runs it, and your Python does three things: submit a job, wait, read the results.

Everything below runs against the BasedOnBusiness REST API, using the requests library. Create a key in Settings, under API and Webhooks. Keys start with bdb_live_.

Submit the job

import os
import requests

BASE = "https://www.basedonb.com/api/v1"
KEY = os.environ["BDB_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}"}

response = requests.post(
    f"{BASE}/scrapes",
    headers={**HEADERS, "Idempotency-Key": "dentists-istanbul-2026-08"},
    json={
        "query": "dentist",
        "country": "TR",
        "state": "TR.34",
        "city": "Istanbul",
        "target_leads": 200,
    },
    timeout=30,
)
response.raise_for_status()
job = response.json()
print(job["id"], job["status"])

Two details worth knowing. state uses the GeoNames dotted format, so TR.34, US.CA, DE.BE. Countries with no state subdivision take only country. If you are unsure what a country expects, the geodata endpoints under /geodata/countries, /geodata/states and /geodata/cities return the accepted values.

The Idempotency-Key header matters more than it looks. If your request times out and you retry, the key stops a second job being created and your credits being spent twice.

Wait for it

import time

def wait_for(job_id, poll_seconds=5, timeout_seconds=1800):
    deadline = time.time() + timeout_seconds
    while time.time() < deadline:
        job = requests.get(f"{BASE}/scrapes/{job_id}", headers=HEADERS, timeout=30).json()
        if job["status"] == "done":
            return job
        if job["status"] in ("failed", "cancelled"):
            raise RuntimeError(f"Job ended as {job['status']}")
        print(f"{job['status']} {job.get('progress', 0):.0%} · {job['leads_found']} found")
        time.sleep(poll_seconds)
    raise TimeoutError("Job did not finish in time")

wait_for(job["id"])

A five second interval is polite and well inside the rate limit, which is 100 requests per minute per key. Do not poll every 200 milliseconds because the loop allows it.

Page through the results

Results come back a page at a time with a cursor. The page object tells you whether to keep going.

def fetch_all(job_id, page_size=500):
    rows, cursor = [], None
    while True:
        params = {"limit": page_size}
        if cursor:
            params["cursor"] = cursor
        page = requests.get(
            f"{BASE}/scrapes/{job_id}/results",
            headers=HEADERS,
            params=params,
            timeout=60,
        ).json()
        rows.extend(page["results"])
        if not page["page"]["has_more"]:
            return rows
        cursor = page["page"]["next_cursor"]

limit accepts up to 500. Each row carries place_id, title, category, address, phone, website, rating, reviews_count, latitude, longitude, price_level and business_status, plus enrich_status and email_enrich_status telling you whether the website enrichment finished for that record.

Write the file

import pandas as pd

frame = pd.DataFrame(fetch_all(job["id"]))
with_site = frame[frame["website"].notna()]
print(f"{len(frame)} records, {len(with_site)} with a website")
frame.to_csv("dentists-istanbul.csv", index=False)

That is the whole program. No selectors, no scroll timing, no browser, and nothing that changes when Google redesigns a panel.

Stop polling, use a webhook

Polling is fine for a script you run yourself. Inside a pipeline you want the job to tell you.

Register a webhook endpoint and a scrape.done event arrives when the job finishes, carrying an X-Webhook-Signature header you verify against the signing secret shown once at creation. Failed deliveries retry after 1 minute, 5 minutes, 30 minutes and 2 hours. Store the X-Webhook-Event-Id and ignore events you have already handled, because a retry can arrive after you processed the first delivery.

A Note on the Legal Part

Choosing Python does not change any of the rules. Public business information is broadly collectable in most jurisdictions, personal data needs a lawful basis, and Google's terms restrict automated collection from its own properties regardless of the tool. Outreach follows separate laws again, and those apply to the list no matter where it came from.

The practical difference between writing your own scraper and calling a provider is who carries that responsibility. Is scraping Google Maps legal covers the distinctions in more detail.

Which One to Build

If you are learning, write the Selenium version. It is a genuinely good exercise and you will understand every tool you buy afterwards.

If something depends on the output, do not. The maintenance cost is real, it lands at inconvenient times, and it is invisible in the estimate you gave your team.

A rough test: if the list feeds anything on a schedule, or anyone other than you reads it, you want an API. If it is a one-off for your own research, anything works, including an export tool with no code at all.

Try It on Real Data

New accounts get 50 one-time export credits with no card, where one credit is one business record. That covers the scripts above end to end: submit a small job, poll it, page the results and open the CSV.

Businesses with a website also get their published emails, social profiles and technology signals attached at no extra credit cost, with the source page recorded, so the records arrive ready for whatever comes next in your pipeline. The full API reference has every endpoint, error code and webhook payload.