In this tutorial we'll write one Python script (~70 lines) that:

1. runs the same Glassdoor search in four cities in four countries, in parallel,
2. saves every posting to a CSV,
3. uses pandas to compare yearly pay ranges, how many postings list a salary at all, and which companies are hiring.

The example is "data analyst" in New York, London, Berlin and Bengaluru. Swap in any role and any locations Glassdoor covers.

Disclosure: the scraper used here is an Apify Actor published by Atlas Data, the author of this post: atlasdata/glassdoor-jobs-scraper-reliable. It is pay-per-result ($0.30 per 1,000 jobs plus a small per-run start fee), and the script sets a cost cap on every run. You could also collect the data another way; the pandas part applies either way.

Setup

pip install apify-client pandas   # tested with apify-client 3.4 and pandas 3.1
export APIFY_TOKEN=...             # https://console.apify.com/settings/integrations

Glassdoor blocks datacenter IPs, so the Actor connects through Apify residential proxies. They are included in the per-job price, so there is nothing to configure.

What one result looks like

The input mirrors Glassdoor's own search: keywords (each keyword is a separate search), location (a city, state or country as Glassdoor knows it), datePosted (any, 1, 3, 7, 14, 30 days), jobType, remoteOnly, easyApplyOnly, minRating, minSalary, seniority, radius, sort and maxResults. includeDetails (on by default) adds the full description and the company profile. We don't need those here, so we turn it off.

Each posting is a flat record. These are the fields we'll use, with values from the test run below:

Two of these matter a lot for a fair comparison:

• salarySource. Some postings carry the employer's own range. For others Glassdoor shows its own estimate (the 10th to 90th percentile for similar jobs). Both are useful, but they are not the same thing, so we count them separately.
• salaryAnnualMin / salaryAnnualMax. Hourly and monthly ranges are converted to a yearly amount, so a London contract paid by the hour and a Berlin internship paid by the month can sit in the same column.

1. The searches

KEYWORD = "data analyst"
LOCATIONS = ["New York, NY", "London", "Berlin", "Bengaluru"]
COMMON = {"datePosted": "30", "maxResults": 30, "includeDetails": False}

datePosted: "30" keeps the comparison to postings from the last month. 30 jobs per city is one page of Glassdoor results. That's enough to test the script; raise maxResults for a real analysis (see the cost section at the end).

Locations are free text and go through Glassdoor's location search. Check searchLocation in the results (or the run's OUTPUT record) to see what was matched. In our run all four resolved as expected: New York, NY (US), London, England (UK), Berlin (Germany) and Bengaluru (India). For ambiguous names, add the state or country ("Springfield, IL").

2. Run the four searches in parallel

def fetch(location: str) -> list[dict]:
    run = client.actor("atlasdata/glassdoor-jobs-scraper-reliable").call(
        run_input={**COMMON, "keywords": [KEYWORD], "location": location},
        max_total_charge_usd=Decimal("0.05"),
        logger=None,
    )
    if run is None or run.status != "SUCCEEDED":
        raise RuntimeError(f"{location}: run failed: {run and run.status}")
    items = list(client.dataset(run.default_dataset_id).iterate_items())
    for item in items:
        item["market"] = location  # the location we asked for
    return items

One run per location, because location is a single value per run. The runs are independent, so a ThreadPoolExecutor starts all four at once. In the run shown below each one finished in 3 to 7 seconds.

max_total_charge_usd is a hard cap on what a single run can cost. The Actor stops cleanly when it reaches it.

3. Clean the salaries before comparing them

Before looking at medians, look at the rows. Our first run turned up two things that would have skewed the numbers:

• Internships. Four of the Berlin postings were internships paid 2,470 EUR a month, which becomes 29,640 a year. That's correct, but it isn't a data analyst salary.
• A typo in a posting. One Berlin employer posted "70 – 90" a year for a senior role. It almost certainly means 70k to 90k, but the data says 70 to 90.

Both are easy to filter:

NOT_COMPARABLE = r"\bintern(?:ship)?\b|werkstudent|working student"

paid = df.dropna(subset=["salaryAnnualMin", "salaryAnnualMax"])
paid = paid[~paid["title"].str.contains(NOT_COMPARABLE, case=False)]  # internships
paid = paid[paid["salaryAnnualMin"] >= 1000]  # "70-90 a year" is a typo for 70k-90k
paid = paid.assign(mid=(paid["salaryAnnualMin"] + paid["salaryAnnualMax"]) / 2)

The \b word boundaries matter: a plain intern would also drop "International" and "Internal Audit" roles. The >= 1000 check is deliberately loose because it has to work for USD, GBP, EUR and INR at the same time.

We use the midpoint of each range as one number per posting. It's a simplification (a 50k–110k range and an 80k range get the same midpoint), so the summary also shows the 25th and 75th percentiles.

4. One row per market

rows.append({
    "market": market,
    "jobs": len(g),
    "salaries": len(p),
    "employer": int((p["salarySource"] == "employer").sum()),
    "currency": p["salaryCurrency"].mode().iat[0] if len(p) else None,
    "p25": p["mid"].quantile(0.25),
    "median": p["mid"].median(),
    "p75": p["mid"].quantile(0.75),
    "rating": g["companyRating"].mean(),
})

salaries is the number of postings that survived the cleaning step, and employer is how many of those were posted by the employer rather than estimated by Glassdoor. Always print these next to the median. A median from 5 postings shouldn't be read the same way as one from 30.

The amounts stay in local currency. If you want one currency, convert with an exchange rate from a date you choose. Keep in mind that salaries in different countries also differ in taxes, benefits and cost of living, so a converted number is still not a like-for-like comparison.

Running it

Real output from our test run (October 2026):

$ python glassdoor_salaries.py
120 jobs saved to glassdoor_jobs.csv

Yearly pay for "data analyst" (midpoint of each range, local currency):
              jobs  salaries  employer currency     p25  median     p75 rating
market
New York, NY    30        30        29      USD  83,809 102,542 117,750    3.7
London          30        28        15      GBP  45,169  48,325  57,547    3.8
Berlin          30         5         5      EUR  70,000  74,500  80,000    3.7
Bengaluru       30        22         0      INR 614,250 686,500 823,625    3.7

Top hiring companies:
  New York, NY: Intuit (2), Capgemini (2), Horizon Media, Inc. (2)
  London: Elexon (2), TELUS Digital (1), Leyland SDM (1)
  Berlin: Institutional Shareholder Services (4), Zalando (3), MMM Consulting GmbH (1)
  Bengaluru: Société Générale (2), Thales (1), Myntra (1)

With only 30 postings per city this is a test of the method, not a salary report. Still, the coverage columns are interesting on their own:

• New York: 29 of 30 postings had a range posted by the employer. New York requires employers to include pay ranges in job ads, and it shows.
• London: 28 of 30 had a salary, about half from employers and half Glassdoor estimates.
• Berlin: only 10 of 30 postings had any salary. After removing the internships and the typo, 5 were left. Don't read anything into that median.
• Bengaluru: 22 of 30 had a salary, and every one of them was a Glassdoor estimate. That is "what Glassdoor estimates similar jobs pay", not what these employers offered.

"Top hiring companies" is also thin at this sample size: most companies appear once or twice. With a few hundred postings per city it becomes more meaningful, and you can group by companyName to see each company's rating and salary range together.

Full script

"""Compare advertised salaries and top hiring companies across countries with Glassdoor job data."""
import os
from concurrent.futures import ThreadPoolExecutor
from decimal import Decimal

import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

KEYWORD = "data analyst"
LOCATIONS = ["New York, NY", "London", "Berlin", "Bengaluru"]
COMMON = {"datePosted": "30", "maxResults": 30, "includeDetails": False}
NOT_COMPARABLE = r"\bintern(?:ship)?\b|werkstudent|working student"


def fetch(location: str) -> list[dict]:
    run = client.actor("atlasdata/glassdoor-jobs-scraper-reliable").call(
        run_input={**COMMON, "keywords": [KEYWORD], "location": location},
        max_total_charge_usd=Decimal("0.05"),
        logger=None,
    )
    if run is None or run.status != "SUCCEEDED":
        raise RuntimeError(f"{location}: run failed: {run and run.status}")
    items = list(client.dataset(run.default_dataset_id).iterate_items())
    for item in items:
        item["market"] = location  # the location we asked for
    return items


def summarize(df: pd.DataFrame) -> pd.DataFrame:
    paid = df.dropna(subset=["salaryAnnualMin", "salaryAnnualMax"])
    paid = paid[~paid["title"].str.contains(NOT_COMPARABLE, case=False)]  # internships
    paid = paid[paid["salaryAnnualMin"] >= 1000]  # "70-90 a year" is a typo for 70k-90k
    paid = paid.assign(mid=(paid["salaryAnnualMin"] + paid["salaryAnnualMax"]) / 2)

    rows = []
    for market, g in df.groupby("market", sort=False):
        p = paid[paid["market"] == market]
        rows.append({
            "market": market,
            "jobs": len(g),
            "salaries": len(p),
            "employer": int((p["salarySource"] == "employer").sum()),
            "currency": p["salaryCurrency"].mode().iat[0] if len(p) else None,
            "p25": p["mid"].quantile(0.25),
            "median": p["mid"].median(),
            "p75": p["mid"].quantile(0.75),
            "rating": g["companyRating"].mean(),
        })
    return pd.DataFrame(rows).set_index("market")


def main() -> None:
    with ThreadPoolExecutor(max_workers=len(LOCATIONS)) as pool:
        batches = list(pool.map(fetch, LOCATIONS))
    df = pd.DataFrame([job for batch in batches for job in batch])
    df.to_csv("glassdoor_jobs.csv", index=False)
    print(f"{len(df)} jobs saved to glassdoor_jobs.csv\n")

    pd.options.display.float_format = "{:,.0f}".format
    summary = summarize(df)
    summary["rating"] = summary["rating"].round(1).astype(str)
    print(f'Yearly pay for "{KEYWORD}" (midpoint of each range, local currency):')
    print(summary.to_string(), "\n")

    print("Top hiring companies:")
    for market, g in df.groupby("market", sort=False):
        top = g["companyName"].value_counts().head(3)
        print(f"  {market}: " + ", ".join(f"{name} ({n})" for name, n in top.items()))


if __name__ == "__main__":
    main()

What it costs

The run above saved 120 jobs in four runs: 120 × $0.0003 plus four run starts at $0.0005 is $0.038. Description and company details are included at the same price if you turn includeDetails back on. For a sturdier comparison, 300 postings per city across the same four cities would be 1,200 jobs, or about $0.36. Duplicates within a run are not charged.

Ideas for extending it

• Employer-posted only: filter on salarySource == "employer" and compare that median with the Glassdoor estimates for the same market.
• More roles: put several titles in keywords (each one is its own search) and pivot by searchQuery × market.
• Company profiles: with includeDetails: True you get companySize, companyRevenue, companyWebsite and companyHeadquarters, which turns "who is hiring" into a lead list.
• Track it over time: run it weekly with datePosted: "7" and append to the CSV or a database. The Google Jobs tracker post below shows a SQLite pattern for first-seen / last-seen dates.

Glassdoor is one job board. Google Jobs Scraper, our sister Actor, covers the Google Jobs panel, which pulls postings from LinkedIn, company career sites and many other boards. We used it in Build a small job-market tracker with Google Jobs, Python and SQLite.

To ask an AI assistant instead ("what do data analyst jobs in London pay right now?"), the Actor also works as an MCP tool through the hosted Apify MCP server. The client configs for Claude, Cursor and VS Code are in atlas-data-mcp; add atlasdata/glassdoor-jobs-scraper-reliable to the tools= list in the endpoint URL. The Actor's Store page documents every input and output field.

This post sets up a Google Trends monitor that:

• runs every morning without a machine of yours being on,
• fetches interest over time and related queries for a list of terms,
• flags terms that are up against their weekly average, plus queries Google marks Breakout,
• and tells you explicitly when something didn't work, instead of returning nothing.

Disclosure: the scraper is an Apify Actor published by Atlas Data, the author of this post: atlasdata/google-trends-scraper-reliable. It is pay-per-result, and the task below caps the cost of every run. Docs and examples in other languages are in the GitHub repo.

Why Trends scrapers get blocked

Google Trends has no official API. The web app calls internal endpoints that are rate-limited per IP address and per cookie session, and the limits are tight: a handful of quick requests from one IP is often enough to trigger 429s, and datacenter IPs are throttled sooner. The usual DIY fixes are time.sleep() and retries. They help, but on a schedule they still fail eventually, because the IP is the same every run.

The Actor does what you would end up building yourself:

• A fresh cookie session per attempt, so a throttled session isn't reused.
• Exponential backoff with a new IP on every retry. maxRetries defaults to 8 (range 0–20).
• Automatic proxy fallback (proxyFallback, on by default). If the current connection keeps getting blocked, it moves from your proxy to Apify datacenter, then to Apify residential if your plan includes it, then to a direct connection.
• A per-term status report in the run's OUTPUT record (SUCCEEDED / PARTIAL / NO_DATA / FAILED), so an empty result is never ambiguous.

None of this is magic. Google can still withhold some data. For example, related topics are currently not returned to any automated session, and city-level regions are often missing. When that happens, the summary says so.

Setup

pip install apify-client   # tested with apify-client 3.4
export APIFY_TOKEN=...      # https://console.apify.com/settings/integrations

1. Define what to watch

ACTOR = "atlasdata/google-trends-scraper-reliable"
TASK_NAME = "trends-watch-ai-tools"
TRENDS_INPUT = {
    "searchTerms": ["chatgpt", "claude ai", "gemini ai", "perplexity"],
    "geo": "US",
    "timeRange": "now 7-d",
    "includeInterestOverTime": True,
    "includeInterestByRegion": False,
    "includeRelatedQueries": True,
    "includeRelatedTopics": False,
    "outputFormat": "flat",
    "maxRetries": 8,  # retries with fresh sessions/proxies when Google returns 429
}

The inputs worth knowing:

• geo: empty for worldwide, a country ("US") or a sub-region ("US-CA").
• timeRange: now 1-H, now 4-H, now 1-d, now 7-d (hourly points), today 1-m, today 3-m, today 12-m, today 5-y, all, or custom with startDate/endDate.
• compareTerms: by default every term is scaled 0–100 on its own. Set it to true to query terms together (up to 5 per group), like the compare view on the website. Do that if you want to say "A is searched more than B". Leave it off if you only care about each term's own trend, as we do here.
• outputFormat: "flat" returns one row per data point. Each row has a recordType of interestOverTime, interestByRegion, relatedQuery or relatedTopic, which is easy to filter in Python.

2. Save it as a task and schedule it

An Apify task is a saved input for an Actor. A schedule runs it on a cron expression in Apify's cloud. Both are created with the API client:

def setup() -> None:
    task = client.tasks().create(
        actor_id=ACTOR,
        name=TASK_NAME,
        task_input=TRENDS_INPUT,
        max_total_charge_usd=Decimal("0.25"),  # cost cap for every run of this task
    )
    schedule = client.schedules().create(
        name=f"{TASK_NAME}-daily",
        cron_expression="0 7 * * *",  # every day at 07:00
        timezone="UTC",
        is_enabled=True,
        is_exclusive=True,  # skip a tick if the previous run is still going
        actions=[{"type": "RUN_ACTOR_TASK", "actorTaskId": task.id}],
    )
    print(f"task {task.id} and schedule {schedule.id} created")

Two settings matter for unattended jobs:

• max_total_charge_usd on the task caps the cost of every scheduled run, so a typo in the input can't turn into a large bill.
• is_exclusive=True stops runs from piling up if one is slow.

You can do the same in the Apify Console (Actor → Save as a new task → Schedule). The code version lives in your repo and is easy to recreate.

3. Read the latest run and flag what's moving

def latest_items() -> list[dict]:
    task = next(t for t in client.tasks().list().items if t.name == TASK_NAME)
    run = client.task(task.id).last_run(status="SUCCEEDED").get()
    if run is None:
        raise SystemExit("No finished run yet - wait for the schedule or start the task once.")
    print(f"latest run {run.id} finished {run.finished_at:%Y-%m-%d %H:%M} UTC")
    return list(client.dataset(run.default_dataset_id).iterate_items())


def check() -> None:
    items = latest_items()

    # 1) Interest over time: compare the last day with the week's average.
    for term in TRENDS_INPUT["searchTerms"]:
        points = [i["value"] for i in items
                  if i["recordType"] == "interestOverTime" and i["searchTerm"] == term
                  and i.get("hasData", True) and not i.get("isPartial")]
        if len(points) >= 48:
            last_day = sum(points[-24:]) / 24
            week = sum(points) / len(points)
            flag = "  <-- up" if last_day > 1.3 * week else ""
            print(f"{term:12s} last 24h avg {last_day:5.1f} vs 7-day avg {week:5.1f}{flag}")

    # 2) Related queries that are rising fast or marked "Breakout".
    rising = [i for i in items if i["recordType"] == "relatedQuery" and i["rankType"] == "rising"]
    print("\nBreakout / fast-rising related queries:")
    for i in sorted(rising, key=lambda i: (not i.get("isBreakout"), -(i.get("value") or 0)))[:15]:
        label = "Breakout" if i.get("isBreakout") else f'+{i["value"]}%'
        print(f'  [{i["searchTerm"]}] {i["query"]}  ({label})')

A few details:

• isPartial marks the newest data point, which Google is still filling in. Leave it out, or the last hour always looks like a drop.
• For relatedQuery rows with rankType: "rising", value is the percentage increase. isBreakout is true when Google shows "Breakout" (an increase above 5000%).
• last_run(status="SUCCEEDED") skips a run that failed, so a bad night doesn't wipe the report.

Running it

python trends_monitor.py setup   # once
python trends_monitor.py check   # whenever you want the report

Output from a real run (US, past 7 days, 4 terms; the run took about 9 seconds):

latest run 47xnXFExBms4IJ2ba finished 2026-10-27 13:27 UTC
chatgpt      last 24h avg  79.3 vs 7-day avg  71.4
claude ai    last 24h avg  71.5 vs 7-day avg  67.2
gemini ai    last 24h avg  69.8 vs 7-day avg  66.5
perplexity   last 24h avg  46.5 vs 7-day avg  45.3

Breakout / fast-rising related queries:
  [claude ai] claude duet  (Breakout)
  [claude ai] anthropic claude duet ai agent  (Breakout)
  [claude ai] claude ai software vulnerability analysis  (Breakout)
  [claude ai] claude ai oblivion fps boost  (Breakout)
  [perplexity] perplexity ai on device mac  (+1800%)
  [claude ai] venice ai  (+50%)
  [perplexity] is perplexity better than chatgpt  (+50%)
  [claude ai] anthropic claude  (+40%)

Nothing crossed the +30% threshold that day. The related queries are where the news shows up: four Breakout queries for one term in a single week.

The run's OUTPUT summary for the same run reported terms: {total: 4, succeeded: 4, partial: 0, noData: 0, failed: 0}, with 676 interest-over-time rows and 101 related-query rows. That is the record to alert on if you want to know about failures as well as trends:

summary = client.key_value_store(run.default_key_value_store_id).get_record("OUTPUT")["value"]
if summary["terms"]["failed"] or summary["terms"]["noData"]:
    print("some terms returned nothing:", [t["searchTerm"] for t in summary["perTerm"] if t["status"] != "SUCCEEDED"])

Getting the report to you

check() only reads stored data and never runs the scraper, so it costs nothing to call. Some ways to wire it up:

• Run it from your own cron or a GitHub Actions schedule an hour after the Apify schedule.
• Add a webhook on the task (Console → task → Integrations) that calls your endpoint when a run succeeds, and have the endpoint run check() and post to Slack.
• Load the dataset into a sheet or BI tool. GET https://api.apify.com/v2/datasets/{datasetId}/items?format=csv also works.

Full script

"""Watch Google Trends on a schedule and flag rising / breakout queries.

    python trends_monitor.py setup   # once: create a saved task + a daily schedule
    python trends_monitor.py check   # any time: read the latest scheduled run
"""
import os
import sys
from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

ACTOR = "atlasdata/google-trends-scraper-reliable"
TASK_NAME = "trends-watch-ai-tools"
TRENDS_INPUT = {
    "searchTerms": ["chatgpt", "claude ai", "gemini ai", "perplexity"],
    "geo": "US",
    "timeRange": "now 7-d",
    "includeInterestOverTime": True,
    "includeInterestByRegion": False,
    "includeRelatedQueries": True,
    "includeRelatedTopics": False,
    "outputFormat": "flat",
    "maxRetries": 8,  # retries with fresh sessions/proxies when Google returns 429
}


def setup() -> None:
    task = client.tasks().create(
        actor_id=ACTOR,
        name=TASK_NAME,
        task_input=TRENDS_INPUT,
        max_total_charge_usd=Decimal("0.25"),  # cost cap for every run of this task
    )
    schedule = client.schedules().create(
        name=f"{TASK_NAME}-daily",
        cron_expression="0 7 * * *",  # every day at 07:00
        timezone="UTC",
        is_enabled=True,
        is_exclusive=True,  # skip a tick if the previous run is still going
        actions=[{"type": "RUN_ACTOR_TASK", "actorTaskId": task.id}],
    )
    print(f"task {task.id} and schedule {schedule.id} created")


def latest_items() -> list[dict]:
    task = next(t for t in client.tasks().list().items if t.name == TASK_NAME)
    run = client.task(task.id).last_run(status="SUCCEEDED").get()
    if run is None:
        raise SystemExit("No finished run yet - wait for the schedule or start the task once.")
    print(f"latest run {run.id} finished {run.finished_at:%Y-%m-%d %H:%M} UTC")
    return list(client.dataset(run.default_dataset_id).iterate_items())


def check() -> None:
    items = latest_items()

    # 1) Interest over time: compare the last day with the week's average.
    for term in TRENDS_INPUT["searchTerms"]:
        points = [i["value"] for i in items
                  if i["recordType"] == "interestOverTime" and i["searchTerm"] == term
                  and i.get("hasData", True) and not i.get("isPartial")]
        if len(points) >= 48:
            last_day = sum(points[-24:]) / 24
            week = sum(points) / len(points)
            flag = "  <-- up" if last_day > 1.3 * week else ""
            print(f"{term:12s} last 24h avg {last_day:5.1f} vs 7-day avg {week:5.1f}{flag}")

    # 2) Related queries that are rising fast or marked "Breakout".
    rising = [i for i in items if i["recordType"] == "relatedQuery" and i["rankType"] == "rising"]
    print("\nBreakout / fast-rising related queries:")
    for i in sorted(rising, key=lambda i: (not i.get("isBreakout"), -(i.get("value") or 0)))[:15]:
        label = "Breakout" if i.get("isBreakout") else f'+{i["value"]}%'
        print(f'  [{i["searchTerm"]}] {i["query"]}  ({label})')


if __name__ == "__main__":
    {"setup": setup, "check": check}[sys.argv[1] if len(sys.argv) > 1 else "check"]()

Variations

• Seasonality: timeRange: "today 5-y" with includeInterestByRegion: True shows when and where demand peaks each year.
• Content planning: feed the Breakout queries into your editorial backlog every Monday.
• Other Google properties: property can be YouTube, News, Images or Shopping searches instead of web search.
• Ask an assistant: the same Actor is available as an MCP tool for Claude, Cursor and other clients. Configs are in atlas-data-mcp.

Every input and output field is documented on the Store page and in the GitHub repo.

We'll build a tracker that:

1. snapshots Google Jobs for a few searches once a day,
2. stores every posting in SQLite and keeps the date it was first and last seen,
3. prints the new postings, the top hiring companies and the median advertised salary.

It's about 90 lines of Python and uses only the standard library plus apify-client.

Disclosure: the scraper used here is an Apify Actor published by Atlas Data, the author of this post: atlasdata/google-jobs-scraper-reliable. It is pay-per-result and the script sets a cost cap on every run. More examples (Node.js, curl, MCP) are in the GitHub repo.

Setup

pip install apify-client   # tested with apify-client 3.4
export APIFY_TOKEN=...      # https://console.apify.com/settings/integrations

What one result looks like

The input mirrors the Google Jobs filters: queries, location, country, language, datePosted (any, today, 3days, week, month), jobType, remoteOnly, maxResults, plus includeDescription and includeApplyLinks to control how much is returned per job.

Each posting comes back as a flat record. These are the fields we'll use:

salary also keeps the raw text ($95K–$112K a year) in case you want to parse it yourself.

1. The searches

SEARCHES = [
    {"queries": ["data engineer"], "location": "Austin, TX", "country": "us"},
    {"queries": ["data engineer"], "location": "Seattle, WA", "country": "us"},
]
COMMON = {"datePosted": "week", "maxResults": 30, "includeDescription": False}

datePosted: "week" is a good choice for a daily job: each run covers a week of postings, and a missed day doesn't leave a gap. includeDescription: False keeps rows small. Turn it on if you want to count skills in job descriptions later.

A tip on localization: match country (and, if you set it, language) to the market you're searching, e.g. "country": "de" with a German query such as "Softwareentwickler". In our tests, an English query with a forced language: "en" in a non-English market sometimes returned an empty Google Jobs panel. The run's OUTPUT record then reports NO_DATA for that search, so it doesn't fail silently.

2. A table that remembers

db = sqlite3.connect("jobs.db")
db.execute("""
CREATE TABLE IF NOT EXISTS jobs (
    job_id TEXT PRIMARY KEY,
    title TEXT, company TEXT, location TEXT, via TEXT, job_type TEXT,
    salary_min REAL, salary_max REAL, salary_currency TEXT, salary_period TEXT,
    posted_at TEXT, apply_url TEXT, search_query TEXT,
    first_seen TEXT, last_seen TEXT
)""")

first_seen never changes after the first insert. last_seen is refreshed on every run that still finds the job. With these two columns you get "new today" (first_seen = today) and "probably filled or closed" (last_seen older than a few days) without storing full snapshots.

3. Fetch and upsert

def fetch(search: dict) -> list[dict]:
    run = client.actor("atlasdata/google-jobs-scraper-reliable").call(
        run_input={**COMMON, **search},
        max_total_charge_usd=Decimal("0.30"),
        logger=None,
    )
    if run is None or run.status != "SUCCEEDED":
        raise RuntimeError(f"run failed: {run and run.status}")
    return list(client.dataset(run.default_dataset_id).iterate_items())


def upsert(job: dict, today: str) -> bool:
    """Insert a job or refresh last_seen. Returns True when the job is new."""
    is_new = db.execute("SELECT 1 FROM jobs WHERE job_id = ?", (job["jobId"],)).fetchone() is None
    db.execute(
        """INSERT INTO jobs VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
           ON CONFLICT(job_id) DO UPDATE SET last_seen = excluded.last_seen""",
        (
            job["jobId"], job["title"], job["companyName"], job.get("location"), job.get("via"),
            job.get("jobType"), job.get("salaryMin"), job.get("salaryMax"), job.get("salaryCurrency"),
            job.get("salaryPeriod"), job.get("postedAt"), job.get("applyUrl"), job.get("searchQuery"),
            today, today,
        ),
    )
    return is_new

ON CONFLICT ... DO UPDATE (SQLite 3.24+) turns the insert into an upsert, so the script can be re-run safely any number of times.

4. Simple market stats

    print("\nTop hiring companies (last 30 days):")
    for company, n in db.execute(
        """SELECT company, COUNT(*) FROM jobs WHERE last_seen >= date('now', '-30 day')
           GROUP BY company ORDER BY COUNT(*) DESC LIMIT 10"""
    ):
        print(f"  {n:3d}  {company}")

    yearly = [
        (lo + hi) / 2
        for lo, hi in db.execute(
            """SELECT salary_min, salary_max FROM jobs
               WHERE salary_period = 'year' AND salary_currency = 'USD'
                 AND salary_min >= 10000 AND salary_max IS NOT NULL"""  # skip non-annual figures
        )
    ]

Only some postings include a salary. Print how many did, so a median based on 13 postings isn't read as covering all 60. The salary_min >= 10000 condition is a cheap sanity check that keeps hourly or monthly figures out of an annual median.

Running it

First run (real output):

$ python jobs_tracker.py
Austin, TX: 30 jobs
Seattle, WA: 30 jobs
60 new postings today

Top hiring companies (last 30 days):
    2  ICF
    2  Amazon
    2  Alignerr
    1  Vaspire Technologies
    ...

Median advertised salary (USD/year): 166,500 from 13 of 60 postings that list one

From the second day on, "new postings today" only counts jobs that weren't seen before. Put it in cron:

0 6 * * *  cd /path/to/tracker && APIFY_TOKEN=... python jobs_tracker.py >> tracker.log 2>&1

or use an Apify schedule if you'd rather not keep a machine running. The companion Google Trends post shows how.

Full script

"""Daily Google Jobs snapshot -> SQLite, with simple market stats."""
import os
import sqlite3
from datetime import date
from decimal import Decimal
from statistics import median

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

SEARCHES = [
    {"queries": ["data engineer"], "location": "Austin, TX", "country": "us"},
    {"queries": ["data engineer"], "location": "Seattle, WA", "country": "us"},
]
COMMON = {"datePosted": "week", "maxResults": 30, "includeDescription": False}

db = sqlite3.connect("jobs.db")
db.execute("""
CREATE TABLE IF NOT EXISTS jobs (
    job_id TEXT PRIMARY KEY,
    title TEXT, company TEXT, location TEXT, via TEXT, job_type TEXT,
    salary_min REAL, salary_max REAL, salary_currency TEXT, salary_period TEXT,
    posted_at TEXT, apply_url TEXT, search_query TEXT,
    first_seen TEXT, last_seen TEXT
)""")


def fetch(search: dict) -> list[dict]:
    run = client.actor("atlasdata/google-jobs-scraper-reliable").call(
        run_input={**COMMON, **search},
        max_total_charge_usd=Decimal("0.30"),
        logger=None,
    )
    if run is None or run.status != "SUCCEEDED":
        raise RuntimeError(f"run failed: {run and run.status}")
    return list(client.dataset(run.default_dataset_id).iterate_items())


def upsert(job: dict, today: str) -> bool:
    """Insert a job or refresh last_seen. Returns True when the job is new."""
    is_new = db.execute("SELECT 1 FROM jobs WHERE job_id = ?", (job["jobId"],)).fetchone() is None
    db.execute(
        """INSERT INTO jobs VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
           ON CONFLICT(job_id) DO UPDATE SET last_seen = excluded.last_seen""",
        (
            job["jobId"], job["title"], job["companyName"], job.get("location"), job.get("via"),
            job.get("jobType"), job.get("salaryMin"), job.get("salaryMax"), job.get("salaryCurrency"),
            job.get("salaryPeriod"), job.get("postedAt"), job.get("applyUrl"), job.get("searchQuery"),
            today, today,
        ),
    )
    return is_new


def main() -> None:
    today = date.today().isoformat()
    new_jobs = 0
    for search in SEARCHES:
        jobs = fetch(search)
        new_jobs += sum(upsert(job, today) for job in jobs)
        print(f'{search["location"]}: {len(jobs)} jobs')
    db.commit()
    print(f"{new_jobs} new postings today")

    print("\nTop hiring companies (last 30 days):")
    for company, n in db.execute(
        """SELECT company, COUNT(*) FROM jobs WHERE last_seen >= date('now', '-30 day')
           GROUP BY company ORDER BY COUNT(*) DESC LIMIT 10"""
    ):
        print(f"  {n:3d}  {company}")

    yearly = [
        (lo + hi) / 2
        for lo, hi in db.execute(
            """SELECT salary_min, salary_max FROM jobs
               WHERE salary_period = 'year' AND salary_currency = 'USD'
                 AND salary_min >= 10000 AND salary_max IS NOT NULL"""  # skip non-annual figures
        )
    ]
    total = db.execute("SELECT COUNT(*) FROM jobs").fetchone()[0]
    if yearly:
        print(f"\nMedian advertised salary (USD/year): {median(yearly):,.0f} "
              f"from {len(yearly)} of {total} postings that list one")


if __name__ == "__main__":
    main()

Ideas for extending it

• Time-to-close: julianday(last_seen) - julianday(first_seen) per company shows how fast roles disappear.
• Skills: set includeDescription: True and count mentions of dbt, Spark, Airflow, etc. per week.
• Remote share: run the same query with remoteOnly: True and compare counts over time.
• Where jobs are posted: GROUP BY via shows whether a market leans on LinkedIn, company career sites or niche boards.

The Actor's Store page documents every input and output field, and the GitHub repo has runnable examples. To query jobs from an AI assistant instead, the MCP setup is in atlas-data-mcp.

• the Google Ads Transparency Center,
• the LinkedIn Ad Library,
• the TikTok Ad Library (EU/EEA, Switzerland, the UK and Turkey).

All three are useful for competitive research: what a brand is promoting, which formats it uses, which landing pages it sends people to, and how long an ad stays live. None of them has a convenient export, and the three sites look nothing alike.

In this tutorial we'll write one Python script (~120 lines) that:

1. fetches a brand's ads from all three libraries in parallel,
2. maps them to one common table (platform, ad_id, advertiser, format, text, first_shown, last_shown, landing_url, ad_url),
3. keeps only ads from the brand itself (keyword searches return other advertisers too),
4. remembers which ads it has already seen, so each run tells you what is new.

The example brand is Canva, because it runs ads on all three platforms. Swap in any company you like.

Disclosure: the three scrapers used here are Apify Actors published by Atlas Data, the author of this post. They are pay-per-result, and every call below sets a hard cost cap. You could also write the scraping yourself; the parts on normalizing and diffing apply either way.

Setup

pip install apify-client   # tested with apify-client 3.4
export APIFY_TOKEN=...      # https://console.apify.com/settings/integrations

The three Actors:

1. One input per platform

Each library is searched a little differently, and that affects what you get back:

• Google works best with the advertiser's domain (canva.com). You can also pass an advertiser name, an advertiser ID (AR...) or a Transparency Center URL. region is an ISO country code.
• LinkedIn has an advertisers field that matches the company page name. dateRange takes values such as last-30-days, current-month or custom.
• TikTok searchTerms match both the advertiser name and the ad text. Wrapping the term in double quotes makes it an exact phrase. countries must be EU/EEA, CH, GB or TR, because those are the only markets TikTok publishes.

BRAND_NAME = "Canva"         # used for TikTok + LinkedIn searches
BRAND_DOMAIN = "canva.com"    # used for Google Ads Transparency
MAX_ADS = 15                  # per platform, keep it small while testing

JOBS = {
    "google": (
        "atlasdata/google-ads-transparency-scraper-reliable",
        {"searchTerms": [BRAND_DOMAIN], "maxAdsPerSearch": MAX_ADS, "region": "DE"},
    ),
    "linkedin": (
        "atlasdata/linkedin-ad-library-scraper-reliable",
        {"advertisers": [BRAND_NAME], "maxAdsPerSearch": MAX_ADS, "dateRange": "last-30-days"},
    ),
    "tiktok": (
        "atlasdata/tiktok-ad-library-scraper-reliable",
        {"searchTerms": [f'"{BRAND_NAME}"'], "countries": ["DE", "FR"], "maxAdsPerSearch": MAX_ADS,
         "dateRange": "last-30-days", "includeDetails": True},
    ),
}

On TikTok, includeDetails: True opens each ad's detail page. That adds the landing page, CTA, objective, targeting and impressions per country, at the cost of a slower run.

2. Run the three scrapers in parallel

ActorClient.call() starts a run and waits for it to finish. The calls mostly spend their time waiting, so a thread pool runs all three at once.

def run_actor(platform: str) -> tuple[str, list[dict]]:
    actor_id, run_input = JOBS[platform]
    run = client.actor(actor_id).call(
        run_input=run_input,
        max_total_charge_usd=Decimal("0.50"),  # hard cost cap per run
        logger=None,                           # don't stream the run log
    )
    if run is None or run.status != "SUCCEEDED":
        # One blocked platform shouldn't sink the whole report: warn and move on.
        print(f"{platform}: run {run and run.status} - {run and run.status_message}")
        return platform, []
    # Each Actor also writes a short summary (per-search status, errors) to OUTPUT.
    summary = client.key_value_store(run.default_key_value_store_id).get_record("OUTPUT")
    if summary and summary["value"].get("status") != "SUCCEEDED":
        print(f"{platform}: {summary['value'].get('status')} - check the run's OUTPUT record")
    items = list(client.dataset(run.default_dataset_id).iterate_items())
    return platform, items

Two notes:

• In apify-client v3, call() returns a Run model, so you read run.default_dataset_id. The old run["defaultDatasetId"] from v1/v2 code no longer works.
• Public ad libraries do block automated traffic from time to time, LinkedIn's Cloudflare in particular. A run can end PARTIAL, for example when a few detail pages were refused, or FAILED when every attempt was blocked. The OUTPUT record says which. For a daily report, it is better to log the problem and keep the other platforms' data than to crash.

3. Normalize to one schema

Every platform has its own field names. This function maps them onto one row:

def normalize(platform: str, ad: dict) -> dict:
    """Map each platform's fields onto one common row."""
    if platform == "google":
        return {
            "platform": "google",
            "ad_id": ad["creativeId"],
            "advertiser": ad["advertiserName"],
            "format": ad["format"],
            "text": " | ".join(filter(None, [ad.get("headline"), ad.get("description")])),
            "first_shown": (ad.get("firstShown") or "")[:10],
            "last_shown": (ad.get("lastShown") or "")[:10],
            "landing_url": ad.get("targetDomain"),
            "ad_url": ad["adUrl"],
        }
    if platform == "linkedin":
        return {
            "platform": "linkedin",
            "ad_id": ad["adId"],
            "advertiser": ad["advertiserName"],
            "format": ad.get("format"),
            "text": (ad.get("adText") or "")[:300],
            "first_shown": ad.get("firstShown") or "",  # LinkedIn often hides dates
            "last_shown": ad.get("lastShown") or "",
            "landing_url": ad.get("destinationUrl"),
            "ad_url": ad["adUrl"],
        }
    return {  # tiktok
        "platform": "tiktok",
        "ad_id": ad["adId"],
        "advertiser": ad["advertiserName"],
        "format": ad.get("mediaType"),
        "text": (ad.get("adText") or "")[:300],
        "first_shown": ad.get("firstShown"),
        "last_shown": ad.get("lastShown"),
        "landing_url": ad.get("landingPageUrl"),
        "ad_url": ad["adUrl"],
    }

Things you'll notice in real data:

• Google text ads often come back with an empty headline/description. The Transparency Center stores many text ads as rendered images, and imageUrl points to that image. Dates (firstShown, lastShown) are reliable, and regions[] shows where the ad ran.
• LinkedIn shows a first/last-shown date and impression ranges only for some ads (mostly those served in the EU), so expect blanks. adText, headline, ctaText and format are almost always present.
• TikTok gives the most detail: estimatedAudience, impressionsByCountry, targetAges, targetGenders and more, if you want to add columns.

4. Keep the brand's own ads and diff against the last run

A keyword search for "Canva" on TikTok also returns template creators and agencies whose ads mention Canva. A domain search on Google can return resellers that link to canva.com. If you only want the brand's own ads, filter on the advertiser name:

def main() -> None:
    with ThreadPoolExecutor(max_workers=3) as pool:
        results = list(pool.map(run_actor, JOBS))

    rows = [normalize(platform, ad) for platform, items in results for ad in items]
    print("fetched:", {platform: len(items) for platform, items in results})

    # Keyword and domain searches can also return *other* advertisers
    # (resellers, partners, anyone mentioning the brand). Keep the brand's own ads.
    rows = [r for r in rows if BRAND_NAME.lower() in (r["advertiser"] or "").lower()]
    print("from", BRAND_NAME, "itself:", len(rows))

    # Compare with the previous snapshot to find ads we have not seen before.
    seen_file = Path("seen_ads.json")
    seen = set(json.loads(seen_file.read_text())) if seen_file.exists() else set()
    new_rows = [r for r in rows if f'{r["platform"]}:{r["ad_id"]}' not in seen]
    seen_file.write_text(json.dumps(sorted(seen | {f'{r["platform"]}:{r["ad_id"]}' for r in rows})))

    with open("competitor_ads.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()) if rows else ["platform"])
        writer.writeheader()
        writer.writerows(rows)

    print(f"{len(rows)} ads saved, {len(new_rows)} new since last run")
    for r in new_rows[:10]:
        print(f'[{r["platform"]}] {r["first_shown"]} {r["format"]}: {r["text"][:80]!r} -> {r["landing_url"]}')

Leave out the filter if you do want to see who else is bidding on a competitor's name. That is often the more interesting list.

Running it

$ python competitor_ads.py
fetched: {'google': 15, 'linkedin': 15, 'tiktok': 15}
from Canva itself: 39
39 ads saved, 39 new since last run
[google] 2024-01-14 TEXT: '' -> canva.com
[google] 2023-10-20 TEXT: '' -> canva.com
...

Of the 45 ads fetched, 39 came from Canva's own accounts: 15 on Google, 15 on LinkedIn and 9 on TikTok. The other 6 TikTok ads were from third parties that mention Canva. The CSV has one row per ad across all three platforms, ready for a spreadsheet or pandas.

Run it again tomorrow, for example from cron or a GitHub Actions schedule, and only the creatives that appeared since then are reported as new. With 15 ads per platform, a run takes about two minutes, and its cost is capped by max_total_charge_usd.

Full script

"""Collect a competitor's ads from TikTok, LinkedIn and Google into one table."""
import csv
import json
import os
from concurrent.futures import ThreadPoolExecutor
from decimal import Decimal
from pathlib import Path

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

BRAND_NAME = "Canva"         # used for TikTok + LinkedIn searches
BRAND_DOMAIN = "canva.com"    # used for Google Ads Transparency
MAX_ADS = 15                  # per platform, keep it small while testing

JOBS = {
    "google": (
        "atlasdata/google-ads-transparency-scraper-reliable",
        {"searchTerms": [BRAND_DOMAIN], "maxAdsPerSearch": MAX_ADS, "region": "DE"},
    ),
    "linkedin": (
        "atlasdata/linkedin-ad-library-scraper-reliable",
        {"advertisers": [BRAND_NAME], "maxAdsPerSearch": MAX_ADS, "dateRange": "last-30-days"},
    ),
    "tiktok": (
        "atlasdata/tiktok-ad-library-scraper-reliable",
        {"searchTerms": [f'"{BRAND_NAME}"'], "countries": ["DE", "FR"], "maxAdsPerSearch": MAX_ADS,
         "dateRange": "last-30-days", "includeDetails": True},
    ),
}


def run_actor(platform: str) -> tuple[str, list[dict]]:
    actor_id, run_input = JOBS[platform]
    run = client.actor(actor_id).call(
        run_input=run_input,
        max_total_charge_usd=Decimal("0.50"),  # hard cost cap per run
        logger=None,                           # don't stream the run log
    )
    if run is None or run.status != "SUCCEEDED":
        # One blocked platform shouldn't sink the whole report: warn and move on.
        print(f"{platform}: run {run and run.status} - {run and run.status_message}")
        return platform, []
    # Each Actor also writes a short summary (per-search status, errors) to OUTPUT.
    summary = client.key_value_store(run.default_key_value_store_id).get_record("OUTPUT")
    if summary and summary["value"].get("status") != "SUCCEEDED":
        print(f"{platform}: {summary['value'].get('status')} - check the run's OUTPUT record")
    items = list(client.dataset(run.default_dataset_id).iterate_items())
    return platform, items


def normalize(platform: str, ad: dict) -> dict:
    """Map each platform's fields onto one common row."""
    if platform == "google":
        return {
            "platform": "google",
            "ad_id": ad["creativeId"],
            "advertiser": ad["advertiserName"],
            "format": ad["format"],
            "text": " | ".join(filter(None, [ad.get("headline"), ad.get("description")])),
            "first_shown": (ad.get("firstShown") or "")[:10],
            "last_shown": (ad.get("lastShown") or "")[:10],
            "landing_url": ad.get("targetDomain"),
            "ad_url": ad["adUrl"],
        }
    if platform == "linkedin":
        return {
            "platform": "linkedin",
            "ad_id": ad["adId"],
            "advertiser": ad["advertiserName"],
            "format": ad.get("format"),
            "text": (ad.get("adText") or "")[:300],
            "first_shown": ad.get("firstShown") or "",  # LinkedIn often hides dates
            "last_shown": ad.get("lastShown") or "",
            "landing_url": ad.get("destinationUrl"),
            "ad_url": ad["adUrl"],
        }
    return {  # tiktok
        "platform": "tiktok",
        "ad_id": ad["adId"],
        "advertiser": ad["advertiserName"],
        "format": ad.get("mediaType"),
        "text": (ad.get("adText") or "")[:300],
        "first_shown": ad.get("firstShown"),
        "last_shown": ad.get("lastShown"),
        "landing_url": ad.get("landingPageUrl"),
        "ad_url": ad["adUrl"],
    }


def main() -> None:
    with ThreadPoolExecutor(max_workers=3) as pool:
        results = list(pool.map(run_actor, JOBS))

    rows = [normalize(platform, ad) for platform, items in results for ad in items]
    print("fetched:", {platform: len(items) for platform, items in results})

    # Keyword and domain searches can also return *other* advertisers
    # (resellers, partners, anyone mentioning the brand). Keep the brand's own ads.
    rows = [r for r in rows if BRAND_NAME.lower() in (r["advertiser"] or "").lower()]
    print("from", BRAND_NAME, "itself:", len(rows))

    # Compare with the previous snapshot to find ads we have not seen before.
    seen_file = Path("seen_ads.json")
    seen = set(json.loads(seen_file.read_text())) if seen_file.exists() else set()
    new_rows = [r for r in rows if f'{r["platform"]}:{r["ad_id"]}' not in seen]
    seen_file.write_text(json.dumps(sorted(seen | {f'{r["platform"]}:{r["ad_id"]}' for r in rows})))

    with open("competitor_ads.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()) if rows else ["platform"])
        writer.writeheader()
        writer.writerows(rows)

    print(f"{len(rows)} ads saved, {len(new_rows)} new since last run")
    for r in new_rows[:10]:
        print(f'[{r["platform"]}] {r["first_shown"]} {r["format"]}: {r["text"][:80]!r} -> {r["landing_url"]}')


if __name__ == "__main__":
    main()

Where to go next

• Add impressionsByCountry (LinkedIn/TikTok) or regions (Google) as columns to see where spend is concentrated.
• Download videoUrl and imageUrls for a swipe file. Some media URLs are signed and expire, so download them soon after the run.
• Each repo has Node.js and curl versions of the same calls under examples/: Google, LinkedIn, TikTok.
• If you'd rather ask an AI assistant ("what has Canva been advertising on LinkedIn this month?"), the same tools can be used over MCP. The setup for Claude Desktop and Cursor is in atlas-data-mcp.