Siftsmith Run on Apify
← All tools

Find recently updated pages on a website

Most company websites publish a sitemap that lists every page with the date it last changed. Read it and you can see what a company touched in the last month (new integrations, a changelog entry, a pricing edit) without setting up a monitor first. This page shows how to do that by hand, which dates to distrust, and how to do it for a list of companies in one run.

Why the sitemap is the place to look

A sitemap is an XML file, usually at /sitemap.xml and named in /robots.txt, that a site publishes for search engines. Each page gets a <loc> (its URL) and often a <lastmod> (when the site says it last changed). Because the dates are already there, you get the last few months of history on the first look. A page monitor only sees changes made after you add a page to it.

Grouped by section, the dates say what a company is working on:

What it can’t tell you: what changed on the page, or whether the page is new or edited. The date says only that the site considers the page changed.

Read a sitemap by hand

1. Find the sitemap

Open https://domain/robots.txt and look for Sitemap: lines. If there are none, try /sitemap.xml and /sitemap_index.xml. If the file you open is a <sitemapindex>, it lists more sitemap files rather than pages; open each <loc> inside it (sitemap-blog.xml, sitemap-product-pages.xml and so on).

2. List the most recently changed pages

For a single sitemap file this prints every dated page, newest first:

curl -sL https://linear.app/sitemap.xml | tr -d '\n' | awk '{gsub(/<url>/,"\n")}1' \
  | sed -n 's:.*<loc>\(.*\)</loc>.*<lastmod>\(.*\)</lastmod>.*:\2 \1:p' \
  | sort -r | head

On 27 September 2026 it returned:

2026-09-26T12:50:49Z https://linear.app/integrations/teamretro
2026-09-25T20:22:07Z https://linear.app/docs/billing-and-plans
2026-09-25T15:00:20Z https://linear.app/docs/diffs-integrations
2026-09-25T07:59:41Z https://linear.app/changelog/2026-09-24-new-controls-for-linear-coding-agent
2026-09-25T01:42:26Z https://linear.app/docs/jira
2026-09-24T22:36:54Z https://linear.app/enablement/guides/coding-sessions
2026-09-24T22:36:54Z https://linear.app/enablement/guide/coding-sessions
2026-09-24T22:36:54Z https://linear.app/docs/coding-sessions
2026-09-24T16:10:54Z https://linear.app/docs/projects
2026-09-24T15:57:36Z https://linear.app/docs/project-status

A new integration page, a changelog entry and a batch of docs, all in the last three days. The command assumes <lastmod> comes after <loc> inside each <url>, which is the usual order. If you don’t use a terminal, open the sitemap in a browser and search the page for this month’s date, e.g. 2026-09.

3. Count changes in the last 30 days

# GNU date (Linux); on macOS replace the date part with: date -u -v-30d +%FT%TZ
curl -sL https://linear.app/sitemap.xml | tr -d '\n' | awk '{gsub(/<url>/,"\n")}1' \
  | sed -n 's:.*<lastmod>\(.*\)</lastmod>.*:\1:p' \
  | awk -v d="$(date -u -d '30 days ago' +%FT%TZ)" '$0 >= d' | wc -l

That compares dates as text, which works for UTC dates ending in Z and for date-only values; a date written with an offset such as +02:00 can be off by a few hours. For linear.app it printed 97. Before trusting a count like that, check how the dates were written.

Which dates to distrust

A <lastmod> is whatever the site’s software writes. Plenty of sites write dates that have nothing to do with anyone editing a page. Before you trust a count, run this on the same file. It prints the most common exact timestamps. For vercel.com on 27 September 2026:

curl -sL https://vercel.com/sitemap.xml | tr -d '\n' | awk '{gsub(/<url>/,"\n")}1' \
  | sed -n 's:.*<lastmod>\(.*\)</lastmod>.*:\1:p' | sort | uniq -c | sort -rn | head -4
    831 2026-06-17T09:43:59.816Z
    809 2026-08-18T06:10:12.418Z
    539 2026-09-27T11:09:45.522Z
    434 2026-08-29T03:11:19.144Z

809 pages don’t get edited on the same millisecond. Those are jobs that rewrote the dates in bulk, and the third line was stamped the same morning we ran the command. Count vercel.com’s last 30 days with the command from step 3 and you get 3,827 of its 7,378 dated URLs, nearly all of them noise. The patterns to look for:

Filtering this by hand for one site is a few minutes of sort | uniq -c. For a list of 50 companies, each with several sitemap files, it’s an afternoon.

Filter out same-day lastmod restamps

The simplest cleanup that makes a real difference: count URLs per day, and drop any single day that carries a large share of the whole sitemap. Nobody edits a quarter of a website on one day; a deploy script does. Here that share is 10% of all dated URLs. Save the dates once, then count both ways:

curl -sL https://vercel.com/sitemap.xml | tr -d '\n' | awk '{gsub(/<url>/,"\n")}1' \
  | sed -n 's:.*<lastmod>\(.*\)</lastmod>.*:\1:p' > dates.txt

since=$(date -u -d '30 days ago' +%F)   # GNU date; on macOS: date -u -v-30d +%F

wc -l < dates.txt                                   # URLs with a lastmod
awk -v d="$since" '$0 >= d' dates.txt | wc -l        # naive: last 30 days
cut -c1-10 dates.txt | sort | uniq -c | sort -rn | head -3   # busiest days

# drop every day holding 10% or more of all dated URLs, then recount
awk -v d="$since" 'NR==FNR{n[substr($0,1,10)]++; t++; next}
     n[substr($0,1,10)] < t/10 && $0 >= d' dates.txt dates.txt | wc -l

For a sitemap index, such as resend.com’s, fetch every file it lists into the same dates.txt, then run the counts above:

curl -sL https://resend.com/sitemap.xml | grep -o '<loc>[^<]*</loc>' | sed 's:</*loc>::g' \
  | xargs -n1 curl -sL | tr -d '\n' | awk '{gsub(/<url>/,"\n")}1' \
  | sed -n 's:.*<lastmod>\(.*\)</lastmod>.*:\1:p' > dates.txt

Four sites, measured 30 September 2026

SiteURLsWith lastmodNaive last 30 daysDays dropped in the last 30 (share of dated URLs)After dropping them
linear.app (1 file)1,040987102none; busiest day is 2021-10-25 with 62 (6%)102
resend.com (11 files)1,0081,0085662026-09-30: 316 (31%)
2026-09-24: 139 (14%)
111
vercel.com (1 file)8,4568,4564,6622026-09-30: 2,364 (28%)2,298
posthog.com (1 file)13,3410no dates to count

What the numbers show:

Check when a single page last changed

When you care about one URL, say a pricing page, there are three more places a date can come from. Each has a catch.

Check a list of companies in one run

Launch Signals does the work of this page for a list of domains. It finds the sitemaps through robots.txt, follows indexes and gzipped files, catches the build-date stamps and same-timestamp batches described above along with several more patterns (sweeps through one section, sections restamped together, careers pages rebuilt weekly, tag and category pages that echo a single post), and returns one row per domain with 7, 30 and 90-day counts by section. Put domains into Company domains on the Apify Store page, or send JSON through the API:

{ "domains": ["linear.app", "resend.com", "vercel.com", "posthog.com"] }

Sites like PostHog, with no dates, and sites where most recent dates look machine-written come back as free rows that say why, rather than a number. It leans toward under-counting, so a genuine launch shipped in one big deploy can be dropped with the noise. The product page has a sample run, every output field and the exact thresholds.

The crawler sends SiftsmithBot/1.0 (+https://siftsmith.com) as its user agent, respects robots.txt, and requests robots.txt and sitemap files. The only page it requests is the homepage, and only when no sitemap is found, to check whether the domain now redirects elsewhere.

Other ways to do it

ToolWhat it isPrice (official page)
VisualpingPage-change monitor: checks the pages you add on a schedule and shows what changed on themFree for 5 pages checked hourly; Personal $14/month for 10 pages ($10/month billed yearly); Business $140/month for 200 pages ($100/month billed yearly)
Sitemap URL Extractor (an Apify Actor by another developer)Lists every URL in a site’s sitemaps with its lastmod, optionally only those changed since a date$0.50 per 1,000 URLs on Apify’s free plan, from $0.20 per 1,000 on higher tiers
Launch SignalsFiltered 7/30/90-day counts by section for a list of domains$0.03 per domain with a trustworthy result; other rows free; no subscription

They answer different questions. A monitor like Visualping shows you the actual text or visual change on a page, which a sitemap never will, but only on pages you chose in advance and only from the day you added them. A raw sitemap export gives you every URL and date, which is the right tool for SEO work, and its own page says the lastmod “is reported as the sitemap writes it; it is not independently verified”, so the checks above are up to you. Launch Signals gives one filtered summary per company across all sections, with history from the first run, and nothing about what changed inside a page.

Prices checked 27 September 2026 on Visualping pricing and the Sitemap URL Extractor Store page.

What it costs

Analyzed domain$0.03 ($30 per 1,000), no subscription

You pay only for a domain that comes back status: "ok": at least 80% of its sitemap files were read, at least 5 of its dates from the last 90 days survived the checks, and in the last 30 days it changed something worth a pitch: a changelog, pricing, integrations, product, customers or careers page, 3 or more blog or news pages, or 10 or more docs pages. Sites with no dates, untrustworthy dates, too little activity or a sitemap that couldn’t be read get a free row with changes: null and a status saying which. The same site entered twice is charged once. The product page lists every status.

Set a maximum total charge for the run and you’re never charged past it. Once it’s reached, every input left over still gets a row with skipped: true, uncharged.

FAQ

Can I see when a company last changed its pricing page?

Only if the pricing page is in the sitemap with a <lastmod>. Search the sitemap for pricing; if it’s there with a date, that’s the site’s own claim of the last change. When a pricing change turns up in a run, it gets its own pitch reason. To see what changed in the prices, compare two Wayback Machine captures (see Check when a single page last changed) or put a page monitor on it.

Does a recent lastmod mean the page is new?

No. It means the site considers the page changed. A sitemap has no field for when a page was created. To separate new pages from edited ones, keep last week’s list of URLs and compare: a URL that wasn’t there before is new. Launch Signals does that comparison for you with compareWithPreviousRun.

What does it cost to check 200 companies?

At most $6. Sites with no dates, unreliable dates or nothing notable in the last 30 days are free, so it’s usually less. Of the four sites measured above, PostHog would be a free row on dates alone.

Can I run it every week?

Yes, and with compareWithPreviousRun on, each weekly run is a diff: new, updated and removed URLs since last week. The first run saves a baseline in your own Apify account; the option doesn’t change the price. Schedule it on Apify, or call it from Clay, Make, Zapier or the API. Using our tools in Clay has the steps, and the MCP page shows how to give it to an AI agent.

Check your list on the Apify Store →

Fields, statuses and pricing in one place: the Launch Signals product page.