Indexing many URLs: batch files, stdin, and sitemaps

One URL at a time is fine for a quick check, but the tool is built for lists. Here are the three ways to feed it many URLs. and they all compose.

1. Batch file (-batch)

Put one URL per line. Lines starting with # are ignored, as are blank lines.

urls.txt:

## pages to refresh
https://www.toolsura.com/post-a/
https://www.toolsura.com/post-b/
https://www.toolsura.com/post-c/

./gsc-indexer -creds sa.json -batch urls.txt

2. Stdin pipe

Pipe a list straight in. handy in scripts:

cat urls.txt | ./gsc-indexer -creds sa.json

The tool auto-detects a piped stdin and reads from it (it only reads stdin when input isn't coming from an interactive terminal, so your keyboard won't block it).

3. Positional arguments

Multiple URLs as plain arguments also work:

./gsc-indexer -creds sa.json "https://www.toolsura.com/a/" "https://www.toolsura.com/b/"

4. Sitemaps (.xml)

Hand it a sitemap URL and it fetches the file and expands it into every <loc> entry:

./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -report ./report

• Sitemap indexes are followed recursively. a sitemap that lists child sitemaps (<sitemap><loc>) is expanded all the way down.
• This is the easiest way to (re-)request indexing for an entire site.

All sources compose and dedupe

Mix them freely. the tool merges batch + args + sitemap + stdin, removes duplicates, and runs the unique set together:

./gsc-indexer -creds sa.json -batch urls.txt "https://www.toolsura.com/extra/" "https://www.toolsura.com/sitemap.xml"

The URL cap

Expansion is capped at 50,000 URLs to protect against a runaway sitemap hanging the run. If you somehow exceed it you'll get a clear error telling you so. For context, a normal blog sitemap is a few hundred URLs.

Next

• Track progress across runs: -report and -diff.
• Big sitemap run stalling? Throttling and quiet mode.

Sources

• Google Search Console
• URL Inspection API documentation
• Google Search APIs overview
• Service account OAuth (Google Identity)
• About service accounts (Google Cloud IAM)
• Verify site ownership
• Property types: URL-prefix vs Domain
• URL Inspection tool help
• Indexing API (JobPosting/BroadcastEvent only)
• Request indexing / inspect a URL