Indexing many URLs: batch files, stdin, and sitemaps
Abhay Khant
Jan 1, 1970 • 2 min read
One URL at a time is fine for a quick check, but the tool is built for lists. Here are the three ways to feed it many URLs. and they all compose.
1. Batch file (-batch)
Put one URL per line. Lines starting with # are ignored, as are blank lines.
urls.txt:
## pages to refresh
https://www.toolsura.com/post-a/
https://www.toolsura.com/post-b/
https://www.toolsura.com/post-c/
./gsc-indexer -creds sa.json -batch urls.txt
2. Stdin pipe
Pipe a list straight in. handy in scripts:
cat urls.txt | ./gsc-indexer -creds sa.json
The tool auto-detects a piped stdin and reads from it (it only reads stdin when input isn't coming from an interactive terminal, so your keyboard won't block it).
3. Positional arguments
Multiple URLs as plain arguments also work:
./gsc-indexer -creds sa.json "https://www.toolsura.com/a/" "https://www.toolsura.com/b/"
4. Sitemaps (.xml)
Hand it a sitemap URL and it fetches the file and expands it into every
<loc> entry:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -report ./report
- Sitemap indexes are followed recursively. a sitemap that lists child
sitemaps (
<sitemap><loc>) is expanded all the way down. - This is the easiest way to (re-)request indexing for an entire site.
All sources compose and dedupe
Mix them freely. the tool merges batch + args + sitemap + stdin, removes duplicates, and runs the unique set together:
./gsc-indexer -creds sa.json -batch urls.txt "https://www.toolsura.com/extra/" "https://www.toolsura.com/sitemap.xml"
The URL cap
Expansion is capped at 50,000 URLs to protect against a runaway sitemap hanging the run. If you somehow exceed it you'll get a clear error telling you so. For context, a normal blog sitemap is a few hundred URLs.
Next
- Track progress across runs: -report and -diff.
- Big sitemap run stalling? Throttling and quiet mode.
