siteprobe v1.5.1

Check every URL in a sitemap.

Siteprobe is a command-line tool written in Rust. It reads a sitemap.xml or a plain list of URLs, requests every page concurrently, and reports which ones failed and which ones were slow. Results go to the terminal, or to a CSV, JSON, or HTML file.

brew install bartTC/siteprobe/siteprobe

also pipx install siteprobe · cargo install siteprobe · or run it directly with uvx siteprobe

Read the README Source on GitHub
  • After a deployRequest every URL in the sitemap and see whether any of them fail.
  • Before a launchFind 404s, 500s, timeouts, and pages slower than a threshold you set.
  • In CIThe exit code reports failed or slow URLs, so a pipeline step can fail on them.

Fig. 1A run against a real sitemap: 303 URLs, 8 concurrent requests, 2.44 s in total.

01

Where the URLs come from.

A sitemap, a text file, or URLs given on the command line. The format is detected automatically, so any source may be a sitemap, a sitemap index, or a plain list.

sitemap.xml

A sitemap, or a sitemap index

Sitemap index files are followed recursively, including nested ones, so one URL covers a whole site.

siteprobe https://example.com/sitemap.xml
urls.txt · stdin

A plain-text list

One URL per line. Blank lines and # comments are ignored, and anything that is not an http(s) URL is skipped with a warning. Load it from a URL, a local file, or stdin.

siteprobe ./urls.txt
curl -s https://example.com/sitemap.xml | siteprobe -
-u, --url

URLs on the command line

-u can be repeated, and combined with a source. The URLs are merged and deduplicated.

siteprobe -u https://example.com/ -u https://example.com/pricing
02

Options.

Concurrency and rate limits, authentication, redirects, retries, and a threshold for slow responses. Most of these can also be set in a config file.

-c, --concurrency-limit

Concurrency limit

The number of requests in flight at once. The default is 4.

-l, --rate-limit

Rate limit

A cap for the whole run, written as requests per time span: -l 300/5m for 300 requests per 5 minutes, or -l 100/1h. Units are s, m, and h.

-t, --request-timeout · --retries

Timeouts and retries

A per-request timeout, 10 seconds by default, and an optional number of retries for network errors and 5xx responses.

--basic-auth · -H, --header

Authentication

Basic authentication, or any custom header such as a bearer token or a session cookie. -H can be repeated.

-f, --follow-redirects

Redirects

Redirects are only followed when enabled, up to 10 per URL. Basic auth credentials are not forwarded across a redirect.

-s, --slow-threshold

Slow threshold

A threshold in seconds, such as 3 or 0.05. Responses above it are listed in the report, and the run exits with code 3.

-a, --append-timestamp

Cache bypass

Appends a random timestamp to every URL, so that caches are bypassed and the origin answers each request.

-o, --output-dir

Save the documents

Writes every downloaded document to a directory, for inspection or as a static mirror of the site.

--config · .siteprobe.toml

Config file

Options can be set in .siteprobe.toml in the current directory, or in a file given with --config. Command-line flags take precedence.

03

Reports.

Every run prints statistics to the terminal. A report can also be written to a file, or printed as JSON to stdout.

-r, --report-path

CSV

One row per URL: status code, response time in milliseconds, and response size.

-j, --report-path-json

JSON

The full report, including the source of the URLs and every per-URL result.

--report-path-html

HTML

A single self-contained HTML file with a sortable table.

--json

JSON on stdout

Prints the JSON report to stdout and suppresses all other output, for piping into other tools.

04

Installation.

Available from Homebrew, PyPI, and crates.io. Each release also ships prebuilt binaries for macOS, Linux, and Windows.

Homebrew · macOS, Linux 
brew install bartTC/siteprobe/siteprobe
Run without installing 
uvx siteprobe https://example.com/sitemap.xml
pipx run siteprobe https://example.com/sitemap.xml
pip · pipx 
pipx install siteprobe
pip install siteprobe
Cargo 
cargo install siteprobe

Binaries and a shell installer for each release are on GitHub Releases. Building from source requires Rust 1.86 or newer.

git clone https://github.com/bartTC/siteprobe.git
cd siteprobe
cargo build --release