acurl documentation

acurl fetches a URL once per agent profile, compares every response with a real Chrome visit and tells you which AI agents and crawlers can read the page, and what to fix when they can't.

Installation

acurl is a single static binary for macOS, Linux and Windows (amd64 and arm64).

Check the install with acurl --version. Shell completions: acurl completion zsh > "${fpath[1]}/_acurl" (see completion).

Quick start

Point acurl at any page. A scheme is optional: https:// is assumed.

$ acurl shop.example/products/wool-runner
Fetching https://shop.example/products/wool-runner with 22 profiles...

PROFILE            STATUS  TIME    RESULT             DETAILS
Chrome (Browser)   200     412ms   allowed            Wool Runner | Example Store
ChatGPT-User       403     51ms    challenged         challenge: Cloudflare
Claude-User        200     388ms   allowed            OK
OAI-SearchBot      200     97ms    cloaked            31% similar to browser
GPTBot             200     365ms   robots-disallowed  robots.txt: disallowed
...

AI readiness 71/100 (C)
  ✗ AI assistants can fetch the page            16.7/25  ChatGPT-User (challenged)
  ✓ Content is in the HTML without JavaScript   15.0/15
...
How to improve
  1. AI assistants can fetch the page
     Allow ChatGPT-User, Claude-User and Perplexity-User in your WAF or bot-management rules...

The exit code is 1 because an agent was challenged. Change that with --fail-on.

Reading the output

Below the table: the AI readiness score with each check, and a fix for every failed check.

Usage patterns

Check only the agents you care about

acurl --category ai-assistant,ai-search shop.example     # categories keep the browser baseline
acurl --agents "Chrome (Browser),ChatGPT-User,Claude-User" shop.example
acurl profiles list                                      # every profile and its category

See exactly what an agent missed

acurl --diff OAI-SearchBot shop.example/products/1      # visible-text diff vs the browser, on stderr
acurl -f html -o report.html shop.example/products/1     # shareable report with diffs

Raw output for scripts

acurl -q -f json shop.example | jq '.results[] | select(.classification != "allowed") | .profile'
acurl -q -f json shop.example | jq '.score.value'
acurl -q -f csv  shop.example > agents.csv

Robots and llms.txt without fetching the page

acurl robots shop.example/products/1
acurl robots -f json shop.example | jq '.agents[] | select(.allowed | not) | .token'

Whole site

acurl batch urls.txt                                     # one URL per line, # comments allowed
acurl batch --sitemap auto:shop.example --limit 100      # sitemaps listed in robots.txt
cat urls.txt | acurl batch -f junit -o acurl.xml

Staging, logins and regions

acurl -H "Cookie: country=US" -H "Authorization: Basic dXNlcjpwYXNz" staging.shop.example
acurl --proxy socks5://127.0.0.1:1080 shop.example       # check from another network
acurl -k https://localhost:8443                          # self-signed certificates

Isolate the user agent

acurl --tls chrome shop.example     # bots use Chrome's TLS fingerprint too: only the UA differs
acurl -I shop.example               # HEAD requests: status and headers only

Verdicts

Each result gets exactly one verdict, checked in this order:

Agent profiles

A profile is a user agent string, optional headers and the robots.txt token the agent obeys. The catalog has 22 profiles in six categories:

* robots.txt-only tokens: checked against robots.txt, never fetched.

Browser profiles use Chrome's TLS, HTTP/2 and client hints. Bot profiles send only their own headers over Go's TLS stack, like most real crawlers. Bring your own catalog with --profiles my-profiles.yaml:

profiles:
  - name: "InternalBot"
    category: other
    user_agent: "InternalBot/2.0 (+https://example.com/bot)"
    robots_key: "InternalBot"
    headers:
      X-Team: "growth"

AI readiness score

A 0-100 score from weighted checks that only use what was observed in the run. Checks that don't apply (no product on the page, no agents of that category) are skipped and the rest re-weighted. Blocking training crawlers costs nothing: that is a business decision.

Grades: A 90+, B 80+, C 65+, D 50+, F below. If the browser baseline itself is blocked, the page is not scored: the site is blocking your network, not the agents.

Exit codes and --fail-on

The default is --fail-on blocked,challenged,error. Combine blocked, challenged, rate-limited, cloaked, robots, error and score<N, or use none.

acurl --fail-on blocked,challenged,cloaked,score<80 shop.example

Output formats

Save JSON once and render it again later without re-fetching: acurl report run.json -f html -o run.html.

CI and GitHub Actions

- uses: tragent-io/acurl@v1
  with:
    urls: |
      https://staging.shop.example/
      https://staging.shop.example/products/best-seller
    fail-on: blocked,challenged,score<80

The action installs acurl, writes a job summary, and uploads SARIF to code scanning when the job has security-events: write. Without the action:

go install github.com/tragent-io/acurl/cmd/acurl@latest
acurl batch urls.txt -f sarif -o acurl.sarif --fail-on blocked,challenged

Results from cloud runners can differ from your laptop's: many WAFs score datacenter IP ranges. If the browser baseline is challenged, the runner is being blocked, not the agents.

robots.txt and llms.txt

acurl robots fetches only robots.txt and llms.txt, and reports per agent:

• allow or disallow using Google's robots.txt matcher, with RFC 9309 failure rules (4xx allows all, 5xx or an unreachable server means full disallow)
• Content-Signal directives such as search=yes, ai-train=no
• sitemaps listed in robots.txt
• /llms.txt and /llms-full.txt: presence, title, sections, links and format problems

The main command also reads X-Robots-Tag and <meta name="robots">, including agent-specific tags like <meta name="GPTBot" content="noindex">.

JavaScript rendering

Most AI agents do not run JavaScript. acurl flags client-rendered shells (an empty app root, a "please enable JavaScript" notice, almost no text). With --render, it loads the page in a local headless Chrome and reports how much visible text only exists after scripts run:

$ acurl --render shop.example
JavaScript rendering
  64% of the visible text only appears after JavaScript runs (812 -> 2254 chars).
  Agents that do not run JavaScript miss it.

Rendering needs Chrome or Chromium installed. On Ubuntu 23.10+, Chrome's sandbox needs sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0. Inside a container, set ACURL_CHROME_NO_SANDBOX=1 instead.

Watching for changes

acurl watch shop.example --interval 1h \
  --webhook https://hooks.slack.com/services/... --webhook-format slack

# from cron: run once, compare with the last run, alert on changes
acurl watch shop.example --interval 0 --state /var/lib/acurl/state.json --webhook https://ops.example/hook

Reports every change in an agent's verdict and score moves of at least --min-score-delta points.

Web Bot Auth signing

Sign bot-profile requests with HTTP Message Signatures (RFC 9421, Ed25519), as proposed by Web Bot Auth. Use it to test your verifier or your own agent:

acurl keygen -o mybot.key           # prints the key directory JSON to publish
acurl --sign mybot.key --signature-agent https://bot.example shop.example

Requests carry Signature-Agent, Signature-Input (covering @authority and signature-agent, with created, expires, nonce and tag="web-bot-auth") and Signature. Signing is verified against the RFC 9421 Ed25519 test vector.

AI assistants (MCP)

claude mcp add acurl -- acurl mcp

For Claude Desktop or Cursor, add {"mcpServers": {"acurl": {"command": "acurl", "args": ["mcp"]}}}. Tools: check_url, readiness_score, robots_audit, list_profiles. Private and internal addresses are blocked unless you start it with --allow-private.

Web UI and API

acurl serve                   # http://127.0.0.1:8080
curl -s -X POST localhost:8080/api/check -d '{"url":"shop.example","categories":["ai-assistant"]}'

Configuration

acurl init writes a commented .acurl.yaml. Any flag can be set by its long name. Precedence: flags, then ACURL_* environment variables, then the config file.

category: [ai-assistant, ai-search]
fail-on: blocked,challenged,score<75
timeout: 20s
header:
  - "Cookie: country=US"

acurl looks for ./.acurl.yaml, then ~/.config/acurl/config.yaml, or the file given with --config. Unknown keys are an error, so typos never pass silently.

Go library

import "github.com/tragent-io/acurl/pkg/acurl"

c, err := acurl.New(acurl.Options{BlockPrivate: true})
if err != nil { return err }
rep, err := c.Check(ctx, "https://shop.example/products/1")
if err != nil { return err }
fmt.Println(rep.Score.Value, rep.Summary.Blocked)

Command reference

Generated from the CLI itself with acurl reference, so it always matches the installed version.

acurl <url> [flags]

  acurl https://shop.example/products/1
  acurl --category ai-assistant,ai-search shop.example
  acurl --agents GPTBot,ClaudeBot --diff GPTBot shop.example
  acurl -f json -o report.json --fail-on blocked,score<80 shop.example
  acurl --render shop.example          # measure JavaScript-only content

acurl batch [file] [flags]

  acurl batch urls.txt
  acurl batch products.csv -f json -o report.json
  cat urls.txt | acurl batch
  acurl batch --sitemap https://shop.example/sitemap.xml --limit 20
  acurl batch --sitemap auto:shop.example

acurl completion bash

acurl completion fish [flags]

acurl completion powershell [flags]

acurl completion zsh [flags]

acurl init [flags]

  acurl init
  acurl init --force

acurl keygen [flags]

  acurl keygen -o mybot.key
  acurl keygen --key mybot.key > http-message-signatures-directory

acurl mcp [flags]

  claude mcp add acurl -- acurl mcp
  acurl mcp --timeout 20s

acurl profiles list [flags]

  acurl profiles list
  acurl profiles list --category ai-assistant -f json

acurl profiles show <name>

  acurl profiles show GPTBot

acurl report <report.json> [flags]

  acurl -f json -o run.json https://shop.example
  acurl report run.json -f html -o run.html

acurl robots <url> [flags]

  acurl robots shop.example
  acurl robots --category ai-training -f json shop.example/products/1

acurl serve [flags]

  acurl serve
  acurl serve --addr 0.0.0.0:8080 --render

acurl watch <url>... [flags]

  acurl watch https://shop.example --interval 1h --webhook https://hooks.slack.com/... --webhook-format slack
  acurl watch https://shop.example --interval 0 --state acurl-state.json   # from cron

acurl [command] [flags]

acurl is a free, open-source tool by tragent.io. Found a mistake in these docs? Edit this page.