Robots and sitemap audit
$0.025 USDCCompare crawl-discovered pages with robots.txt rules and declared XML sitemap coverage.
Inspect the exact response envelope first. When you need current analysis for your own input, one successful x402 call costs $0.005 USDC on Base or Solana.
Demonstration data only. The paid endpoint returns a fresh result and settlement receipt.
{
"request_id": "018f1f54-7f38-7ba2-8dc3-5f90272d9f1a",
"service": "web.links",
"version": "1",
"data": {
"url": "https://example.com/",
"origin": "https://example.com",
"count": 1,
"internal": 0,
"external": 1,
"links": [
{
"url": "https://iana.org/domains/example",
"text": "Learn more",
"external": true
}
],
"crawl": {
"enabled": true,
"maxPages": 3,
"maxDepth": 2,
"pagesAttempted": 1,
"pagesFetched": 1,
"pagesFailed": 0,
"pagesSkipped": 0,
"truncated": false,
"sameOriginOnly": true,
"linkLimitPerPage": 500,
"robots": {
"respected": true,
"status": "http_404",
"url": "https://example.com/robots.txt"
}
},
"pages": [
{
"url": "https://example.com/",
"depth": 0,
"discoveredFrom": null,
"statusCode": 200,
"title": "Example Domain",
"bytes": 559,
"elapsedMs": 41,
"bodySha256": "ff67a9d764d6a2367a187734e697f6a53217db9a21c101d410a113ca871a299d",
"linkCount": 1,
"internal": 0,
"external": 1
}
],
"errors": [],
"skipped": []
},
"meta": {
"duration_ms": 42,
"cached": false,
"sources": [],
"warnings": [],
"next_actions": [
{
"service": "web.robots-sitemap",
"title": "Robots and sitemap audit",
"reason": "Compare crawl-discovered pages with robots.txt rules and declared XML sitemap coverage.",
"method": "POST",
"path": "/v1/web/robots-sitemap",
"price": "$0.025",
"input_template": {
"url": "$request.url"
},
"use_output": "Reconcile result.data.sitemap.urlCount and robots rules with the crawl inventory.",
"handoff_path": "/catalog/web.robots-sitemap",
"checkout_path": "/try/web.robots-sitemap",
"sample_path": "/samples/web.robots-sitemap",
"maximum_atomic_usdc": "25000",
"handoff_payment_required": false,
"executes_automatically": false,
"authorization_required": true
}
]
}
}{
"url": "https://example.com",
"include_external": true,
"max_pages": 3,
"max_depth": 2,
"respect_robots": true
}Extract and classify links from one public page or safely crawl up to 10 same-origin pages into an agent-ready site map with page provenance, depth limits, and robots.txt enforcement.
POST /v1/web/links
Each option opens a new checkout with its own exact price and requires separate buyer authorization. Nothing runs automatically.
Compare crawl-discovered pages with robots.txt rules and declared XML sitemap coverage.
The checkout verifies the request, selected network, USDC asset, recipient, and price before opening a wallet.