Skip to navigation
Information

robots.txt

Every Conduit RPC URL serves a robots.txt file that asks search engine crawlers not to crawl it. This applies to all networks and to custom RPC domains, and you don’t need to configure anything.

What Conduit serves

A GET request to /robots.txt on any Conduit RPC URL returns HTTP 200 with Content-Type: text/plain; charset=utf-8:

User-agent: *
Disallow: /

To check it on your own RPC URL, run:

curl -i https://rpc-[YOUR_NETWORK_SLUG].t.conduit.xyz/robots.txt

Why RPC URLs block crawlers

An RPC URL serves JSON-RPC over POST requests and has no pages for a crawler to index. Without a robots.txt, crawlers treat the whole host as crawlable. The file tells well-behaved crawlers to skip it.

Effect on your RPC traffic

The robots.txt file doesn’t affect your RPC traffic. It only answers GET and HEAD requests to /robots.txt. JSON-RPC requests, including a POST to /robots.txt, work the same as before.

Use an API key for reliable access

The public RPC URL is shared by everyone who calls it without a key, so it has the lowest throughput limit. For production apps, bots, and agents, get a free API key and append it to the RPC URL. An API key gives you:

  • Higher throughput — the Free Plan allows 10 times the public rate limit, and paid plans go higher. See Throughput.
  • Access to more methods — debug_* and trace_* methods and WebSocket connections require an API key.

Limitations

  • It doesn’t remove existing search results. Disallow stops crawlers from fetching your RPC URL, but a search engine can still list the URL if other sites link to it.
  • Only well-behaved crawlers follow it. Bots that ignore robots.txt aren’t affected. To control who can use your endpoint, see Access controls settings.
  • You can’t customize it. Every RPC URL serves the same file.