> For a complete page index, fetch https://docs.conduit.xyz/llms.txt

# robots.txt

> Every Conduit RPC URL serves a robots.txt that asks search engine crawlers not to crawl it.

Every Conduit RPC URL serves a `robots.txt` file that asks search engine crawlers not to crawl it. This applies to all networks and to custom RPC domains, and you don't need to configure anything.

## What Conduit serves

A `GET` request to `/robots.txt` on any Conduit RPC URL returns HTTP `200` with `Content-Type: text/plain; charset=utf-8`:

```text
User-agent: *
Disallow: /
```

To check it on your own RPC URL, run:

```bash
curl -i https://rpc-[YOUR_NETWORK_SLUG].t.conduit.xyz/robots.txt
```

## Why RPC URLs block crawlers

An RPC URL serves JSON-RPC over `POST` requests and has no pages for a crawler to index. Without a `robots.txt`, crawlers treat the whole host as crawlable. The file tells well-behaved crawlers to skip it.

## Effect on your RPC traffic

The `robots.txt` file doesn't affect your RPC traffic. It only answers `GET` and `HEAD` requests to `/robots.txt`. JSON-RPC requests, including a `POST` to `/robots.txt`, work the same as before.

## Use an API key for reliable access

The public RPC URL is shared by everyone who calls it without a key, so it has the lowest throughput limit. For production apps, bots, and agents, [get a free API key](/rpc-nodes/getting-started/get-api-key) and append it to the RPC URL. An API key gives you:

* **Higher throughput** -- the Free Plan allows 10 times the public rate limit, and paid plans go higher. See [Throughput](/rpc-nodes/information/throughput).
* **Access to more methods** -- `debug_*` and `trace_*` methods and WebSocket connections require an API key.

## Limitations

* **It doesn't remove existing search results.** `Disallow` stops crawlers from fetching your RPC URL, but a search engine can still list the URL if other sites link to it.
* **Only well-behaved crawlers follow it.** Bots that ignore `robots.txt` aren't affected. To control who can use your endpoint, see [Access controls settings](/rpc-nodes/guides/access-controls-settings).
* **You can't customize it.** Every RPC URL serves the same file.