If you found this in your logs

You saw a request carrying this user agent:

SleuthX-auto-capture/1.0 (+https://sleuthx.ai/bot; mailto:support@sleuthx.ai)

It came from SleuthX, a product of SleuthX, Inc.. To stop it, add the two lines under How to block it to your robots.txt. We honour that file, and the rest of this page says exactly what that means and where the limits are.

What it does

SleuthX-auto-capturepreserves individual web pages that a case's own searches surfaced. Someone working a case runs a search, and the first result that looks relevant is fetched once and stored as evidence with its hash and its retrieval time.

It is not a general web crawler. It does not spider your site, it does not follow your internal links looking for more pages, and it does not come back on a schedule. There is no index being built. A page is fetched because one specific investigation surfaced that one specific address.

Content fetched this way is stored for the case it belongs to and is not used to train models.

Which requests carry this token

Only automated ones. This token identifies the route where no person chose the page: a rule picked the first search result and fetched it without anyone looking at it first. That is the route this page describes, and it is the route that reads your robots.txt.

How we read robots.txt

Before fetching a page, this route requests /robots.txt from that origin and applies it under RFC 9309. If your file disallows the path, the page is not fetched. Nothing is retrieved, nothing is stored, and the case records that the origin declined.

We match the token SleuthX-auto-capture and, when no group names it, the wildcard *group. We do not read rules addressed to anyone else. A file that blocks a different company's crawler by name has said nothing about us, and we do not treat that as permission either.

Where the limits are

Three of these are worth stating plainly, because "we obey robots.txt" is true only up to them.

Caching. We cache your robots.txt for up to 24 hours, and we honour a shorter Cache-Control: max-age if your server sends one. A rule you add today can take up to a day to take effect.
Server errors. If your robots.txt cannot be read at all, we do not fetch the page. If it was readable before and your server is now failing, we keep using the last copy we read successfully, for up to 30 days. This follows Google's published behaviour, and it means a rule you removed during an outage may keep applying until your server recovers.
Rate limiting. If your robots.txt returns HTTP 429, we treat that as unreadable and do not fetch. The standard would permit us to read a 429 as "no rules apply". We think that reading is wrong, so we decline instead.

We do not support Crawl-delay. It is not part of RFC 9309, and this route fetches at most a handful of pages per case, so there is no crawl rate for it to govern.

How to block it

Add this to the robots.txt at the root of your domain:

User-agent: SleuthX-auto-capture
Disallow: /

To block a section instead of the whole site, name the path:

User-agent: SleuthX-auto-capture
Disallow: /members/

A wildcard rule works too. If your file already contains User-agent: * with a Disallow that covers the path, that rule applies to us and you need to change nothing.

Verifying a request really came from us

A user agent string is not proof of anything. Anyone can send this one, and some people will. We do not currently publish an IP range you could check a request against, so a request claiming to be SleuthX-auto-capture cannot be confirmed as ours, and we cannot disprove one either.

That is a real gap and we would rather say so than imply a verification path that does not exist. If you are seeing traffic under this name that looks wrong, write to us with the timestamps and the source addresses and we will tell you whether it was us.