The borderbot page fetcher
When someone asks border.bot to classify a product from its web address, border.bot fetches that one public page to read the product details. This page explains what the fetcher does, how to recognise it and how to block it.
How to recognise it
Its user agent is:
Mozilla/5.0 (compatible; borderbot/1.0; +https://border.bot/bot) AppleWebKit/537.36 (KHTML, like Gecko)What it fetches
- Only the page address a person, or the assistant working for them, submits to border.bot through the dashboard, the API, the MCP server or the free tools.
- It reads the product’s title, brand, price, materials and any stated country of origin, to classify the product and work out duties and taxes.
- It doesn’t follow links, crawl your site or build an index, and page content is not used to train AI models.
How often
- One request per submission. The same address is answered from a cache for 15 minutes, so repeat submissions don’t reach your site again.
- Requests come from Cloudflare’s network and give up after 10 seconds.
How to block it
- Every fetch is asked for by a person, like a browser visit, so the fetcher doesn’t read robots.txt.
- To block it, deny requests whose user agent contains borderbot at your firewall or CDN, or email us and we’ll stop fetching your domain.
Questions, or want us to stop fetching your domain? Email support@border.bot.