# ---------------------------------------------------------------------------- # Content Signals Policy (contentsignals.org, IETF draft-romm-aipref-contentsignals) # # As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a content-signal = yes, you may collect content for the corresponding use. # (b) If a content-signal = no, you may not collect content for the corresponding use. # (c) If the operator does not include a content signal for a corresponding use, # the operator neither grants nor restricts permission for that use. # # The content signals and their meanings are: # search: building a search index and providing search results (hyperlinks and # short excerpts). Search does not include AI-generated search summaries. # ai-input: inputting content into one or more AI models (retrieval augmented # generation, grounding, or other real-time use for generative answers). # ai-train: training or fine-tuning AI models. # # TerraLab grants all three. We want to be found, cited, and learned from. # # A crawler obeys ONE group: the most specific one that names it. A named group # REPLACES the wildcard group for that crawler, it does not extend it. So every # named group below repeats the full rule set, the Content-Signal AND both # Disallow lines. Adding a line to the wildcard group means adding it to all # twenty named groups in the same edit, or the AI crawlers keep crawling what # Google no longer does. # ---------------------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /_next/static/ Allow: /_next/image Disallow: /_next/data/ Disallow: /api/ # /docs/ is the developer docs since 2026-10-27 and must stay crawlable. From # May 2026 it answered 410 so search engines dropped the retired plugin docs; # the old /docs/ URLs still 301 to the product pages (src/middleware.ts). # # /admin/ is not listed either, and there is nothing to list: the route does not # exist and returns 404. This file is public, so a Disallow line publishes the # path it names. Naming a route we do not have advertised an attack surface and # protected nothing. # AI Crawlers - explicitly allowed, with the same rules as everyone else. User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: anthropic-ai Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Claude-Web Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Claude-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: cohere-ai Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Meta-ExternalAgent Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Meta-ExternalFetcher Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: YouBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /_next/data/ Disallow: /api/ # xAI's crawler is deliberately NOT listed here (checked 23 Sep 2026): x.ai and # grok.com's own robots.txt name no crawler of their own, only the other # vendors' bots above, and xAI has published no user-agent string for it. # Third-party lists guess "GrokBot" or "xAI-Bot", but none can point to where # that name comes from, and live traffic analyses report Grok fetches arriving # as unidentified browser user agents. Add a named group once xAI documents # one; a guessed token here would either miss the real crawler entirely or, # worse, the group-replaces-wildcard rule explained at the top of this file # would apply to a name nothing ever sends. # Sitemap Sitemap: https://terra-lab.ai/sitemap.xml # Machine-readable entry points for agents and LLMs: # https://terra-lab.ai/llms.txt # https://terra-lab.ai/llms-full.txt # https://terra-lab.ai/.well-known/api-catalog # https://terra-lab.ai/.well-known/agent-skills/index.json # Blog posts also answer Accept: text/markdown with a clean markdown rendition. # Blog RSS, listed for the humans reading this file. robots.txt has no directive # for feeds (only Sitemap is parsed), so this stays a comment; readers and QGIS # Planet find it through the in every page head. # Feed: https://terra-lab.ai/feed.xml (English) # Feed: https://terra-lab.ai/fr/feed.xml (French)