Aller au contenu principal
Standard llmstxt.org · GEO Specification 2026

Free llms.txt Validator & AI Crawler Access Checker

Test your /llms.txt file compliance in 1 click and verify if your robots.txt directives allow or block ChatGPT Search, Perplexity, Claude, and Gemini.

Quick examples:
[ 02 · THE TECHNICAL SPECIFICATION ]

Why Has llms.txt Become the Universal Sitemap for Generative AI?

Proposed by Jeremy Howard (creator of fast.ai and Answer.ai), the llms.txt standard solves a fundamental problem: when an AI agent or reasoning model (ChatGPT, Claude, Perplexity) visits your website, parsing megabytes of HTML, hydration JavaScript, tracking pixels, and ads consumes excessive context tokens and degrades response accuracy.

A standard /llms.txt file provides a clean, machine-readable plain text Markdown summary. It serves as an ultra-curated index containing your project name, a concise 1-2 sentence description, and structured links directly to your key pages, APIs, and documentation.

[ 03 · ROBOTS.TXT & AI GOVERNANCE ]

The Fatal Mistake: Blocking AI Search While Trying to Block AI Training

Many organizations rushed to add blanket Disallow: / rules to their robots.txt, unknowingly cutting off generative search engines that deliver qualified organic traffic. It is essential to differentiate between two distinct bot families:

1. Search & Citation Bots (Must Allow)

OAI-SearchBot, ChatGPT-User, PerplexityBot, and Claude-Web crawl the web in real-time when a user asks a question. If you block them, your company vanishes from AI answers and recommended referral sources.

2. Foundational Training Scrapers (Optional)

GPTBot, CCBot, or Bytespider scrape data to pre-train future model iterations. You can choose to restrict them without hurting your live visibility on ChatGPT Search or Perplexity.

[ 04 · PRODUCTION DEPLOYMENT ]

How to Deploy Your llms.txt File Across Modern Frameworks

Next.js (App Router)

Place the generated file directly inside your public/llms.txt folder. For dynamic generation powered by your CMS or database, create a Route Handler at app/llms.txt/route.ts returning a new Response(content, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } }).

Cloudflare Workers / Fastly / CDN

Set up an edge route or worker matching */llms.txt that serves static Markdown with a 24-hour cache header (Cache-Control: public, max-age=86400). This ensures instant response times (<20ms) for AI agents worldwide.

WordPress / Apache / Nginx

Upload the llms.txt file directly to your web root (public_html/llms.txt). Confirm that your server configuration does not redirect .txt URLs to WordPress 404 pages.

[ 05 · FREQUENTLY ASKED QUESTIONS ]

Questions & Answers About llms.txt & GEO

Does having a /llms.txt file guarantee ranking on ChatGPT Search?

No single file guarantees 100% inclusion, but /llms.txt dramatically lowers token ingestion costs and disambiguates your brand identity for AI agents. Paired with valid Schema.org JSON-LD and fast server-rendered content, it gives you a massive competitive advantage over competitors relying on bloated HTML.

Can I create both /llms.txt and /llms-full.txt?

Yes, and it is highly recommended. /llms.txt serves as the quick navigational index, while /llms-full.txt aggregates your complete documentation or product catalog into a single comprehensive Markdown document for models with 1M+ token context windows.

Why does this validator check Schema.org structured data?

Generative engines (like Perplexity and Google Gemini) use entity graphs to cross-reference data. If your site has rich JSON-LD markup declaring Organization, founder, and services, the AI model has high confidence in citing your platform without hallucinating.