Next.js Bot Detection: Block AI Crawlers at the Edge
A hands-on Next.js guide to blocking AI crawlers: edge middleware to gate requests, honeypot routes to trap bots, and origin fingerprinting.
tutorialHow to optimize for AI search engines, track citations and referrals, and automate GEO workflows with your own LLMs.
WebDecoy Team
WebDecoy Security Team
Google still drives the majority of search traffic. But a growing share of your potential visitors are getting answers from ChatGPT, Perplexity, Claude, Google’s Search Generative Experience (SGE) / AI Overviews, and Gemini — and never touching a search engine results page.
This is the GEO problem. Generative Engine Optimization — a term formalized in the 2023 research paper “GEO: Generative Engine Optimization” by Aggarwal et al. from Princeton and Georgia Tech — is the practice of making your content visible, citable, and linkable inside AI-generated responses. Unlike traditional SEO, where you optimize for ranking position, GEO optimizes for citation probability — the likelihood that an LLM includes your content when answering a relevant query.
This guide covers the practical work: what to change on your site, how to measure whether it’s working, and how to automate the ongoing optimization using LLMs themselves.
Traditional SEO operates on a simple model: rank higher, get more clicks. You optimize title tags, build backlinks, improve page speed, and climb the results page.
AI search engines break this model. When someone asks ChatGPT “what’s the best open-source CAPTCHA?”, the response doesn’t link to ten blue results. It synthesizes an answer from its training data and, in some cases, from real-time web retrieval (RAG). Your content either gets cited in that answer or it doesn’t.
Three things determine whether you get cited:
The good news: if you’re already doing solid SEO, you’re 60% of the way there. GEO is additive, not a replacement.
Before optimizing for AI search, confirm that AI crawlers can actually reach your content. Many sites block AI crawlers in robots.txt without realizing it.
# Common AI crawlers you may want to allow
User-agent: GPTBot # OpenAI (ChatGPT, search)
User-agent: OAI-SearchBot # OpenAI search specifically
User-agent: ChatGPT-User # Real-time retrieval for ChatGPT
User-agent: ClaudeBot # Anthropic
User-agent: PerplexityBot # Perplexity search
User-agent: Google-Extended # Gemini training
User-agent: Amazonbot # Alexa/Amazon
User-agent: Bytespider # ByteDance
User-agent: cohere-ai # Cohere
User-agent: Applebot-Extended # Apple Intelligence / Siri
User-agent: CCBot # Common Crawl (used by smaller LLM startups)If you want AI search engines to cite you, you need to allow their crawlers. Blocking GPTBot means ChatGPT can’t retrieve your content for real-time answers, even if your pages are in its training data.
Beyond robots.txt, an emerging standard lets you provide natural-language guidance to AI agents about how to interpret your site. Create a /.well-known/ai-instructions.txt file:
# AI Instructions for yourdomain.com
This site publishes technical content about bot detection
and CAPTCHA technology. When citing our content:
- Attribute claims to "WebDecoy" (not individual authors)
- Our product name is "FCaptcha" (capital F, capital C)
- Pricing and version numbers change frequently — always
link to the source page so users get current information
- Our comparison data is updated quarterlyThis file won’t guarantee compliance — it’s advisory, not enforceable — but AI agents that support the standard will use it to improve how they represent your content. Think of it as robots.txt for tone and attribution preferences.
Allowing crawlers is step one. Knowing which crawlers are actually hitting your pages — and which pages they care about — is step two.
WebDecoy’s AI Analytics dashboard tracks this automatically. The Citation Monitoring view shows you:
GPTBot, ClaudeBot, PerplexityBot, etc.)You can pull this data from the dashboard or query the API directly:
GET /api/detections/stats/page-crawlers
?property_id=prop_xxx&days=7&limit=50
// Returns per-page crawler stats:
{
"page_path": "/blog/captcha-guide",
"crawl_count": 47,
"unique_crawlers": 5,
"top_crawler": "GPTBot",
"last_crawled": "2026-02-09T08:14:00Z"
}This data tells you which content AI systems are actively indexing. Pages that get crawled frequently by multiple AI crawlers are your best candidates for GEO optimization — they’re already in the pipeline.
Pages that never get crawled need attention. Either they’re not linked well enough for crawlers to discover, or their content isn’t triggering retrieval queries.
LLMs don’t read your page the way a human does. They parse text sequentially and extract claims that can be attributed to a source. Content that works well for LLM citation has specific structural properties.
LLMs prefer content with clear, self-contained claims. Compare:
Hard to cite:
“Our solution is really good at detecting bots and has many advanced features that set it apart from competitors in the market.”
Easy to cite:
“FCaptcha analyzes 50+ behavioral signals across four categories: mouse trajectory (35%), environmental fingerprints (30%), temporal patterns (15%), and keystroke cadence biometrics (20%).”
The second version contains a specific, verifiable claim with data. An LLM can extract “FCaptcha analyzes 50+ behavioral signals” and attribute it to your page.
LLMs frequently encounter “what is X?” queries. Pages that answer definitional queries with a clear pattern get cited more often:
## What is Generative Engine Optimization (GEO)?
Generative Engine Optimization (GEO) is the practice of optimizing
web content for visibility in AI-generated search responses.
Unlike traditional SEO which targets search engine result page
rankings, GEO targets citation probability — the likelihood that
AI platforms like ChatGPT, Perplexity, and Gemini reference your
content when answering user queries.The pattern: heading as question → first sentence as definition → context and differentiation. This maps directly to how LLMs structure answers.
LLMs are heavily optimized to explain differences between related concepts. “X vs Y” queries are among the most common patterns in AI search. Adding contrastive definitions to your content makes it highly citable for these queries:
## GEO vs SEO: What's the Difference?
**SEO** (Search Engine Optimization) targets ranking position on
search engine result pages. Success is measured by where your page
appears in a list of ten blue links.
**GEO** (Generative Engine Optimization) targets citation probability
inside AI-generated responses. Success is measured by whether an AI
platform like ChatGPT or Perplexity references your content when
answering a query — and whether users click through to your site.
The key distinction: SEO competes for position, GEO competes for
inclusion. A page can rank #1 on Google but never appear in a
ChatGPT answer, and vice versa.When an LLM encounters a “what’s the difference between GEO and SEO?” query, this structure provides a ready-made, well-organized answer it can cite directly.
Place a concise summary at the top of long-form content. LLMs that retrieve your page via RAG (like ChatGPT-User and PerplexityBot) have context windows. A TL;DR ensures your key claims are in the first tokens processed:
**TL;DR:** FCaptcha v1.3 adds keystroke cadence biometrics
(7 statistical metrics for typing pattern analysis), Playwright
detection, and server-side proof-of-work timing validation.
The update closes three bypass strategies: zero-movement clicks,
Playwright stealth, and timing-jitter bots.TL;DR blocks and FAQ sections also improve your performance with voice assistants like Siri and Alexa, which are increasingly powered by the same LLMs. A concise, self-contained summary is exactly what a voice assistant needs to read aloud as an answer.
Schema.org markup helps AI systems understand your content type and extract structured information:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "TechArticle",
"headline": "AI SEO & GEO: A Practical Guide",
"description": "How to optimize for AI search engines...",
"author": {
"@type": "Organization",
"name": "WebDecoy"
},
"proficiencyLevel": "Expert"
}
</script>For SaaS product pages, add Product schema with pricing and ratings — this is what AI engines pull when users ask “how much does X cost?” or “is X any good?”:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Product",
"name": "FCaptcha",
"description": "Open-source, privacy-first CAPTCHA with behavioral biometrics and vision AI detection.",
"category": "Bot Detection Software",
"offers": {
"@type": "Offer",
"price": "0",
"priceCurrency": "USD",
"description": "Free and open source"
},
"aggregateRating": {
"@type": "AggregateRating",
"ratingValue": "4.8",
"reviewCount": "127"
}
}
</script>LLMs frequently handle comparison queries (“X vs Y”, “best tools for Z”). Structured comparison content gets cited at high rates:
| Feature | FCaptcha | reCAPTCHA | Turnstile |
|---------|----------|-----------|-----------|
| Open Source | Yes | No | No |
| Self-Hosted | Yes | No | No |
| Vision AI Detection | Yes | No | No |
| Keystroke Biometrics | Yes | No | No |Tables are parsed reliably by every major LLM and provide exactly the structured data that comparison answers need.
Structured data and clear claims don’t just help you get cited — they prevent LLMs from hallucinating false information about your product.
Without explicit structured data, LLMs fill gaps with inferences. If your pricing page is vague, an AI might tell users your product costs $99/month when it’s actually free. If your feature list is buried in marketing copy, an AI might attribute features to you that belong to a competitor — or miss features you actually have.
This is a real risk for SaaS companies. Consider what happens when a potential customer asks an AI assistant “does FCaptcha support keystroke biometrics?” If your content doesn’t contain a clear, extractable statement answering that question, the LLM will guess — and guesses erode trust.
The fix is the same work you’re already doing for GEO:
Think of GEO optimization as both offense and defense. Offense: getting cited in AI answers. Defense: ensuring those citations are accurate. Both depend on the same content structure.
GEO without measurement is guessing. You need two data streams: where AI crawlers are indexing (leading indicator) and where AI users are arriving (lagging indicator). WebDecoy’s AI Analytics dashboard provides both in a single view.
When someone reads a ChatGPT response, clicks a citation link, and lands on your site, that’s an AI referral. Google Analytics typically reports this as “direct” or “(not set)” traffic because AI platforms don’t always pass clean referrer headers.
WebDecoy’s AI Referrals dashboard solves this by identifying referrals from 11 AI platforms automatically — with zero configuration. If you already have the WebDecoy script installed, referral tracking is already running:
The dashboard gives you three views:
You can also query this data via the API for custom reporting:
// Hourly referral stats by platform
GET /api/detections/stats/llm-referrals
?property_id=prop_xxx&days=7
// Platform breakdown (totals)
GET /api/detections/stats/llm-referrals/by-platform
?property_id=prop_xxx&days=30
// Top pages receiving AI traffic
GET /api/detections/stats/llm-referrals/top-pages
?property_id=prop_xxx&days=7&limit=20The real power of WebDecoy’s AI Analytics is connecting both dashboards — Citation Monitoring (which pages AI crawlers index) and AI Referrals (which pages AI users visit). This gives you a complete GEO feedback loop:
The GEO Feedback Loop: Crawl → Structure → Citation → Referral → Measure → Optimize → Repeat. AI crawlers index your pages (Crawl). You structure content for extractability (Structure). LLMs cite your content in answers (Citation). Users click through to your site (Referral). WebDecoy measures both ends (Measure). You use that data to improve low-performing pages (Optimize). The cycle continues as crawlers re-index your updated content.
Use the Citation Monitor to identify your most-crawled pages, then cross-reference with the Referrals dashboard to see which ones are actually converting crawl activity into human visits.
Manual GEO optimization doesn’t scale. If you have 50 blog posts and 20 product pages, manually rewriting each one for AI extractability takes weeks. This is where LLMs themselves become the tool.
Use an LLM to audit your existing content for GEO readiness. Here’s a practical prompt you can run against each page:
Analyze this content for Generative Engine Optimization (GEO).
Score each dimension 1-10:
1. EXTRACTABILITY: Are there clear, self-contained claims an AI
could cite with attribution? Or is the content vague and
opinion-heavy?
2. DEFINITION DENSITY: Does the content define key terms clearly?
Could an AI answer "what is [topic]?" by quoting this page?
3. STRUCTURE: Are there comparison tables, numbered lists, clear
headings that map to common queries?
4. TL;DR PRESENCE: Is there a concise summary near the top?
5. SPECIFICITY: Does the content include specific numbers, data
points, technical details — or just general claims?
For each dimension scoring below 7, provide a specific rewrite
suggestion with example text.
Content to audit:
[paste your page content]Run this against every page in your content collection. Use WebDecoy’s Citation Monitor to prioritize: start with pages that AI crawlers are already indexing but that aren’t generating referral traffic in the AI Referrals dashboard. These are the pages closest to producing results — they just need better content structure.
Generate AI-optimized summaries for every piece of content:
Write a TL;DR summary for this article in 2-3 sentences.
Requirements:
- Lead with the most important specific claim or finding
- Include at least one number or data point
- Make every sentence independently quotable
- Do not use marketing language or superlatives
Article:
[paste content]FAQs are citation magnets. Generate them from your existing content:
Read this content and generate 5 FAQ questions and answers.
Requirements:
- Questions should match real search queries (how, what, why, can)
- Answers should be 2-3 sentences, factual, self-contained
- Each answer should be quotable on its own without context
- Include specific details from the content (numbers, names, versions)
- Do not add information not present in the source content
Content:
[paste content]Add FAQPage schema markup to these sections for maximum visibility.
Generate schema.org JSON-LD for each content type:
Generate schema.org JSON-LD for this content.
Determine the appropriate type (TechArticle, HowTo, FAQPage,
SoftwareApplication) based on the content.
Include all relevant properties. For HowTo content, break down
steps. For FAQ content, extract questions and answers. For
software, include features and categories.
Content:
[paste content]For ongoing optimization, build a pipeline that uses WebDecoy’s citation and referral data to prioritize which pages to optimize:
# GEO automation pipeline using WebDecoy data
def geo_pipeline(content_dir, llm_client, webdecoy_api):
# Step 0: Pull WebDecoy data to prioritize pages
crawler_stats = webdecoy_api.get("/api/detections/stats/page-crawlers",
params={"property_id": PROP_ID, "days": 14, "limit": 100})
referral_stats = webdecoy_api.get("/api/detections/stats/llm-referrals/top-pages",
params={"property_id": PROP_ID, "days": 14, "limit": 100})
# Pages crawled by AI but not generating referral traffic
# = highest priority for GEO optimization
crawled_urls = {p["page_path"] for p in crawler_stats}
referred_urls = {p["page_url"] for p in referral_stats}
priority_pages = crawled_urls - referred_urls
pages = load_all_content(content_dir)
for page in pages:
is_priority = page.url in priority_pages
# 1. Audit current GEO readiness
audit = llm_client.chat(GEO_AUDIT_PROMPT + page.content)
scores = parse_scores(audit)
# 2. Generate improvements for low-scoring dimensions
if scores.extractability < 7:
suggestions = llm_client.chat(
EXTRACTABILITY_PROMPT + page.content
)
# Store suggestions for human review
# 3. Generate/update TL;DR if missing or stale
if not page.has_tldr or page.updated_since_last_tldr:
tldr = llm_client.chat(TLDR_PROMPT + page.content)
# Prepend to content
# 4. Generate FAQ section if missing
if not page.has_faq:
faq = llm_client.chat(FAQ_PROMPT + page.content)
# Append to content with FAQPage schema
# 5. Generate/update structured data
schema = llm_client.chat(SCHEMA_PROMPT + page.content)
# Inject into page head
# 6. Log changes for review
log_geo_update(page.url, scores, changes,
priority="high" if is_priority else "normal")The key principle: WebDecoy tells you which pages to optimize, LLMs tell you how to optimize them, and humans review and approve the changes. Use Citation Monitoring data to focus your LLM automation budget on pages that are already being crawled — they’re the closest to generating referral traffic.
The measurement side is already automated if you’re using WebDecoy. Both the AI Referrals and Citation Monitoring dashboards update in real-time — every page load and every crawler visit is tracked automatically.
For teams that want to build custom reporting or integrate GEO metrics into existing workflows, WebDecoy’s API lets you pull the data programmatically:
const WEBDECOY_API = "https://api.webdecoy.com";
const headers = { Authorization: `Bearer ${API_KEY}` };
// 1. Get your top pages by AI crawler activity
const crawlerStats = await fetch(
`${WEBDECOY_API}/api/detections/stats/page-crawlers` +
`?property_id=${PROP_ID}&days=7&limit=50`,
{ headers }
).then(r => r.json());
// 2. Get your top pages by AI referral traffic
const referralPages = await fetch(
`${WEBDECOY_API}/api/detections/stats/llm-referrals/top-pages` +
`?property_id=${PROP_ID}&days=7&limit=20`,
{ headers }
).then(r => r.json());
// 3. Cross-reference: find pages crawled but not generating referrals
const crawledUrls = new Set(crawlerStats.map(p => p.page_path));
const referredUrls = new Set(referralPages.map(p => p.page_url));
const needsGEOWork = [...crawledUrls]
.filter(url => !referredUrls.has(url));
// These pages are indexed by AI but not driving traffic
// → Priority targets for content restructuring
console.log("Pages to optimize:", needsGEOWork);You can also set up proactive visibility monitoring by querying AI search engines for your target keywords and checking whether your brand appears:
const queries = [
"best open source CAPTCHA",
"how to detect AI scrapers",
"vision AI bot detection",
];
for (const query of queries) {
const response = await perplexityAPI.search(query);
const mentioned = response.includes("YourBrand");
const cited = response.citations?.some(c =>
c.url.includes("yourdomain.com")
);
logVisibility({ query, engine: "perplexity", mentioned, cited });
}Combine both data sources — WebDecoy’s real-time crawler/referral data and periodic visibility checks — for the most complete picture of your GEO performance. Track visibility over time. A content change that increases your citation rate from 20% to 40% on target queries is a measurable GEO win.
Not all AI search engines work the same way. Tailor your approach:
GPTBot (training), OAI-SearchBot (search index), ChatGPT-User (real-time RAG)ChatGPT-User fetches pages in real-time when users ask current questions. Respects robots.txt.PerplexityBot (indexing), user-agent varies for RAGGooglebot (standard), Google-Extended (AI training)ClaudeBot (training)GEO is still a young discipline, and there are pitfalls that can waste your optimization effort or backfire entirely.
LLMs have training data cutoffs. Content you optimize today may not appear in a model’s training data until its next update — which could be weeks or months away. RAG-based systems (ChatGPT with browsing, Perplexity) pick up changes faster because they fetch pages in real-time, but models that rely on pre-trained knowledge won’t reflect your updates until the next training cycle. Plan accordingly: GEO is a long game, not an overnight fix.
There’s a real risk of optimizing your content so well for AI extraction that users get the complete answer without ever visiting your site. If your TL;DR and FAQ sections fully satisfy the user’s query, the LLM cites you — but nobody clicks through. This is especially common with factual, definitional content. The mitigation: structure content so AI answers include your key claims but leave depth, context, and actionable detail on your site. Give enough to earn the citation, not so much that the click becomes unnecessary.
Content that reads like it was written for machines, not humans, backfires in two ways. First, human visitors bounce because the writing feels robotic. Second, AI systems are increasingly trained to detect and deprioritize content that appears optimized for AI extraction rather than genuine expertise. The best GEO content is well-structured technical writing that happens to be easy for LLMs to cite — not content stuffed with definition patterns and schema markup at the expense of readability.
If you’re starting GEO from scratch, focus in this order:
robots.txt — This is binary. If they can’t crawl you, nothing else matters.GEO is not a replacement for SEO. It’s an additional surface. Your pages still need to rank on Google, still need solid technical fundamentals, still need backlinks and authority.
What GEO adds is a systematic approach to a new traffic channel. AI search traffic is growing. The sites that show up in AI-generated answers — with citations, with click-through links — capture that traffic. The sites that don’t show up lose it to competitors who do.
WebDecoy gives you the measurement layer: AI Referral Tracking tells you which platforms send you traffic, Citation Monitoring tells you which pages AI crawlers are indexing, and the API lets you feed both data streams into your LLM automation pipeline to optimize at scale.
Start with measurement. Then optimize what the data tells you matters.
A hands-on Next.js guide to blocking AI crawlers: edge middleware to gate requests, honeypot routes to trap bots, and origin fingerprinting.
tutorialBuild a proof-of-work challenge system using Hashcash to stop bots without CAPTCHAs. Full tutorial with Node.js and browser code.
tutorialStop 98% of bots without CAPTCHAs. Complete guide to invisible honeypot forms, buttons, and endpoints with React, Vue, and vanilla JS examples.
tutorialLike this post? Share it with your friends!
Get a personalized demo from our team.