Next.js Bot Detection: Block AI Crawlers at the Edge
A hands-on Next.js guide to blocking AI crawlers: edge middleware to gate requests, honeypot routes to trap bots, and origin fingerprinting.
tutorialBuild a proof-of-work challenge system using Hashcash to stop bots without CAPTCHAs. Full tutorial with Node.js and browser code.
WebDecoy Team
WebDecoy Security Team
CAPTCHAs are broken. Not conceptually, but practically. Image grids get solved by vision models. Invisible scoring systems leak behavioral data to third parties. And every CAPTCHA you add to a form costs you conversions.
But there’s an older idea that’s worth revisiting: make the client do computational work to prove it’s not spamming you.
This is the core idea behind Hashcash, originally proposed by Adam Back in 1997 to fight email spam. The concept is simple: before accepting a request, require the client to find a partial hash collision. This takes a fraction of a second for a single legitimate request but makes mass automated submissions economically painful.
In this tutorial, we’ll build a complete proof-of-work challenge system you can drop into any web application. No third-party services. No tracking pixels. No “select all the traffic lights.”
Traditional CAPTCHAs test identity: “Are you a human?” Proof-of-work tests cost: “Are you willing to spend resources on this request?”
That distinction matters. A bot operator spinning up 10,000 requests per second doesn’t care about solving one CAPTCHA. But if each request requires 200ms of CPU time, that’s 2,000 CPU-seconds per second. You’d need a serious GPU cluster to maintain that throughput, and the economics stop making sense for most attacks.
Here’s why this approach is compelling:
| Property | Traditional CAPTCHA | Proof-of-Work |
|---|---|---|
| User friction | High (click, wait, retry) | None (runs in background) |
| Privacy | Often sends data to third party | Fully self-contained |
| AI resistance | Declining (vision models solve them) | Irrelevant (it’s pure math) |
| Cost to attacker | Near zero (CAPTCHA farms) | Linear with request volume |
| Accessibility | Often fails for screen readers | Works for everyone |
| Dependencies | External service required | Zero dependencies |
The key insight: a legitimate user submitting one form doesn’t notice 200ms of background computation. A bot trying to submit 10,000 forms absolutely does.
Hashcash asks the client to find a nonce that, when combined with a challenge string and hashed, produces a hash with a certain number of leading zero bits.
Here’s the flow:
1. Client requests a challenge from the server
2. Server generates: { challenge: "abc123", difficulty: 20 }
3. Client tries nonce=0: SHA-256("abc123:0") = "7f3a..." (no leading zeros)
4. Client tries nonce=1: SHA-256("abc123:1") = "c891..." (no leading zeros)
5. ...thousands of attempts...
6. Client finds nonce=48291: SHA-256("abc123:48291") = "00000a3f..." (20 leading zero bits!)
7. Client sends { challenge: "abc123", nonce: 48291 } with the form submission
8. Server verifies: SHA-256("abc123:48291") starts with 20 zero bits? Yes. Accept.The difficulty parameter controls how many leading zero bits are required. Each additional bit doubles the expected work:
| Difficulty (bits) | Average attempts | Time (~laptop) |
|---|---|---|
| 16 | ~65,536 | ~50ms |
| 18 | ~262,144 | ~150ms |
| 20 | ~1,048,576 | ~500ms |
| 22 | ~4,194,304 | ~2s |
| 24 | ~16,777,216 | ~8s |
For most use cases, 18-20 bits hits the sweet spot: imperceptible to humans, brutal at scale for bots.
Let’s build the server side first. We need three things: challenge generation, solution verification, and replay protection.
import crypto from 'node:crypto';
// In-memory store for issued challenges
// In production, use Redis with TTL
const challengeStore = new Map();
function generateChallenge(difficulty = 20) {
const challenge = crypto.randomBytes(16).toString('hex');
const timestamp = Date.now();
challengeStore.set(challenge, {
difficulty,
timestamp,
used: false,
});
// Auto-expire after 5 minutes
setTimeout(() => challengeStore.delete(challenge), 5 * 60 * 1000);
return { challenge, difficulty, timestamp };
}function verifySolution(challenge, nonce) {
const entry = challengeStore.get(challenge);
if (!entry) {
return { valid: false, reason: 'unknown_challenge' };
}
if (entry.used) {
return { valid: false, reason: 'already_used' };
}
// Check if challenge has expired (5 min window)
if (Date.now() - entry.timestamp > 5 * 60 * 1000) {
challengeStore.delete(challenge);
return { valid: false, reason: 'expired' };
}
// Verify the hash
const input = `${challenge}:${nonce}`;
const hash = crypto.createHash('sha256').update(input).digest();
if (!hasLeadingZeroBits(hash, entry.difficulty)) {
return { valid: false, reason: 'insufficient_work' };
}
// Mark as used to prevent replay
entry.used = true;
return { valid: true };
}
function hasLeadingZeroBits(hash, requiredBits) {
const fullBytes = Math.floor(requiredBits / 8);
const remainingBits = requiredBits % 8;
// Check full zero bytes
for (let i = 0; i < fullBytes; i++) {
if (hash[i] !== 0) return false;
}
// Check remaining bits in the next byte
if (remainingBits > 0) {
const mask = 0xff << (8 - remainingBits);
if ((hash[fullBytes] & mask) !== 0) return false;
}
return true;
}import express from 'express';
const app = express();
app.use(express.json());
// Issue a challenge
app.post('/api/challenge', (req, res) => {
const challenge = generateChallenge(20);
res.json(challenge);
});
// Submit a form with proof-of-work
app.post('/api/contact', (req, res) => {
const { challenge, nonce, ...formData } = req.body;
const result = verifySolution(challenge, nonce);
if (!result.valid) {
return res.status(429).json({
error: 'Invalid proof of work',
reason: result.reason,
});
}
// Challenge verified. Process the form normally.
handleContactForm(formData);
res.json({ success: true });
});That’s the entire server. No API keys, no external services, no user data leaving your infrastructure.
The client needs to solve the challenge by brute-forcing nonces until it finds one that produces a hash with enough leading zeros.
async function solveChallenge(challenge, difficulty) {
const target = difficulty;
let nonce = 0;
while (true) {
const input = `${challenge}:${nonce}`;
const encoder = new TextEncoder();
const data = encoder.encode(input);
const hashBuffer = await crypto.subtle.digest('SHA-256', data);
const hashArray = new Uint8Array(hashBuffer);
if (hasLeadingZeroBits(hashArray, target)) {
return nonce;
}
nonce++;
}
}
function hasLeadingZeroBits(hash, requiredBits) {
const fullBytes = Math.floor(requiredBits / 8);
const remainingBits = requiredBits % 8;
for (let i = 0; i < fullBytes; i++) {
if (hash[i] !== 0) return false;
}
if (remainingBits > 0) {
const mask = 0xff << (8 - remainingBits);
if ((hash[fullBytes] & mask) !== 0) return false;
}
return true;
}The Web Crypto API is async, which means each hash involves a microtask. For tight loops, a synchronous approach using a bundled SHA-256 is significantly faster. Here’s a version using a minimal SHA-256 implementation:
// If you're already bundling, use a library like 'js-sha256'
// npm install js-sha256
import { sha256 } from 'js-sha256';
function solveChallengeSync(challenge, difficulty) {
let nonce = 0;
while (true) {
const input = `${challenge}:${nonce}`;
const hash = sha256.arrayBuffer(input);
const view = new Uint8Array(hash);
if (hasLeadingZeroBits(view, difficulty)) {
return nonce;
}
nonce++;
}
}This version is typically 3-5x faster than the async Web Crypto version because it avoids the overhead of promise scheduling on every iteration.
Here’s how to wire proof-of-work into a standard contact form. The challenge gets fetched and solved in the background while the user types.
<form id="contact-form">
<input type="text" name="name" placeholder="Name" required />
<input type="email" name="email" placeholder="Email" required />
<textarea name="message" placeholder="Message" required></textarea>
<button type="submit">Send</button>
</form>
<script type="module">
const form = document.getElementById('contact-form');
let powResult = null;
// Start solving as soon as the page loads
async function prepareProofOfWork() {
const res = await fetch('/api/challenge', { method: 'POST' });
const { challenge, difficulty } = await res.json();
const nonce = await solveChallenge(challenge, difficulty);
powResult = { challenge, nonce };
}
prepareProofOfWork();
form.addEventListener('submit', async (e) => {
e.preventDefault();
// If the challenge isn't solved yet, wait for it
if (!powResult) {
const btn = form.querySelector('button');
btn.textContent = 'Verifying...';
btn.disabled = true;
// Poll until ready
while (!powResult) {
await new Promise((r) => setTimeout(r, 100));
}
}
const formData = new FormData(form);
const body = {
...Object.fromEntries(formData),
...powResult,
};
const res = await fetch('/api/contact', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
if (res.ok) {
form.reset();
powResult = null;
prepareProofOfWork(); // Get ready for next submission
}
});
</script>import { useState, useEffect, useRef } from 'react';
function ProtectedForm() {
const [status, setStatus] = useState('idle');
const powRef = useRef(null);
useEffect(() => {
async function prepare() {
const res = await fetch('/api/challenge', { method: 'POST' });
const { challenge, difficulty } = await res.json();
const nonce = await solveChallenge(challenge, difficulty);
powRef.current = { challenge, nonce };
}
prepare();
}, []);
async function handleSubmit(e) {
e.preventDefault();
setStatus('submitting');
// Wait for proof-of-work if still computing
while (!powRef.current) {
await new Promise((r) => setTimeout(r, 100));
}
const formData = new FormData(e.target);
const body = {
...Object.fromEntries(formData),
...powRef.current,
};
const res = await fetch('/api/contact', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
setStatus(res.ok ? 'sent' : 'error');
powRef.current = null;
}
return (
<form onSubmit={handleSubmit}>
<input name="name" placeholder="Name" required />
<input name="email" type="email" placeholder="Email" required />
<textarea name="message" placeholder="Message" required />
<button type="submit" disabled={status === 'submitting'}>
{status === 'submitting' ? 'Verifying...' : 'Send'}
</button>
</form>
);
}In both cases, the user never sees the proof-of-work happening. The challenge gets solved while they’re filling out the form.
Static difficulty works, but you can do better. Adjusting difficulty based on context makes the system smarter.
function getDifficulty(req) {
let difficulty = 18; // baseline
// Rate limiting: increase difficulty for repeat visitors
const recentRequests = getRecentRequestCount(req.ip, '5m');
if (recentRequests > 5) difficulty += 2;
if (recentRequests > 20) difficulty += 4;
// High-value endpoints get higher difficulty
if (req.path === '/api/signup') difficulty += 2;
// Known good sessions get lower difficulty
if (req.session?.verified) difficulty -= 2;
// Time of day: increase during known attack windows
const hour = new Date().getUTCHours();
if (hour >= 2 && hour <= 6) difficulty += 1;
return Math.max(16, Math.min(difficulty, 26));
}Let the client report how long the solve took. This gives you telemetry to tune difficulty without guessing.
async function solveAndReport(challenge, difficulty) {
const start = performance.now();
const nonce = await solveChallenge(challenge, difficulty);
const elapsed = performance.now() - start;
return { challenge, nonce, solveTimeMs: Math.round(elapsed) };
}On the server, track these timings. If average solve times drift above 2 seconds, your difficulty might be too high. If they’re consistently under 50ms, you might want to bump it up.
The synchronous solver blocks the main thread. For a better experience, offload the work to a Web Worker.
// worker-pow.js
import { sha256 } from 'js-sha256';
self.onmessage = function (e) {
const { challenge, difficulty } = e.data;
let nonce = 0;
while (true) {
const input = `${challenge}:${nonce}`;
const hash = sha256.arrayBuffer(input);
const view = new Uint8Array(hash);
if (hasLeadingZeroBits(view, difficulty)) {
self.postMessage({ nonce, attempts: nonce + 1 });
return;
}
// Report progress every 100k attempts
if (nonce % 100000 === 0) {
self.postMessage({ progress: nonce });
}
nonce++;
}
};
function hasLeadingZeroBits(hash, requiredBits) {
const fullBytes = Math.floor(requiredBits / 8);
const remainingBits = requiredBits % 8;
for (let i = 0; i < fullBytes; i++) {
if (hash[i] !== 0) return false;
}
if (remainingBits > 0) {
const mask = 0xff << (8 - remainingBits);
if ((hash[fullBytes] & mask) !== 0) return false;
}
return true;
}function solveInWorker(challenge, difficulty) {
return new Promise((resolve) => {
const worker = new Worker('/worker-pow.js');
worker.onmessage = (e) => {
if (e.data.nonce !== undefined) {
resolve(e.data.nonce);
worker.terminate();
}
};
worker.postMessage({ challenge, difficulty });
});
}This keeps the UI completely responsive. The user can type, scroll, and interact normally while the worker crunches hashes in a background thread.
If you want to get aggressive, split the nonce space across multiple workers:
function solveParallel(challenge, difficulty, workerCount = 4) {
return new Promise((resolve) => {
const workers = [];
let solved = false;
for (let i = 0; i < workerCount; i++) {
const worker = new Worker('/worker-pow.js');
workers.push(worker);
worker.onmessage = (e) => {
if (e.data.nonce !== undefined && !solved) {
solved = true;
resolve(e.data.nonce);
workers.forEach((w) => w.terminate());
}
};
// Each worker starts at a different offset and steps by workerCount
worker.postMessage({
challenge,
difficulty,
startNonce: i,
step: workerCount,
});
}
});
}On a 4-core machine, this cuts solve time by roughly 4x.
Proof-of-work isn’t just for forms. Here are other places where it makes sense.
Instead of hard rate limits that block legitimate power users, require proof-of-work that scales with request volume:
app.use('/api', async (req, res, next) => {
const pow = req.headers['x-pow-challenge'];
const nonce = req.headers['x-pow-nonce'];
if (!pow || !nonce) {
// First request: no PoW needed, but issue a challenge
const challenge = generateChallenge(16);
res.set('X-PoW-Challenge', challenge.challenge);
res.set('X-PoW-Difficulty', String(challenge.difficulty));
return next();
}
const result = verifySolution(pow, parseInt(nonce));
if (!result.valid) {
return res.status(429).json({ error: 'Invalid proof of work' });
}
next();
});Make each login attempt cost CPU time. Legitimate users logging in once won’t notice. An attacker trying 10,000 passwords will:
app.post('/api/login', (req, res) => {
const { username, password, challenge, nonce } = req.body;
// Verify proof of work first
const pow = verifySolution(challenge, nonce);
if (!pow.valid) {
return res.status(429).json({ error: 'Complete the challenge first' });
}
// Increase difficulty after failed attempts
const failures = getFailedLoginCount(username, '15m');
const nextDifficulty = Math.min(16 + failures * 2, 28);
// Normal login logic here...
const user = authenticate(username, password);
if (!user) {
recordFailedLogin(username);
const newChallenge = generateChallenge(nextDifficulty);
return res.status(401).json({
error: 'Invalid credentials',
challenge: newChallenge,
});
}
res.json({ token: generateToken(user) });
});After 5 failed attempts, the attacker needs to solve a difficulty-26 challenge per attempt. That’s roughly 30 seconds of CPU time per guess. Brute forcing becomes completely impractical.
Require proof-of-work before establishing WebSocket connections to prevent resource exhaustion:
import { WebSocketServer } from 'ws';
const wss = new WebSocketServer({ noServer: true });
server.on('upgrade', (req, socket, head) => {
const url = new URL(req.url, `http://${req.headers.host}`);
const challenge = url.searchParams.get('challenge');
const nonce = url.searchParams.get('nonce');
const result = verifySolution(challenge, parseInt(nonce));
if (!result.valid) {
socket.write('HTTP/1.1 429 Too Many Requests\r\n\r\n');
socket.destroy();
return;
}
wss.handleUpgrade(req, socket, head, (ws) => {
wss.emit('connection', ws, req);
});
});Blog comments, forum posts, review submissions. Any user-generated content endpoint benefits from proof-of-work:
app.post('/api/comments', (req, res) => {
const { challenge, nonce, ...comment } = req.body;
const result = verifySolution(challenge, nonce);
if (!result.valid) {
return res.status(429).json({ error: 'Proof of work required' });
}
// A single comment is fine. Posting 500 comments costs
// 500 * ~500ms = 250 seconds of CPU time.
saveComment(comment);
res.json({ success: true });
});Proof-of-work is not a silver bullet. Be aware of these tradeoffs before adopting it.
It penalizes slow devices. A difficulty-20 challenge that takes 200ms on a modern laptop might take 2 seconds on a budget phone. If your audience skews toward lower-end devices, keep difficulty low or use adaptive difficulty based on the User-Agent.
It doesn’t prove humanity. Proof-of-work proves that someone spent CPU time. A well-resourced attacker with GPU clusters can still solve challenges faster than your users. It raises the cost, but doesn’t eliminate the possibility. For high-stakes scenarios (account creation, payments), combine it with other signals.
It uses client energy. Each challenge burns a small amount of battery and CPU. For a single form submission, this is negligible. But if your page loads trigger proof-of-work challenges on every navigation, mobile users will notice the battery drain.
GPU solvers exist. SHA-256 is exactly the kind of work that GPUs excel at. An attacker with a modern GPU can solve challenges orders of magnitude faster than a browser. For most use cases this doesn’t matter since the economics still don’t favor mass abuse, but keep it in mind for threat modeling.
Good fit:
Not ideal on its own:
Proof-of-work works best as one layer in a defense stack:
Layer 1: Proof-of-Work → raises cost of mass requests
Layer 2: Honeypot fields → catches unsophisticated bots
Layer 3: Behavioral analysis → detects automation patterns
Layer 4: Rate limiting → hard cap on request volumeEach layer catches a different class of bot. Together, they make automated abuse extremely difficult without adding any user-facing friction.
Hashcash was invented almost 30 years ago to fight email spam. The email world moved on to other solutions, but the core idea of making requests computationally expensive is more relevant than ever for web applications.
The implementation is dead simple: a few hundred lines of code, zero external dependencies, zero privacy concerns, and zero user friction. Your users never see a challenge. Bots feel every single one.
The full code from this tutorial is straightforward to adapt. Start with difficulty 18-20 on your most abused endpoints and adjust from there based on solve-time telemetry.
If you want a production-ready proof-of-work system that handles adaptive difficulty, worker management, and integration with broader bot detection, check out FCaptcha and the WebDecoy platform.
Related Reading:
A hands-on Next.js guide to blocking AI crawlers: edge middleware to gate requests, honeypot routes to trap bots, and origin fingerprinting.
tutorialHow to optimize for AI search engines, track citations and referrals, and automate GEO workflows with your own LLMs.
tutorialStop 98% of bots without CAPTCHAs. Complete guide to invisible honeypot forms, buttons, and endpoints with React, Vue, and vanilla JS examples.
tutorialLike this post? Share it with your friends!
Get a personalized demo from our team.