HTTP 429: slow or interrupted crawls

Why a crawl slows down

HTTP 429 — Too Many Requests means the site, its host or an intermediary is limiting requests. The limit may depend on an IP address, session, path or time window, so a low request rate does not guarantee immediate access. A Retry-After header may specify how long to wait. Official HTTP definition.

Cuik spaces out requests. Progress can remain unchanged during a deliberate wait without the engine having crashed. Avoid running several crawls against the same site at once: they may contribute to the same limit.

What Cuik does automatically

The request rate decreases progressively and does not automatically increase again. For each origin of the audited site:

The engine progressively halves the rate without skipping these thresholds. A manually selected slower rate is kept and faster thresholds are skipped. Resuming preserves the reduced rate and pauses already used.

Retry-After is independent: seconds or an HTTP date, capped at 60 seconds. An explicit pause and a simultaneous Retry-After use the later deadline; they are not added together. The wait applies to subsequent requests to that origin.

429 and 5xx responses are recorded without automatically retrying the same URL. Network failures without an HTTP response retain a bounded retry budget. External checks do not trigger the audited site's long pauses. A server wait exceeding their 30-second budget may result in a deferred external check, without inventing an HTTP status.

What if refusals continue?

After these pauses, Cuik continues through the queue at 0.5 requests/s, or the slower rate selected manually. Persistent 429 responses no longer trigger a global stop, extra probe or automatic proxy switch. Retry-After still applies, capped at one minute per response.

The final partial analysis warning counts 429 and server errors. Their URLs remain available through response-code filters. Their content and links found only on those pages could not be analysed. Finishing the queue does not guarantee full site coverage. Crawl duration and volume limits, and manual stops, still apply.

Understanding the displayed rate

The measured rate is observed throughput. The network ceiling is a limit, not a guaranteed constant speed. Response times, retries and pauses can reduce actual throughput to zero.

At 0.1 URL/s, the ceiling is roughly one request every ten seconds. Redirects and retries can use requests without producing as many newly analysed pages. This number alone cannot guarantee a completion time. A paused crawl is not expected to progress.

What to do if 429 responses continue

1. Wait for the requested cooldown. Avoid repeated resumes and concurrent crawls of the same site.
2. Review site and firewall logs for the crawl's time window: the triggered rule, source IP and limits per path or time period. Request a targeted allowance for the audit rather than disabling protection globally. See crawl troubleshooting for the network source to check.
3. Resume at a lower rate over a small scope once the cause has been addressed; verify responses before expanding.
4. Consider the residential proxy if available, when site logs indicate datacenter IP filtering. It does not guarantee access. Select it explicitly when launching: the engine does not switch proxies automatically.
5. Contact support with the crawl ID, domain, times and settings so the team can inspect the journal and assess resuming the existing results.

Do not confuse 429 with robots.txt

“Blocked by robots.txt” means Cuik did not request the page because a crawl rule prohibited it. There is no HTTP response code for that page: an empty cell or technical code 0 is neither a 429 nor a network failure. A lower rate or a different proxy does not remove that rule.

A 429, by contrast, is a response to a request that was sent. Distinguish these cases before choosing a remedy.