Fix crawl errors and access blocks
HTTP errors, robots.txt, IP allowlisting, user agents and export failures.
On this page
Identify what failed
A URL returning 404 does not necessarily mean the crawl failed: finding it can be the purpose of the audit. Distinguish an engine failure, an individual page error and a missing Sheets or Slides document. Check the message, collected URL count and results already available.
Keep the crawl ID and full message, including any code in brackets. The codes below describe Cuik’s crawl pipeline.
The site refuses or limits requests
| Message or code | Meaning | Action |
|---|---|---|
site_blocked, HTTP 403, CAPTCHA |
The site or firewall rejected the request. | Check firewall logs. Allow the crawl source on your site or try the residential proxy in network settings. |
HTTP 429, rate_limit_persistent |
The site is rate-limiting requests. Refusals are recorded in partial results; rate_limit_persistent refers to older stopped crawls. |
Lower the speed, wait for the limit to clear and test a small scope. Avoid repeated restarts. |
robots_blocked |
A robots.txt rule blocks this URL for the selected user agent. | Check the matching group and path. Correct the rule on your site if unintended. |
robots_unavailable |
The engine could not read robots.txt successfully. | Check the file’s HTTP response, redirects and firewall rules. |
site_http_error |
The starting URL returned an HTTP error. | For 401 check access; for 404/410 check the URL; for 5xx investigate the site’s server. |
A residential proxy can help with datacenter IP filtering. It does not guarantee access, resolve every protection or replace the site owner’s authorisation. Verify the option for each launch: it is not retained like an ordinary preference.
Which IP should you allow?
For a server crawl over IPv4 without a residential proxy, the outbound address verified on 4 October 2026 is 46.4.101.91. Allow it only for the site and requests needed by your audit. Ask your hosting or WAF administrator to inspect blocked requests around the test time. You do not need to disable protections globally.
This address does not apply to the desktop app, IPv6 connections or residential proxy: their network exit differs. Select IPv4 under IP connection for a reproducible diagnosis. For another path, confirm the address observed in your site’s logs or ask support with the crawl ID. Do not substitute the DNS address of cuik.io or api.cuik.io: a web service’s address is not necessarily the source of outbound requests.
Infrastructure can change. If an older rule stops working, verify the observed source address before adding permissions.
User agents, robots.txt and JavaScript
A user agent is the identity declared in a request. Cuikbot, Googlebot mobile, Chrome Desktop and custom values let you compare different behaviours. Selecting Googlebot does not make Cuik a Google crawler: a firewall may verify the IP too. Allowing a “Googlebot” or “Cuikbot” string alone is not authentication.
Respect robots.txt controls exploration rules. Respect Crawl-delay independently applies the declared delay. Check both before assuming a malfunction. If an authorised diagnosis requires ignoring a rule, use a limited scope and restore normal settings afterwards.
Crawl JavaScript files checks resources; it does not render the page. If content is injected by JavaScript, enable JavaScript rendering and compare a small set of URLs. Chrome Desktop, JavaScript rendering and Chrome network fingerprinting are separate settings.
Connection failures or no usable pages
| Code | What to check |
|---|---|
site_unreachable |
Check the domain, DNS, TLS certificate, redirects and public access. Compare IPv4 if the problem depends on IPv6. Do not disable certificate verification to hide a site error. |
timeout |
Reduce scope and speed; check whether the site is slow or processing is waiting for capacity. A timeout does not prove the site is offline. |
no_usable_html |
Check the starting URL, HTTP responses, scope, robots.txt and protections: no usable internal HTML page was collected. |
database_invalid |
Crawl data is missing or unusable. Send the crawl ID to support; changing the user agent cannot repair this database. |
Partial results may also reflect URL, depth or variant limits, a narrow scope or links missing from the explored HTML. Sitemap discovery does not automatically enqueue every sitemap URL. See advanced configuration.
The crawl finished but a document is missing
google_api or report_backend_unavailable may concern document preparation even when collection succeeded. Open results before starting another audit. If documents were not requested, use Generate documents, select available templates and check both Sheet and Slides selections.
API or MCP calls do not necessarily request documents. Address template or Google access errors at the document stage instead of repeating exploration. For external_api, identify the enrichment involved: a PageSpeed, Ads or DataForSEO failure has a different cause from your site blocking the crawler.
When to contact support
crawler_unavailable and renderer_unavailable indicate an unavailable engine or browser; interrupted means processing was interrupted. stopped_by_user means a stop was requested. unknown requires the full message to understand the failure.
In Help and support, include the crawl ID, URL, approximate time, engine, error message and network/rendering settings. Add a screenshot or the HTTP response observed on the site. Do not share API keys, passwords or Google tokens. A retryable error does not guarantee that immediately starting again will succeed or leave your allowance unchanged.
For rate reductions, the 1-, 5- and 10-minute pauses and partial results, see HTTP 429: slow or interrupted crawls.