Schedule crawls with the API
A scheduled workflow, status checks and safeguards against duplicate audits.
On this page
Choose a useful frequency
An automation can check a site after deployment or prepare a periodic audit. Start slowly, for example with a weekly audit. Every launch uses the account’s audit allowance, including launches from your own script or an MCP assistant. A scheduled workflow still needs suitable scope and available capacity.
This guide describes a workflow in your automation tool or script, not a built-in Cuik scheduler. First complete a manual run using the crawl API guide.
1. Prepare the launch request
Store the key in your tool’s secret manager. Never expose it in a public web page or Git repository. See API authentication for the header format.
| Workflow setting | Value |
|---|---|
| Method | POST |
| Address | https://api.cuik.io/v1/crawl |
| Header | Authorization: <your Cuik key> |
| Content type | application/json |
| Automatic POST retry | Disabled |
Start with a body limited to 100 URLs and no documents:
{
"url": "https://example.com/",
"engine": "legacy",
"max_urls": 100,
"urls_per_second": 2,
"js": false,
"sheet": false,
"slide": false
}Replace the domain. legacy selects Classic; cuikspider requires beta access on the environment used. Keep configurations consistent between scheduled runs to compare a similar scope.
2. Build the workflow
| Step | Action | Persist |
|---|---|---|
| Trigger | Weekly schedule or deployment event. | Site, timestamp and your workflow execution ID. |
| Guard | Check no previous execution is active or has an uncertain outcome. | A persistent workflow log. |
| Launch | Send one POST request. | The response’s id, immediately. |
| Wait | Wait at least 10 seconds. | The same id. |
| Monitor | Call GET https://api.cuik.io/v1/crawl/{id} with the same header. |
The crawl object and latest status. |
| Decide | Wait again, finish or alert according to status. | Available result link and error message. |
Persistence matters: if your tool restarts, it must resume monitoring the same crawl rather than create another audit. Set a total waiting limit, such as one hour for a small test. On expiry, alert with the crawl ID instead of sending another POST.
3. Handle states without losing results
Launch returns id directly; monitoring returns a crawl object. Read crawl.status, crawl.crawl_status when present, counters and any error. The two calls do not use the same response shape.
- Queued, running or finalising: wait, then read again.
- Collection succeeded: retrieve the available result; documents may still be processing.
- Error, interruption, stop or pause: retain the crawl and message, then alert the responsible person rather than loop indefinitely.
- Unknown or new status: keep the ID and flag it; do not assume success.
If requesting Sheets/Slides, successful collection does not guarantee both documents are ready. Check available document links separately. You can also generate documents later from the existing audit in the app.
4. Prevent duplicate audits
A POST timeout means your tool did not receive a response. It does not prove Cuik failed to start the audit. After a connection failure, use GET /v1/crawls to find the relevant site and time before another attempt. Paginate when needed; do not blindly select the account’s first result.
Do not assume an arbitrary idempotency header will prevent duplicates: this endpoint does not guarantee that contract. Persist launches, keep their IDs and block automatic retries when the outcome is uncertain.
5. Test before scheduling
Test a small site, an access refusal and an interruption of your workflow after receiving the ID. Verify that it resumes the same crawl, stops on failure and sends an alert containing the correct link or ID.
Keep alerts in one channel. For IP blocks, robots.txt, external quotas or documents, use the troubleshooting guide. To launch an analysis with an assistant, use the MCP connector; a conversation alone is not a recurring scheduler.