Create Squid
Create a new squid container for a specific crawler to organize and run scraping tasks
https://api.lobstr.io/v1/squidsThis endpoint creates a new squid for a specified crawler by providing the crawler's hash ID. A squid is a container that groups together related tasks and configurations for a specific scraping operation.
What is a Squid?
A squid acts as a project workspace for your scraping tasks:
- Groups tasks: All URLs or items you want to scrape with the same crawler
- Stores configuration: Crawler parameters, concurrency settings, delivery options
- Manages runs: Tracks execution history and results
- Enables scheduling: Set up automated recurring scrapes
Headers
AuthorizationstringheaderrequiredYour API authentication token. Value: Token YOUR_API_KEY
Content-TypestringheaderrequiredMust be application/json. Value: application/json
Request Body
crawlerstringrequiredThe unique ID (hash) of the crawler to use for this squid. Example: "4734d096159ef05210e0e1677e8be823"
namestringCustom name for the squid. If not provided, an auto-generated name will be used. Example: "My Google Maps Scraper"
paramsobjectThe crawler's squid-level settings, set at creation. Unset ones keep the crawler's defaults. Example: {"received_only": true, "fetch_since": "7d"}
accountsarrayHashes of the connected accounts the squid uses, for crawlers that need one. Example: ["0ecdae5e8cdd4a2c53c034a3047ea9b1"]
Any other field of Update Squid (concurrency, cron_expression, timezone...) can be sent too. They are checked like an update: if one is refused, the creation fails with that error and no squid is created.
Response Field Explanations
idstringUnique squid identifier. Example: "c106a44a98044ef18acc59986ae10967"
namestringSquid name (auto-generated as "Crawler Name (N)" if not provided). Example: "Google Maps (1)"
crawlerstringID of the associated crawler. Example: "4734d096159ef05210e0e1677e8be823"
is_activebooleanWhether the squid is active and can run. Example: true
concurrencyintegerNumber of concurrent tasks (default: 1). Example: 1
paramsobjectSquid-level parameters with default values from crawler. Example: {}
scheduleobject | nullCron schedule configuration (null if not scheduled). Example: null
to_completebooleanWhether to stop after all tasks complete (false = run continuously). Example: false
Send the squid's settings and accounts with the creation, or change them later with the Update Squid endpoint.
The squid automatically inherits default parameters from the crawler. Check the response 'params' field to see what defaults were applied.
Custom names help organize multiple squids using the same crawler. Without a custom name, squids are numbered sequentially.
Make sure to get the crawler ID from the List Crawlers endpoint first. Using an invalid crawler ID will result in an error.
Code Examples
curl -X POST "https://api.lobstr.io/v1/squids" \ -H "Authorization: Token YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "crawler": "4734d096159ef05210e0e1677e8be823", "name": "My Google Maps Scraper" }'
Response
{ "id": "e86b29c032024b66aff529e1d43c2bd7", "account": [], "concurrency": 1, "crawler": "4734d096159ef05210e0e1677e8be823", "created_at": "2025-02-03T14:24:23Z", "is_active": true, "name": "My Google Maps Scraper", "params": { "max_results": 200, "ratings": "Any rating", "country": "United States", "language": "English (United States)" }, "schedule": null, "to_complete": false }