Skip to content
Squid

Create Squid

Create a new squid container for a specific crawler to organize and run scraping tasks

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.
POSThttps://api.lobstr.io/v1/squids

This endpoint creates a new squid for a specified crawler by providing the crawler's hash ID. A squid is a container that groups together related tasks and configurations for a specific scraping operation.

What is a Squid?

A squid acts as a project workspace for your scraping tasks:

  • Groups tasks: All URLs or items you want to scrape with the same crawler
  • Stores configuration: Crawler parameters, concurrency settings, delivery options
  • Manages runs: Tracks execution history and results
  • Enables scheduling: Set up automated recurring scrapes

Headers

Authorizationstringheaderrequired

Your API authentication token. Value: Token YOUR_API_KEY

Content-Typestringheaderrequired

Must be application/json. Value: application/json

Request Body

crawlerstringrequired

The unique ID (hash) of the crawler to use for this squid. Example: "4734d096159ef05210e0e1677e8be823"

namestring

Custom name for the squid. If not provided, an auto-generated name will be used. Example: "My Google Maps Scraper"

paramsobject

The crawler's squid-level settings, set at creation. Unset ones keep the crawler's defaults. Example: {"received_only": true, "fetch_since": "7d"}

accountsarray

Hashes of the connected accounts the squid uses, for crawlers that need one. Example: ["0ecdae5e8cdd4a2c53c034a3047ea9b1"]

Any other field of Update Squid (concurrency, cron_expression, timezone...) can be sent too. They are checked like an update: if one is refused, the creation fails with that error and no squid is created.

Response Field Explanations

idstring

Unique squid identifier. Example: "c106a44a98044ef18acc59986ae10967"

namestring

Squid name (auto-generated as "Crawler Name (N)" if not provided). Example: "Google Maps (1)"

crawlerstring

ID of the associated crawler. Example: "4734d096159ef05210e0e1677e8be823"

is_activeboolean

Whether the squid is active and can run. Example: true

concurrencyinteger

Number of concurrent tasks (default: 1). Example: 1

paramsobject

Squid-level parameters with default values from crawler. Example: {}

scheduleobject | null

Cron schedule configuration (null if not scheduled). Example: null

to_completeboolean

Whether to stop after all tasks complete (false = run continuously). Example: false

Send the squid's settings and accounts with the creation, or change them later with the Update Squid endpoint.

The squid automatically inherits default parameters from the crawler. Check the response 'params' field to see what defaults were applied.

Custom names help organize multiple squids using the same crawler. Without a custom name, squids are numbered sequentially.

Make sure to get the crawler ID from the List Crawlers endpoint first. Using an invalid crawler ID will result in an error.

Code Examples

curl -X POST "https://api.lobstr.io/v1/squids" \
  -H "Authorization: Token YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "crawler": "4734d096159ef05210e0e1677e8be823",
    "name": "My Google Maps Scraper"
  }'

Response

201
{
  "id": "e86b29c032024b66aff529e1d43c2bd7",
  "account": [],
  "concurrency": 1,
  "crawler": "4734d096159ef05210e0e1677e8be823",
  "created_at": "2025-02-03T14:24:23Z",
  "is_active": true,
  "name": "My Google Maps Scraper",
  "params": {
    "max_results": 200,
    "ratings": "Any rating",
    "country": "United States",
    "language": "English (United States)"
  },
  "schedule": null,
  "to_complete": false
}