Skip to content
Task

Add Tasks

Create new tasks within a specified squid

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.
POSThttps://api.lobstr.io/v1/tasks

This endpoint creates new tasks within a specified squid. Tasks define the specific scraping jobs to be executed, such as URLs to scrape or search parameters.

Note: The accepted task input keys (e.g., url, city, activity) depend on the crawler used in your squid. Use the Get Crawler Parameters endpoint to discover valid keys for your crawler.

Headers

Authorizationstringheaderrequired

Your API authentication token. Value: Token YOUR_API_KEY

Content-Typestringheaderrequired

Must be application/json. Value: application/json

Request Body

squidstringrequired

The hash ID of the squid to add tasks to. Example: "a1b2c3d4e5f6g7h8i9j0"

tasksarrayrequired

Array of task objects. Each task contains parameters specific to the crawler. Example: [{"url": "https://example.com"}]

Response Field Explanations

duplicated_countinteger

Number of tasks that were duplicates and not added. Example: 0

tasksarray

Array of created task objects.

tasks[].idstring

Unique task identifier (hash). Example: "c5e29d2aba8b77cdc56391e7405302de"

tasks[].objectstring

Always "task".

tasks[].created_atstring

Task creation timestamp (ISO 8601). Example: "2025-02-10T14:19:32.916901"

tasks[].is_activeboolean

Whether the task is active and ready to run. Example: true

tasks[].paramsobject

Task parameters specific to the crawler (URL, search terms, etc.). Example: {"url": "https://..."}

tasks[].modulestring

Identifier of the crawler module this task belongs to.

Use the Get Crawler Parameters endpoint to discover which task keys are valid for your specific crawler.

Duplicate tasks (same parameters) are automatically detected and not added. Check duplicated_count in the response.

Tasks you add while a run is in progress join that run: it checks the task list again each time a scraper finishes a task. If the run has already finished, start a new one with Start Run.

A run processes every task of the squid, including the ones an earlier run already did, and their results are charged again. To run only new inputs, put them in a new squid, or first remove the done tasks with Delete Task (one task) or Empty Squid (every task): the results of earlier runs stay downloadable.

Task parameters vary by crawler. Using invalid keys will result in task creation failure.

For bulk task creation, consider using the Upload Tasks endpoint with a CSV file instead.

Code Examples

curl -X POST "https://api.lobstr.io/v1/tasks" \
  -H "Authorization: Token YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "squid": "a1b2c3d4e5f6g7h8i9j0",
    "tasks": [
      {"url": "https://www.linkedin.com/in/johndoe"},
      {"url": "https://www.linkedin.com/in/janedoe"}
    ]
  }'

Response

200
{
  "duplicated_count": 0,
  "tasks": [
    {
      "id": "6b9d3afdf4e76df374915a50d7a495c4",
      "object": "task",
      "created_at": "2025-02-10T14:19:32.916901",
      "is_active": true,
      "params": {"url": "https://www.linkedin.com/in/johndoe"},
      "module": "4734d096159ef05210e0e1677e8be823"
    },
    {
      "id": "c5e29d2aba8b77cdc56391e7405302de",
      "object": "task",
      "created_at": "2025-02-10T14:19:32.916901",
      "is_active": true,
      "params": {"url": "https://www.linkedin.com/in/janedoe"},
      "module": "4734d096159ef05210e0e1677e8be823"
    }
  ]
}
400
{
  "error": {
    "message": "Invalid task parameters. Each task must contain required fields for the crawler.",
    "type": "validation_error",
    "param": "tasks"
  }
}