Add Tasks
Create new tasks within a specified squid
https://api.lobstr.io/v1/tasksThis endpoint creates new tasks within a specified squid. Tasks define the specific scraping jobs to be executed, such as URLs to scrape or search parameters.
Note: The accepted task input keys (e.g., url, city, activity) depend on the crawler used in your squid. Use the Get Crawler Parameters endpoint to discover valid keys for your crawler.
Headers
AuthorizationstringheaderrequiredYour API authentication token. Value: Token YOUR_API_KEY
Content-TypestringheaderrequiredMust be application/json. Value: application/json
Request Body
squidstringrequiredThe hash ID of the squid to add tasks to. Example: "a1b2c3d4e5f6g7h8i9j0"
tasksarrayrequiredArray of task objects. Each task contains parameters specific to the crawler. Example: [{"url": "https://example.com"}]
Response Field Explanations
duplicated_countintegerNumber of tasks that were duplicates and not added. Example: 0
tasksarrayArray of created task objects.
tasks[].idstringUnique task identifier (hash). Example: "c5e29d2aba8b77cdc56391e7405302de"
tasks[].objectstringAlways "task".
tasks[].created_atstringTask creation timestamp (ISO 8601). Example: "2025-02-10T14:19:32.916901"
tasks[].is_activebooleanWhether the task is active and ready to run. Example: true
tasks[].paramsobjectTask parameters specific to the crawler (URL, search terms, etc.). Example: {"url": "https://..."}
tasks[].modulestringIdentifier of the crawler module this task belongs to.
Use the Get Crawler Parameters endpoint to discover which task keys are valid for your specific crawler.
Duplicate tasks (same parameters) are automatically detected and not added. Check duplicated_count in the response.
Tasks you add while a run is in progress join that run: it checks the task list again each time a scraper finishes a task. If the run has already finished, start a new one with Start Run.
A run processes every task of the squid, including the ones an earlier run already did, and their results are charged again. To run only new inputs, put them in a new squid, or first remove the done tasks with Delete Task (one task) or Empty Squid (every task): the results of earlier runs stay downloadable.
Task parameters vary by crawler. Using invalid keys will result in task creation failure.
For bulk task creation, consider using the Upload Tasks endpoint with a CSV file instead.
Code Examples
curl -X POST "https://api.lobstr.io/v1/tasks" \ -H "Authorization: Token YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "squid": "a1b2c3d4e5f6g7h8i9j0", "tasks": [ {"url": "https://www.linkedin.com/in/johndoe"}, {"url": "https://www.linkedin.com/in/janedoe"} ] }'
Response
{ "duplicated_count": 0, "tasks": [ { "id": "6b9d3afdf4e76df374915a50d7a495c4", "object": "task", "created_at": "2025-02-10T14:19:32.916901", "is_active": true, "params": {"url": "https://www.linkedin.com/in/johndoe"}, "module": "4734d096159ef05210e0e1677e8be823" }, { "id": "c5e29d2aba8b77cdc56391e7405302de", "object": "task", "created_at": "2025-02-10T14:19:32.916901", "is_active": true, "params": {"url": "https://www.linkedin.com/in/janedoe"}, "module": "4734d096159ef05210e0e1677e8be823" } ] }
{ "error": { "message": "Invalid task parameters. Each task must contain required fields for the crawler.", "type": "validation_error", "param": "tasks" } }