Skip to content
Crawler

Get Crawler Parameters

Retrieve the list of configurable input parameters for a specific crawler

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.
GEThttps://api.lobstr.io/v1/crawlers/{crawler_hash}/params

This endpoint returns all configurable input parameters for a specific crawler. The response is organized into two main sections: task and squid.

Response Structure

The response contains:

  • task: Object containing task-level parameters (parameter name as key)
  • squid: Object containing squid-level parameters and functions
  • squid.functions: Optional add-on features with extra credit costs

Task-level Parameter Fields

task.{parameter_name}.typestring

Data type (string, int, boolean, etc.). Example: "string"

task.{parameter_name}.groupstring

Logical grouping (e.g., "url", "location"). Example: "url"

task.{parameter_name}.regexstring

Validation pattern for string inputs. Example: "^https?://.*"

task.{parameter_name}.requiredboolean

Whether the parameter is mandatory. Example: true

task.{parameter_name}.defaultstring | number | boolean | null

Default value if not provided. Example: null

task.{parameter_name}.attributestring

Internal attribute name. Example: "url"

task.{parameter_name}.is_paramsboolean

Whether it's used for URL parameter generation. Example: true

Squid-level Parameter Fields

squid.{parameter_name}.typestring

Data type of the parameter. Example: "string"

squid.{parameter_name}.allowedarray

Array of valid values (for enum types). Example: ["en", "es", "fr"]

squid.{parameter_name}.defaultstring | number | boolean | null

Default value if not provided. Example: "en"

squid.{parameter_name}.requiredboolean

Whether the parameter is mandatory. Example: false

squid.{parameter_name}.attributestring

Internal attribute name. Example: "language"

Functions (Optional Add-ons) Fields

squid.functions.{function_name}.sortinteger

Display order of the function. Example: 1

squid.functions.{function_name}.defaultboolean

Whether the function is enabled by default. Example: false

squid.functions.{function_name}.credits_per_functionobject

Extra credits per result when this function is on, keyed by plan type like credits_per_row: current is the rate on today's plans, legacy the rate kept by grandfathered subscriptions (Get User Profile → plan[].is_legacy). Example: {"legacy": 80, "current": 4}

squid.functions.{function_name}.skip_squidboolean

If true, a result this squid already ran the function on is not processed again, and not charged again, on later runs. Example: true

Headers

Authorizationstringheaderrequired

Your API authentication token. Value: Token YOUR_API_KEY

Content-TypestringheaderDefault application/json

Content type of the request. Value: application/json

Path Parameters

crawler_hashstringpathrequired

The unique identifier (hash/id) of the crawler. Example: 4734d096159ef05210e0e1677e8be823

The response structure uses parameter names as object keys. Access task parameters via params.task.url, squid parameters via params.squid.language, etc.

Functions under squid.functions are optional add-ons. Each function adds its credits_per_function cost (the current or legacy rate) to every result it runs on.

Pay attention to 'allowed' arrays in squid parameters - these only accept specific values. Sending invalid values will cause validation errors.

Use 'regex' patterns in task parameters to validate input before sending. This helps catch errors early and reduces failed tasks.

Parameters with 'group' field are logically related. For location-based crawlers, you'll typically need to provide all parameters in the 'location' group.

Code Examples

curl -X GET "https://api.lobstr.io/v1/crawlers/4734d096159ef05210e0e1677e8be823/params" \
  -H "Authorization: Token YOUR_API_KEY"

Response

200
{
  "task": {
    "url": {
      "type": "string",
      "group": "url",
      "regex": "http(.*)google(.*)/maps/(search|place)(.*)",
      "default": null,
      "required": true,
      "attribute": "url",
      "is_params": false
    },
    "city": {
      "type": "string",
      "group": "location",
      "default": null,
      "required": true,
      "attribute": "city",
      "is_params": true
    },
    "region": {
      "type": "string",
      "group": "location",
      "default": null,
      "required": false,
      "attribute": "region",
      "is_params": true
    },
    "country": {
      "type": "string",
      "group": "location",
      "default": null,
      "required": true,
      "attribute": "country",
      "is_params": true
    },
    "category": {
      "type": "string",
      "group": "location",
      "default": null,
      "required": true,
      "attribute": "category",
      "is_params": true
    },
    "district": {
      "type": "string",
      "group": "location",
      "default": null,
      "required": false,
      "attribute": "district",
      "is_params": true
    }
  },
  "squid": {
    "country": {
      "type": "string",
      "allowed": [
        "Afghanistan",
        "Albania",
        "Algeria",
        "United States",
        "United Kingdom",
        "France",
        "Germany",
        "Japan",
        "Brazil"
      ],
      "default": "United States",
      "required": false,
      "attribute": "country",
      "is_params": false
    },
    "ratings": {
      "type": "string",
      "allowed": [
        "Any rating",
        "2.0+",
        "2.5+",
        "3.0+",
        "3.5+",
        "4.0+",
        "4.5+"
      ],
      "default": "Any rating",
      "required": false,
      "attribute": "ratings",
      "is_params": false
    },
    "language": {
      "type": "string",
      "allowed": [
        "Afrikaans",
        "English (United States)",
        "Español (España)",
        "Français (France)",
        "Deutsch (Deutschland)",
        "Italiano",
        "Português (Brasil)",
        "日本語",
        "简体中文",
        "繁體中文"
      ],
      "default": "English (United States)",
      "required": true,
      "attribute": "language",
      "is_params": false
    },
    "functions": {
      "fetch_business_images": {
        "sort": 1,
        "default": false,
        "skip_squid": true,
        "credits_per_function": { "legacy": 10, "current": 1 }
      },
      "collect_business_details": {
        "sort": 0,
        "default": false,
        "skip_squid": true,
        "credits_per_function": { "legacy": 10, "current": 1 }
      },
      "extract_emails_from_website": {
        "sort": 2,
        "default": false,
        "skip_squid": true,
        "credits_per_function": { "legacy": 10, "current": 1 }
      }
    }
  }
}