Skip to content
Crawler

Get Crawler Details

Retrieve detailed information about a specific crawler including input parameters and output fields

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.
GEThttps://api.lobstr.io/v1/crawlers/{crawler_hash}

This endpoint retrieves detailed information about a specific crawler using its hash ID. It helps you understand each crawler's availability, cost, parameters, and potential issues before using it.

Headers

Authorizationstringheaderrequired

Your API authentication token. Value: Token YOUR_API_KEY

Content-TypestringheaderDefault application/json

Content type of the request. Value: application/json

Path Parameters

crawler_hashstringpathrequired

The unique identifier (hash/id) of the crawler. Example: 4734d096159ef05210e0e1677e8be823

Response Field Explanations

idstring

Unique identifier for the crawler. Example: "c106a44a98044ef18acc59986ae10967"

namestring

Human-readable name (e.g., "Google Maps Search Export"). Example: "Google Maps (1)"

slugstring

URL-friendly identifier. Example: "google-maps-leads-scraper"

descriptionstring

What the crawler collects, as shown in the store. Example: "Give us your properly built Google Maps Search URL(s). Grab all results available. Phone, rating, website included!"

help_articlestring | null

Link to the crawler's help article, or null when it has none. Example: null

rankinteger | null

Position in the public store listing. null for crawlers that are not ranked. Example: 1

is_publicboolean

Whether the crawler is publicly accessible. Example: true

is_availableboolean

Whether the crawler is available for runs (false if paused). Example: true

is_premiumboolean

Requires premium subscription or extra credits. Example: false

has_issuesboolean

Currently experiencing issues (runs paused until fixed). Example: false

accountobject | null

Required account sync details (null if no authentication needed). Example: null

inputarray

Array of configurable parameters (task-level and squid-level). Example: []

resultarray

Array of output field names this crawler collects. Example: []

max_concurrencyinteger

Maximum number of workers that can run this crawler in parallel. Example: 20

credits_per_rowobject

Credits charged per result row, keyed by plan type: current is the rate on today's plans, legacy the rate kept by grandfathered subscriptions. Example: {"legacy": 1, "current": 1}

credits_per_emailobject

Credits charged per email found, keyed the same way as credits_per_row. Only meaningful for crawlers with an email enrichment function. Example: {"legacy": 100, "current": 2}

has_email_verificationboolean

Whether this crawler can verify the email addresses it finds. Example: true

default_worker_statsobject | null

Throughput profile of the crawler's default worker — success_ratio, min_rate_per_worker, and max_rate_per_worker. Used to project how long a run will take. null when no profile is published. Example: {"success_ratio": 1, "max_rate_per_worker": 200, "min_rate_per_worker": 100}

email_worker_statsobject | null

Same throughput profile for the email enrichment worker, which is typically slower and has a lower success ratio. null when the crawler has no email function. Example: {"success_ratio": 0.3, "max_rate_per_worker": 30, "min_rate_per_worker": 20}

chainsarray

Crawlers this one can chain into — each entry describes the target crawler, how its fields map from this crawler's output (field_map), whether it autostarts by default, and the squid-level parameters it accepts. Empty when the crawler supports no chaining. See Configure Chain for a worked example.

Understanding Input Parameters

input[].levelstring

Parameter level: "task" (per URL/task) or "squid" (overall run behavior). Example: ""

input[].is_paramsstring

For task-level: true indicates this parameter is used per task submission. Example: ""

input[].is_detailsstring

For squid-level: true indicates this controls overall behavior or features. Example: ""

input[].functionstring

If true, this is an optional add-on feature with extra credit costs. Example: ""

input[].credits_per_functionobject

Extra credits per result when this function is on, keyed by plan type like credits_per_row: current is the rate on today's plans, legacy the rate kept by grandfathered subscriptions (Get User Profile → plan[].is_legacy). Example: {"legacy": 80, "current": 4}

input[].credits_per_filterobject

Credits per result checked when this filter is on, keyed the same way. Example: {"legacy": 0.2, "current": 0.2}

input[].namestring

Parameter name to use when configuring the crawler. Example: "Google Maps (1)"

input[].typestring

Data type of the parameter (string, int, boolean, etc.). Example: ""

input[].requiredstring

Whether this parameter is mandatory. Example: ""

input[].defaultstring

Default value if parameter is not provided. Example: ""

Check the 'has_issues' flag before creating tasks. If true, the crawler is experiencing problems and runs are paused.

Parameters with 'function: true' are optional add-ons that cost extra credits per result (specified in 'credits_per_function', keyed by plan type).

Some crawlers require account synchronization (see 'account' field). You'll need to sync cookies before running these crawlers.

Use the 'result' array to know exactly what fields will be returned in your data. This helps you plan your data processing pipeline.

Code Examples

curl -X GET "https://api.lobstr.io/v1/crawlers/4734d096159ef05210e0e1677e8be823" \
  -H "Authorization: Token YOUR_API_KEY"

Response

200
{
  "id": "4734d096159ef05210e0e1677e8be823",
  "object": "crawler",
  "account": null,
  "name": "Google Maps Leads Scraper",
  "slug": "google-maps-leads-scraper",
  "description": "Give us your properly built Google Maps Search URL(s). Grab all results available. Phone, rating, website included!",
  "help_article": null,
  "rank": 1,
  "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
  "is_public": true,
  "is_available": true,
  "is_premium": false,
  "has_issues": false,
  "input": [
    {
      "name": "url",
      "type": "string",
      "level": "task",
      "regex": "http(.*)google(.*)/maps/(search|place)(.*)",
      "default": null,
      "example": "https://www.google.com/maps/search/restaurant/@43.2928346,5.3662584,14z",
      "required": false,
      "is_details": false,
      "description": "A string representing the Google Maps Search URL, containing all establishments the crawler will collect."
    },
    {
      "name": "activity",
      "type": "string",
      "level": "task",
      "value": "restaurant",
      "example": "restaurant",
      "required": false,
      "is_params": true,
      "description": "The activity of the search location."
    },
    {
      "max": 500,
      "name": "max_results",
      "type": "int",
      "level": "squid",
      "default": "",
      "example": 200,
      "required": false,
      "is_details": true,
      "description": "Maximum number of results retrieved *per task*. Leave empty to collect everything Google Maps returns for the search: up to about 330 places where the map is dense, fewer elsewhere."
    },
    {
      "name": "extract_emails_from_website",
      "type": "boolean",
      "level": "squid",
      "default": false,
      "display": "Extract Emails From Website",
      "function": true,
      "priority": 3,
      "required": false,
      "is_details": true,
      "description": "Extract email addresses from the business's website.",
      "credits_per_function": { "legacy": 10, "current": 1 }
    },
    {
      "name": "collect_business_details",
      "type": "boolean",
      "level": "squid",
      "default": false,
      "display": "Collect Business Details",
      "function": true,
      "priority": 1,
      "required": false,
      "is_details": true,
      "description": "Extract attributes like plus_code, poi, health, and last_opening_hours_updated_at.",
      "credits_per_function": { "legacy": 10, "current": 1 }
    }
  ],
  "result": [
    "zero_x",
    "cid",
    "url",
    "name",
    "address",
    "country_name",
    "country_code",
    "zip_code",
    "city",
    "lat",
    "lng",
    "ratings",
    "score",
    "category",
    "main_image_url",
    "phone",
    "website",
    "menu",
    "plus_code",
    "is_temporarily_closed",
    "is_permanently_closed",
    "has_owner",
    "opening_hours",
    "email",
    "facebook",
    "instagram"
  ],
  "max_concurrency": 20,
  "credits_per_row": {
    "legacy": 1,
    "current": 1
  },
  "credits_per_email": {
    "legacy": 100,
    "current": 2
  },
  "has_email_verification": true,
  "default_worker_stats": {
    "success_ratio": 1,
    "max_rate_per_worker": 200,
    "min_rate_per_worker": 100
  },
  "email_worker_stats": {
    "success_ratio": 0.3,
    "max_rate_per_worker": 30,
    "min_rate_per_worker": 20
  },
  "chains": [
    {
      "title": "Google Maps Reviews Scraper",
      "subtitle": "Collect all reviews from a GoogleMaps Etablishment URL!",
      "target": "googlemapreviews_matrix",
      "target_module_id": "e625ce52e98a6d4d1b53ef36dff7dacb",
      "field_map": { "url": "url" },
      "autostart_default": true
    }
  ]
}

The input and result arrays above are abridged. At the time of writing this crawler exposes 25 input parameters and 87 result fields — always read them from the live response rather than hard-coding them.