Skip to content
Crawler

List Crawlers

Retrieve a list of all available crawlers and their capabilities

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.
GEThttps://api.lobstr.io/v1/crawlers

This endpoint returns a paginated list of all available crawlers on the lobstr.io platform. Each crawler is designed to extract specific types of data from different sources.

Use this endpoint to discover which crawlers are available for your data collection needs before creating squids or adding tasks.

Headers

Authorizationstringheaderrequired

Your API authentication token. Value: Token YOUR_API_KEY

Content-TypestringheaderDefault application/json

Content type of the request. Value: application/json

Response Structure

total_resultsinteger

Total number of crawlers available. Example: 71

limitinteger

Number of results per page. Example: 50

pageinteger

Current page number. Example: 1

total_pagesinteger

Total number of pages available. Example: 1

result_frominteger

Starting index of results in current page. Example: 1

result_tointeger

Ending index of results in current page. Example: 71

dataarray

Array of crawler objects with comprehensive details

nextstring | null

Full URL to the next page (null if on last page). Example: null

previousstring | null

Full URL to the previous page (null if on first page). Example: null

Crawler Object Fields

data[].idstring

Unique identifier (hash) for the crawler. Example: "4734d096159ef05210e0e1677e8be823"

data[].namestring

Human-readable name of the crawler. Example: "Google Maps Search Export"

data[].slugstring

URL-friendly identifier. Example: "google-maps-search-export"

data[].is_publicboolean

Whether the crawler is publicly available. Example: true

data[].is_availableboolean

Whether the crawler is currently available for use (false if paused). Example: true

data[].is_premiumboolean

Requires premium subscription or extra credits. Example: false

data[].has_issuesboolean

Currently experiencing issues (runs paused until fixed). Example: false

data[].max_concurrencyinteger

Maximum number of concurrent tasks allowed for this crawler. Example: 20

data[].credits_per_rowobject

Credits charged per result row, keyed by plan type: current is the rate on today's plans, legacy the rate kept by grandfathered subscriptions. Example: {"legacy": 1, "current": 1}

data[].credits_per_emailobject

Credits charged per email found, keyed the same way as credits_per_row. Only meaningful for crawlers with an email enrichment function. Example: {"legacy": 100, "current": 2}

data[].has_email_verificationboolean

Whether this crawler can verify the email addresses it finds. Example: true

data[].descriptionstring

What the crawler collects, as shown in the store. Example: "Export every follower of a public Instagram account"

data[].help_articlestring | null

Link to the crawler's help article, or null when it has none. Example: null

data[].rankinteger | null

Position in the public store listing. null for crawlers that are not ranked. Example: 1

data[].default_worker_statsobject | null

Throughput profile of the crawler's default worker — success_ratio, min_rate_per_worker, and max_rate_per_worker. null when no profile is published.

data[].email_worker_statsobject | null

Same throughput profile for the email enrichment worker. null when the crawler has no email function.

data[].chainsarray

Crawlers this one can chain into. Empty when the crawler supports no chaining — see Get Crawler Details for the shape of each entry.

data[].accountobject | null

Required account sync details (null if no authentication needed). Example: null

data[].iconstring

Base64-encoded SVG icon for UI display. Example: "data:image/svg+xml;base64,PHN2ZyB4bWxucz..."

Cache the list of crawlers locally as it doesn't change frequently. This reduces unnecessary API calls and improves your application's performance.

Each crawler has a unique hash identifier that you'll use when creating squids or configuring data collection tasks.

Check the crawler parameters endpoint to see what configuration options are available for each crawler before starting your scraping operations.

Available Crawlers

Quick reference of all public crawlers and their IDs. Use the Crawler ID as the squid_hash when creating a squid or as the crawler_id when querying by crawler.

Crawler Name Crawler ID
Amazon Reviews Scraper 787abfafd394be3b12ee2ac7d14303fe
AutoScout24 Listings Scraper 0a09add2a3c455037a972db81b1c0219
Bien'Ici Search Export 53317cbe332e408c5c46372a18e4fc76
Contact Details Scraper 7d83eacc7b94f14d5ffda28c6d357961
Facebook Ad Library Scraper f667dfc7398156ba2359836105184825
Facebook Business Page Scraper c80e36ed76bfae3c332e8983971403d7
Facebook Comments Scraper 3cf27be3402dbc529ed0db7d18135e8c
Facebook Marketplace Scraper 640e3d2f62769800fdb05f957f3a2415
Facebook Pages Search Export c47dfc2beba9fdc2472d3b3321b06b66
Facebook Private Group Posts Scraper 3eea12bc7d80cf05b6fc38eaf9f28f77
Facebook Public Group Posts Scraper 9b0bb2e04c6f7c475a9da9bb20530dac
Glassdoor Jobs Scraper 4bc153aae1e138d894b630bf165825ee
Google Maps Leads Scraper 4734d096159ef05210e0e1677e8be823
Google Maps Reviews Scraper e625ce52e98a6d4d1b53ef36dff7dacb
Google News Search Export 06faf01896336c01ffb5e96d13dd406c
Google Search Scraper f588385d11c48f8af51560e561077238
Idealista Listings Search Export 5f4b269e29e601e09fa50f4155b3970e
Immoweb Listings & Phones Search Export 9bca6a0a4cc5237b37798f8640dce087
Instagram Followers Scraper 5575499b33df1f165e72b29a12892a9e
Instagram Post Comments Scraper 438eabd1f513b6773600c166ad3ddbb4
Instagram Post Scraper d25fddae52eb78eb385e60bc2adb6d1e
Instagram Profile Scraper 5c55ffbf10fbbc7ee63ebb2bcaa332b7
Instagram Reels Scraper 4aad3ef0de6a5cea65b48dd575531a8c
Instagram Search & Hashtags Scraper 84adfa9c253a9a481b2f093a81e019e2
LaCentrale Cars Scraper 665c260fd9139121a4530d23b35b26db
LaCentrale Listing Status Checker 1d7fbb550feb1cf93e3e439039bcd687
Leboncoin Auto Message Sender fa9c988768a3d61f358916400f3c4b65
Leboncoin Boutiques Scraper c6e88128aef71c079e58f0518687e10c
Leboncoin Listing Scraper 9eade2d2a693bd871851806650e7fb4e
Leboncoin Listing Status Checker 0c60c33b95db65c86d5c9fc127b7b2aa
Leboncoin Listings & Phone Search Export 7bc4acdb18f2b90fdd5eb42b8e8251e9
Leboncoin Listings Search Export 33db1ca85160105eeb84d5aa51cfad10
LinkedIn Company Employees Scraper 08ac061ab290b5ae0f831e086c98d4b5
LinkedIn Company Scraper a6fab652671e54f962ce9ebe48d5ffed
LinkedIn Leads Scraper aa164b3c44d4f435fe5e04ddf6757296
LinkedIn Posts Commenters and Likers Scraper 8a309e0e0df48511111a9da6382b2ef8
LinkedIn Posts Scraper 6a9159b682cfb187977ae5c645e4df7a
LinkedIn Profile Scraper 5c11752d8687df2332c08247c4fb655a
Naukri Jobs Scraper 28ca7564a332a081a1dabda552d5c35b
Pages Jaunes Companies Search Export cd3e6b39d78365cd9d0ffbe1c7f5e5bd
PAP Search Export 9980d62d6a23161a08d1134c0cabdf5c
ParuVendu Listings Scraper 6a564030e814e6de4931637f7fb0c06e
Quora Scraper 7a8473daa49fb96711016aa655384bb3
Realtor Property Listings Scraper f388a9866f943bc5c29bde8b33587bd3
Reddit Scraper 6f99262f742b533d96936bf4088fba3e
Sales Navigator Companies Scraper 6b66cba236b64604581864230e1f4d36
Sales Navigator Leads Scraper a1d243d3824a1ac773414416eb80362d
Sales Navigator Profile Scraper 40d08f9bfec4a8775bb5586ce7b33a35
SeLoger Search Export 78f5839ee4b97c30e67eec391b907dd0
Threads Scraper 7308ae543e6abc70438d6b968d3a1b38
TikTok Comments Scraper 5fc7b468267ec881344a11ac05c15a76
TikTok Profile Scraper 089d76c66c2fd5c57d7ce7338d9ae7de
TikTok Top Ads Scraper b5079ede3002655d286a03ad29604ac6
TikTok Video Scraper 1e6cba50f2713539a2b0ac784184ed92
TripAdvisor Restaurants Search Export f781435f026b36b19ef74d591a077cb7
TripAdvisor Reviews Scraper 83970312f2402accbb3553b326a2c8eb
Trustpilot Reviews Scraper b3362ab52c6fab3d8897af79cbba380e
Twitter Followers & Following Scraper 8c6eed231ba0378c42efbe391f70d6d8
Twitter Profile Scraper 6c5227261fbbd3430f926d409f4c44bf
Twitter Search Results Scraper 1b16ff414d27920fb325b68436dbf5fc
Twitter User Tweets Scraper 9d6b83aaf1ae9d5b4a775d223e3eb5df
Upwork Jobs Scraper 681a0d2e2641326f7b87cda35513618a
Vinted Products Scraper ffd34f9b42a79b7323a048f09fc158e6
Yellow Pages Companies Search Export 03fcec3e9345dbf9e42a0a2f2f04c017
Yelp Reviews Scraper ea1c83f60e32a4e1de91a1ed1e41b268
Yelp Search Export ac92974d3c1c9f8d3997767f090fa0b8
YouTube Channel Scraper 8dce74bb8c2330fdafee70baa22b4709
YouTube Comments Scraper 66e9fbe07ae0a543576dc6ff241c229c
YouTube Transcript Scraper 7a9835f866cd42905e75aaeadebe6aba
Zillow Property Listings Scraper ed06bb5ca6544bbb638c544bac48ede1

Code Examples

curl -X GET "https://api.lobstr.io/v1/crawlers" \
  -H "Authorization: Token YOUR_API_KEY"

Response

200
{
  "total_results": 198,
  "limit": 50,
  "page": 1,
  "total_pages": 4,
  "result_from": 1,
  "result_to": 50,
  "data": [
    {
      "id": "6c5227261fbbd3430f926d409f4c44bf",
      "object": "crawler",
      "account": null,
      "name": "Twitter Profile Scraper",
      "slug": "twitter-profile-scraper",
      "description": "Scrape Twitter profile informations from a given username.",
      "help_article": null,
      "rank": 11,
      "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
      "is_public": true,
      "is_available": true,
      "is_premium": false,
      "has_issues": false,
      "max_concurrency": 20,
      "credits_per_row": { "legacy": 2, "current": 1 },
      "credits_per_email": { "legacy": 2, "current": 2 },
      "has_email_verification": false,
      "default_worker_stats": null,
      "email_worker_stats": null,
      "chains": []
    },
    {
      "id": "c47dfc2beba9fdc2472d3b3321b06b66",
      "object": "crawler",
      "account": {
        "type": "facebook-sync",
        "baseurl": "https://www.facebook.com",
        "cookies": [
          {
            "name": "c_user",
            "required": true
          },
          {
            "name": "xs",
            "required": true
          }
        ],
        "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+"
      },
      "name": "Facebook Pages Search Export",
      "slug": "facebook-pages-search-export",
      "description": "Collect all facebook pages from a keyword",
      "help_article": null,
      "rank": 14,
      "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
      "is_public": true,
      "is_available": true,
      "is_premium": false,
      "has_issues": false,
      "max_concurrency": 20,
      "credits_per_row": { "legacy": 20, "current": 1 },
      "credits_per_email": { "legacy": 2, "current": 2 },
      "has_email_verification": false,
      "default_worker_stats": null,
      "email_worker_stats": null,
      "chains": []
    },
    {
      "id": "a1d243d3824a1ac773414416eb80362d",
      "object": "crawler",
      "account": {
        "type": "sales-nav-sync",
        "baseurl": "https://www.linkedin.com",
        "cookies": [
          {
            "name": "li_at",
            "required": true
          },
          {
            "name": "JSESSIONID",
            "required": true
          }
        ],
        "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+"
      },
      "name": "Sales Navigator Leads Scraper",
      "slug": "sales-navigator-leads-scraper",
      "description": "Give us your properly built Sales Navigator Leads search URL(s). Grab all profiles available. Use our third-party service, and get mails!",
      "help_article": null,
      "rank": 4,
      "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
      "is_public": true,
      "is_available": true,
      "is_premium": false,
      "has_issues": false,
      "max_concurrency": 20,
      "credits_per_row": { "legacy": 5, "current": 1 },
      "credits_per_email": { "legacy": 2, "current": 2 },
      "has_email_verification": false,
      "default_worker_stats": null,
      "email_worker_stats": null,
      "chains": []
    }
  ],
  "next": null,
  "previous": null
}