Skip to content
Developer Tools

CLI

Command-line interface for the lobstr.io API. Run scrapers, manage squids, download results — all from your terminal.

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.

Installation

pip install lobstrio

Requires Python 3.10+.

Environment variables

Variable Description
LOBSTR_TOKEN API token, overriding the one saved by lobstr config set-token.
LOBSTR_API_BASE Point the CLI at a different API base URL (e.g. a staging or self-hosted instance). Defaults to production (https://api.lobstr.io/v1/) when unset.
export LOBSTR_TOKEN=your_api_key
lobstr go

Combines create, configure, add tasks, run, and download into a single command.

55+ Crawlers

Search and explore every crawler by slug before running.

CSV & JSON Export

Download results with a live progress bar while runs complete in the background.

Webhooks & Cloud

Configure delivery to email, Google Sheets, S3, SFTP, or webhook from the terminal.

Quick Start

# Scrape Google Maps in one command
lobstr go google-maps-leads-scraper \
  "https://google.com/maps/search/restaurants+paris" \
  -o leads.csv

# Keyword-based search
lobstr go google-search-scraper "pizza delivery" --key keyword

# With parameters and concurrency
lobstr go google-maps-leads-scraper url1 url2 \
  --param max_results=200 --concurrency 3

# With add-ons (extra credits)
lobstr go youtube-channel-scraper "https://www.youtube.com/@YouTube" \
  --function get_videos

# Don't download, just kick off
lobstr go twitter-profile-scraper @elonmusk --no-download

# Account-backed crawler (e.g. Sales Navigator): auto-picks the one
# healthy, unlocked matching account if there's exactly one
lobstr go sales-navigator-leads-scraper "https://linkedin.com/..."

# ...or name it explicitly (needed when there's more than one candidate)
lobstr go sales-navigator-leads-scraper "https://linkedin.com/..." --account jane@example.com

Account-backed crawlers (LinkedIn, Sales Navigator, …) need a connected account before they can run. go links one automatically when exactly one healthy, unlocked account matches the crawler's account type; when it can't, it names the matching candidates, or the account type to connect, so you can pass --account yourself. It never auto-picks onto a squid that already has an account, so a reused squid (go --name) keeps what it had. A wrong-type account, an unknown account, or --account on a crawler that doesn't need one each fail with a clear message before anything is created; if linking fails after the squid is created, or the command is interrupted before the run starts, the squid is removed rather than left behind.

Command Reference

Crawlers

Command Description
crawlers ls List all available crawlers
crawlers search <query> Search crawlers by keyword
crawlers show <slug> Get crawler details by slug
crawlers params <slug> Get supported parameters for a crawler
crawlers attrs <slug> Get output attributes for a crawler

crawlers ls, show, and search show which account type a crawler needs (for example sales-nav-sync) in the "Needs Account" column, in place of a plain yes or no — that's the value to look for when picking a matching account for --account below. A crawler that needs no account still shows "no". --json output is unchanged.

Squids

Command Description
squid create <crawler> --name "My Scraper" Create a new squid for a crawler
squid ls List all squids in your account
squid show <id> Get squid details by ID
squid update <id> --param key=value Update squid settings or parameters
squid update <id> --function <name> Turn an add-on on (<name>=false turns it off). Repeatable; names from crawlers params <crawler>. Also on go.
squid update <id> --account <username or hash> Link a connected account to the squid, in addition to any it already has
squid update <id> --replace-accounts --account <username or hash> Set the squid's account list outright, detaching any account not passed
squid empty <id> Remove all tasks from a squid
squid rm <id> Permanently delete a squid

Linking an account. --account (repeatable) takes either an account's username or a hash — a hash prefix is enough. It's additive: the account is added to whatever the squid already has. --replace-accounts sets the list outright instead, printing which accounts it's about to detach before doing it; passed with no --account at all, it detaches every account. squid create has no --account — the API only accepts an account list on squid update.

Tasks

Command Description
task add <squid> <url1> <url2> ... Add one or more task URLs to a squid
task upload <squid> <file.csv> Bulk upload tasks from a CSV file
task ls <squid> List tasks for a squid
task rm <id> Delete a task by ID

Runs

Command Description
run start <squid> Start a new run and return immediately
run start <squid> --wait Start a run and poll until it completes
run start <squid> --wait --timeout <s> --poll <s> Wait with a time limit and custom poll interval — on timeout, exits non-zero and the run keeps going
run watch <id> [--timeout <s>] [--poll <s>] Attach to a running run and follow its progress
run ls <squid> List all runs for a squid
run show <id> Get run details by ID
run stats <id> Get statistics for a run
run abort <id> Abort an in-progress run
run download <id> -o results.csv Download run results to a file

--wait, --timeout and --download all follow the run to completion. --timeout <seconds> bounds the wait — if the run hasn't finished by then, the command exits non-zero and prints how to re-attach (lobstr run watch <id>); the run is not aborted and keeps running server-side. --poll <seconds> sets how often status is checked (default 3).

Results

Command Description
results get <squid> --format csv -o data.csv Get results for a squid and export to file

Delivery

Command Description
delivery email <squid> --email user@example.com Configure email delivery for a squid
delivery webhook <squid> --url https://... Configure webhook delivery for a squid
delivery s3 <squid> --bucket my-bucket Configure Amazon S3 delivery for a squid
delivery googlesheet <squid> --url https://... Configure Google Sheet delivery for a squid
delivery sftp <squid> --host ... --username ... --password ... --directory ... Configure SFTP delivery for a squid
delivery test-email --email user@example.com Test an email delivery config before activating it
delivery test-webhook --url https://... Test a webhook delivery config before activating it
delivery test-s3 --bucket my-bucket Test an S3 delivery config before activating it
delivery test-googlesheet --url https://... Test a Google Sheet delivery config before activating it
delivery test-sftp --host ... --username ... --password ... --directory ... Test an SFTP delivery config before activating it