CLI
Command-line interface for the lobstr.io API. Run scrapers, manage squids, download results — all from your terminal.
Installation
pip install lobstrioRequires Python 3.10+.
Environment variables
| Variable | Description |
|---|---|
LOBSTR_TOKEN | API token, overriding the one saved by lobstr config set-token. |
LOBSTR_API_BASE | Point the CLI at a different API base URL (e.g. a staging or self-hosted instance). Defaults to production (https://api.lobstr.io/v1/) when unset. |
export LOBSTR_TOKEN=your_api_key
Combines create, configure, add tasks, run, and download into a single command.
Search and explore every crawler by slug before running.
Download results with a live progress bar while runs complete in the background.
Configure delivery to email, Google Sheets, S3, SFTP, or webhook from the terminal.
Quick Start
# Scrape Google Maps in one command lobstr go google-maps-leads-scraper \ "https://google.com/maps/search/restaurants+paris" \ -o leads.csv # Keyword-based search lobstr go google-search-scraper "pizza delivery" --key keyword # With parameters and concurrency lobstr go google-maps-leads-scraper url1 url2 \ --param max_results=200 --concurrency 3 # With add-ons (extra credits) lobstr go youtube-channel-scraper "https://www.youtube.com/@YouTube" \ --function get_videos # Don't download, just kick off lobstr go twitter-profile-scraper @elonmusk --no-download # Account-backed crawler (e.g. Sales Navigator): auto-picks the one # healthy, unlocked matching account if there's exactly one lobstr go sales-navigator-leads-scraper "https://linkedin.com/..." # ...or name it explicitly (needed when there's more than one candidate) lobstr go sales-navigator-leads-scraper "https://linkedin.com/..." --account jane@example.com
Account-backed crawlers (LinkedIn, Sales Navigator, …) need a connected account before they can run. go links one automatically when exactly one healthy, unlocked account matches the crawler's account type; when it can't, it names the matching candidates, or the account type to connect, so you can pass --account yourself. It never auto-picks onto a squid that already has an account, so a reused squid (go --name) keeps what it had. A wrong-type account, an unknown account, or --account on a crawler that doesn't need one each fail with a clear message before anything is created; if linking fails after the squid is created, or the command is interrupted before the run starts, the squid is removed rather than left behind.
Command Reference
Crawlers
| Command | Description |
|---|---|
crawlers ls | List all available crawlers |
crawlers search <query> | Search crawlers by keyword |
crawlers show <slug> | Get crawler details by slug |
crawlers params <slug> | Get supported parameters for a crawler |
crawlers attrs <slug> | Get output attributes for a crawler |
crawlers ls, show, and search show which account type a crawler needs (for example sales-nav-sync) in the "Needs Account" column, in place of a plain yes or no — that's the value to look for when picking a matching account for --account below. A crawler that needs no account still shows "no". --json output is unchanged.
Squids
| Command | Description |
|---|---|
squid create <crawler> --name "My Scraper" | Create a new squid for a crawler |
squid ls | List all squids in your account |
squid show <id> | Get squid details by ID |
squid update <id> --param key=value | Update squid settings or parameters |
squid update <id> --function <name> | Turn an add-on on (<name>=false turns it off). Repeatable; names from crawlers params <crawler>. Also on go. |
squid update <id> --account <username or hash> | Link a connected account to the squid, in addition to any it already has |
squid update <id> --replace-accounts --account <username or hash> | Set the squid's account list outright, detaching any account not passed |
squid empty <id> | Remove all tasks from a squid |
squid rm <id> | Permanently delete a squid |
Linking an account. --account (repeatable) takes either an account's username or a hash — a hash prefix is enough. It's additive: the account is added to whatever the squid already has. --replace-accounts sets the list outright instead, printing which accounts it's about to detach before doing it; passed with no --account at all, it detaches every account. squid create has no --account — the API only accepts an account list on squid update.
Tasks
| Command | Description |
|---|---|
task add <squid> <url1> <url2> ... | Add one or more task URLs to a squid |
task upload <squid> <file.csv> | Bulk upload tasks from a CSV file |
task ls <squid> | List tasks for a squid |
task rm <id> | Delete a task by ID |
Runs
| Command | Description |
|---|---|
run start <squid> | Start a new run and return immediately |
run start <squid> --wait | Start a run and poll until it completes |
run start <squid> --wait --timeout <s> --poll <s> | Wait with a time limit and custom poll interval — on timeout, exits non-zero and the run keeps going |
run watch <id> [--timeout <s>] [--poll <s>] | Attach to a running run and follow its progress |
run ls <squid> | List all runs for a squid |
run show <id> | Get run details by ID |
run stats <id> | Get statistics for a run |
run abort <id> | Abort an in-progress run |
run download <id> -o results.csv | Download run results to a file |
--wait, --timeout and --download all follow the run to completion. --timeout <seconds> bounds the wait — if the run hasn't finished by then, the command exits non-zero and prints how to re-attach (lobstr run watch <id>); the run is not aborted and keeps running server-side. --poll <seconds> sets how often status is checked (default 3).
Results
| Command | Description |
|---|---|
results get <squid> --format csv -o data.csv | Get results for a squid and export to file |
Delivery
| Command | Description |
|---|---|
delivery email <squid> --email user@example.com | Configure email delivery for a squid |
delivery webhook <squid> --url https://... | Configure webhook delivery for a squid |
delivery s3 <squid> --bucket my-bucket | Configure Amazon S3 delivery for a squid |
delivery googlesheet <squid> --url https://... | Configure Google Sheet delivery for a squid |
delivery sftp <squid> --host ... --username ... --password ... --directory ... | Configure SFTP delivery for a squid |
delivery test-email --email user@example.com | Test an email delivery config before activating it |
delivery test-webhook --url https://... | Test a webhook delivery config before activating it |
delivery test-s3 --bucket my-bucket | Test an S3 delivery config before activating it |
delivery test-googlesheet --url https://... | Test a Google Sheet delivery config before activating it |
delivery test-sftp --host ... --username ... --password ... --directory ... | Test an SFTP delivery config before activating it |