# CLI

Command-line interface for the lobstr.io API. Run scrapers, manage squids, download results — all from your terminal.

## Installation

```bash
pip install lobstrio
```

Requires Python 3.10+.

## Environment variables

| Variable | Description |
|---|---|
| `LOBSTR_TOKEN` | API token, overriding the one saved by `lobstr config set-token`. |
| `LOBSTR_API_BASE` | Point the CLI at a different API base URL (e.g. a staging or self-hosted instance). Defaults to production (`https://api.lobstr.io/v1/`) when unset. |

```bash
export LOBSTR_TOKEN=your_api_key
```

<CardGroup cols={4}>
  <Card title="lobstr go" icon="terminal">
    Combines create, configure, add tasks, run, and download into a single command.
  </Card>
  <Card title="55+ Crawlers" icon="layer-group">
    Search and explore every crawler by slug before running.
  </Card>
  <Card title="CSV & JSON Export" icon="file-export">
    Download results with a live progress bar while runs complete in the background.
  </Card>
  <Card title="Webhooks & Cloud" icon="cloud">
    Configure delivery to email, Google Sheets, S3, SFTP, or webhook from the terminal.
  </Card>
</CardGroup>

## Quick Start

<CodeGroup>
```bash lobstr go
# Scrape Google Maps in one command
lobstr go google-maps-leads-scraper \
  "https://google.com/maps/search/restaurants+paris" \
  -o leads.csv

# Keyword-based search
lobstr go google-search-scraper "pizza delivery" --key keyword

# With parameters and concurrency
lobstr go google-maps-leads-scraper url1 url2 \
  --param max_results=200 --concurrency 3

# With add-ons (extra credits)
lobstr go youtube-channel-scraper "https://www.youtube.com/@YouTube" \
  --function get_videos

# Don't download, just kick off
lobstr go twitter-profile-scraper @elonmusk --no-download

# Account-backed crawler (e.g. Sales Navigator): auto-picks the one
# healthy, unlocked matching account if there's exactly one
lobstr go sales-navigator-leads-scraper "https://linkedin.com/..."

# ...or name it explicitly (needed when there's more than one candidate)
lobstr go sales-navigator-leads-scraper "https://linkedin.com/..." --account jane@example.com
```

```bash Step by step
lobstr crawlers search "google maps"
lobstr squid create google-maps-leads-scraper --name "Paris Restaurants"
lobstr task add SQUID_ID "https://google.com/maps/search/restaurants+paris"
lobstr run start SQUID_ID --wait
lobstr results get SQUID_ID --format csv -o results.csv

# Check who you are
lobstr whoami
```
</CodeGroup>

<Note>
  **Account-backed crawlers** (LinkedIn, Sales Navigator, …) need a connected account before they can run. `go` links one automatically when exactly one healthy, unlocked account matches the crawler's account type; when it can't, it names the matching candidates, or the account type to connect, so you can pass `--account` yourself. It never auto-picks onto a squid that already has an account, so a reused squid (`go --name`) keeps what it had. A wrong-type account, an unknown account, or `--account` on a crawler that doesn't need one each fail with a clear message before anything is created; if linking fails after the squid is created, or the command is interrupted before the run starts, the squid is removed rather than left behind.
</Note>

## Command Reference

### Crawlers

| Command | Description |
|---|---|
| `crawlers ls` | List all available crawlers |
| `crawlers search <query>` | Search crawlers by keyword |
| `crawlers show <slug>` | Get crawler details by slug |
| `crawlers params <slug>` | Get supported parameters for a crawler |
| `crawlers attrs <slug>` | Get output attributes for a crawler |

<Note>
  `crawlers ls`, `show`, and `search` show which account type a crawler needs (for example `sales-nav-sync`) in the "Needs Account" column, in place of a plain yes or no — that's the value to look for when picking a matching account for `--account` below. A crawler that needs no account still shows "no". `--json` output is unchanged.
</Note>

### Squids

| Command | Description |
|---|---|
| `squid create <crawler> --name "My Scraper"` | Create a new squid for a crawler |
| `squid ls` | List all squids in your account |
| `squid show <id>` | Get squid details by ID |
| `squid update <id> --param key=value` | Update squid settings or parameters |
| `squid update <id> --function <name>` | Turn an add-on on (`<name>=false` turns it off). Repeatable; names from `crawlers params <crawler>`. Also on `go`. |
| `squid update <id> --account <username or hash>` | Link a connected account to the squid, in addition to any it already has |
| `squid update <id> --replace-accounts --account <username or hash>` | Set the squid's account list outright, detaching any account not passed |
| `squid empty <id>` | Remove all tasks from a squid |
| `squid rm <id>` | Permanently delete a squid |

<Note>
  **Linking an account.** `--account` (repeatable) takes either an account's username or a hash — a hash prefix is enough. It's additive: the account is added to whatever the squid already has. `--replace-accounts` sets the list outright instead, printing which accounts it's about to detach before doing it; passed with no `--account` at all, it detaches every account. `squid create` has no `--account` — the API only accepts an account list on `squid update`.
</Note>

### Tasks

| Command | Description |
|---|---|
| `task add <squid> <url1> <url2> ...` | Add one or more task URLs to a squid |
| `task upload <squid> <file.csv>` | Bulk upload tasks from a CSV file |
| `task ls <squid>` | List tasks for a squid |
| `task rm <id>` | Delete a task by ID |

### Runs

| Command | Description |
|---|---|
| `run start <squid>` | Start a new run and return immediately |
| `run start <squid> --wait` | Start a run and poll until it completes |
| `run start <squid> --wait --timeout <s> --poll <s>` | Wait with a time limit and custom poll interval — on timeout, exits non-zero and the run keeps going |
| `run watch <id> [--timeout <s>] [--poll <s>]` | Attach to a running run and follow its progress |
| `run ls <squid>` | List all runs for a squid |
| `run show <id>` | Get run details by ID |
| `run stats <id>` | Get statistics for a run |
| `run abort <id>` | Abort an in-progress run |
| `run download <id> -o results.csv` | Download run results to a file |

<Note>
  `--wait`, `--timeout` and `--download` all follow the run to completion. `--timeout <seconds>` bounds the wait — if the run hasn't finished by then, the command exits non-zero and prints how to re-attach (`lobstr run watch <id>`); the run is **not** aborted and keeps running server-side. `--poll <seconds>` sets how often status is checked (default 3).
</Note>

### Results

| Command | Description |
|---|---|
| `results get <squid> --format csv -o data.csv` | Get results for a squid and export to file |

### Delivery

| Command | Description |
|---|---|
| `delivery email <squid> --email user@example.com` | Configure email delivery for a squid |
| `delivery webhook <squid> --url https://...` | Configure webhook delivery for a squid |
| `delivery s3 <squid> --bucket my-bucket` | Configure Amazon S3 delivery for a squid |
| `delivery googlesheet <squid> --url https://...` | Configure Google Sheet delivery for a squid |
| `delivery sftp <squid> --host ... --username ... --password ... --directory ...` | Configure SFTP delivery for a squid |
| `delivery test-email --email user@example.com` | Test an email delivery config before activating it |
| `delivery test-webhook --url https://...` | Test a webhook delivery config before activating it |
| `delivery test-s3 --bucket my-bucket` | Test an S3 delivery config before activating it |
| `delivery test-googlesheet --url https://...` | Test a Google Sheet delivery config before activating it |
| `delivery test-sftp --host ... --username ... --password ... --directory ...` | Test an SFTP delivery config before activating it |
