Skip to content
Delivery

Configure Amazon S3 Delivery

Export squid run results directly to an AWS S3 bucket

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.
POSThttps://api.lobstr.io/v1/delivery?squid={squid_hash}

Configure Amazon S3 delivery to export run results directly to an AWS S3 bucket. Results will be automatically uploaded to your specified bucket and path when runs complete.

Authentication Options

You can authenticate with S3 in two ways:

  1. AWS Credentials: Provide aws_access_key and aws_secret_key in the request
  2. Bucket Policy: If you don't provide credentials, ensure your bucket policy allows public WRITE permissions

Headers

Authorizationstringheaderrequired

Your API authentication token. Value: Token YOUR_API_KEY

Content-Typestringheaderrequired

Must be application/json. Value: application/json

Query Parameters

squidstringqueryrequired

The unique identifier (hash) of the squid for which to configure S3 delivery. Example: c106a44a98044ef18acc59986ae10967

Request Body

s3_fields.bucketstringrequired

Name of the S3 bucket where data will be stored. Example: "my-bucket"

s3_fields.is_activebooleanrequired

Set to true to enable S3 delivery, false to disable. Example: true

s3_fields.target_pathstringrequired

Folder path inside the bucket where results will be saved. Example: "exports"

s3_fields.aws_access_keystring

(Optional) AWS Access Key for authentication. If not provided, bucket must have public WRITE permissions. Example: "AKIAIOSFODNN7EXAMPLE"

s3_fields.aws_secret_keystring

(Optional) AWS Secret Key for authentication. If not provided, bucket must have public WRITE permissions. Example: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Response Field Explanations

s3_fields.bucketstring

S3 bucket name. Example: "my-bucket"

s3_fields.is_activeboolean

Whether S3 delivery is active. Example: true

s3_fields.target_pathstring

Target folder path in bucket. Example: "exports"

s3_fields.aws_access_keystring

AWS Access Key (returned if provided). Example: "AKIAIOSFODNN7EXAMPLE"

s3_fields.aws_secret_keystring

AWS Secret Key (returned if provided). Example: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

If you don't provide AWS credentials, your bucket policy MUST allow public WRITE permissions for Lobstr to upload files.

For better security, provide AWS credentials instead of making your bucket publicly writable. Use IAM users with limited permissions.

The target_path creates a folder structure in your bucket. Results will be uploaded to s3://bucket/target_path/filename.csv

Use the Test S3 endpoint to verify your bucket permissions and credentials before activating automatic delivery.

AWS credentials are stored and returned in API responses. Ensure you're using HTTPS and handle responses securely.

Code Examples

curl -X POST "https://api.lobstr.io/v1/delivery?squid=YOUR_SQUID_HASH" \
  -H "Authorization: Token YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "s3_fields": {
      "bucket": "my-bucket",
      "is_active": true,
      "target_path": "exports",
      "aws_access_key": "your_access_key",
      "aws_secret_key": "your_secret_key"
    }
  }'

Response

201
{
  "s3_fields": {
    "bucket": "my-bucket",
    "is_active": true,
    "target_path": "exports",
    "aws_access_key": "your_access_key",
    "aws_secret_key": "your_secret_key"
  }
}