Skip to content
Core Concepts

Pagination

How to iterate through large result sets using the lobstr.io API

The v1 API, for batchesSquids, tasks and runs: many calls on a schedule, delivered where you work. For one call that answers in seconds, use the API v2.

All list endpoints return paginated responses. This guide explains the pagination model and how to fetch all records efficiently.

Response structure

Every paginated endpoint returns the same envelope:

{
  "total_results": 243,
  "limit": 10,
  "page": 1,
  "total_pages": 25,
  "result_from": 1,
  "result_to": 10,
  "data": [...]
}
Field Type Description
total_results integer Total number of matching records
limit integer Number of records per page (max 100)
page integer Current page number (1-indexed)
total_pages integer Total number of pages
result_from integer Index of the first record on this page
result_to integer Index of the last record on this page
data array The records for this page

Parameters

Parameter Default Max Description
page 1 — Page number to retrieve
page_size 10 100 Number of results per page

Pass these as query parameters:

GET /v1/results?squid=abc123&page=2&page_size=50

The page-size query parameter's name varies by endpoint — Get Results uses page_size, while List Accounts uses limit. Check the specific endpoint's own page if you're unsure.

Iterating through all pages

import requests

API_KEY = "YOUR_API_KEY"
headers = {"Authorization": f"Token {API_KEY}"}

def get_all_results(squid_id):
    all_data = []
    page = 1
    page_size = 100

    while True:
        response = requests.get(
            "https://api.lobstr.io/v1/results",
            headers=headers,
            params={"squid": squid_id, "page": page, "page_size": page_size}
        )
        body = response.json()

        all_data.extend(body["data"])
        print(f"Fetched page {page}/{body['total_pages']} ({len(all_data)}/{body['total_results']} records)")

        if page >= body["total_pages"]:
            break

        page += 1

    return all_data

results = get_all_results("YOUR_SQUID_ID")
print(f"Total records fetched: {len(results)}")

Free plan

Every plan, the free plan included, pages through every result: there is no row cap since November 3, 2026, and no warning field on the response.

Endpoints that support pagination

Endpoint Path
List Squids GET /v1/squids
List Tasks GET /v1/tasks
List Runs GET /v1/runs
Get Run Tasks GET /v1/runs/{run_hash}/tasks
Get Results GET /v1/results
List Accounts GET /v1/accounts

Use the maximum page size (page_size=100 for Get Results; check each endpoint's own parameter name) to minimize the number of requests when fetching large datasets.