parse

Copy as Markdown

Give agent-data a website URL and describe the data you need. It checks for that website’s existing public API or starts a build for a new source. Building an API is free, subject to daily build limits. Calls to the resulting API use your included allowance or credits, including calls to an API you requested.

Start a build

Complete CLI setup first, including your API key. Replace the URL and task with the website and data you want:

agent-data parse https://books.toscrape.com --task "Search books by title and return title, price, availability and product URL"

There is one canonical public API per source hostname. If one already exists, parse returns it with matched: true, match_scope: "source" and a documentation command. A match does not confirm that it supports your task: task_support: "unverified" means you must inspect the docs. Use the existing endpoints if they fit, or request an additional endpoint for missing data.

Hostnames are case-insensitive and ignore a leading www, paths and query strings. Other subdomains remain separate sources. Public APIs cannot be duplicated or arbitrarily overwritten.

For a new source, the response includes a parse_job_id and links to follow the build. Keep those identifiers. A source with a build already running returns a conflict instead of starting another build.

Follow progress

Replace <job-id> with the returned parse_job_id:

agent-data parse list
agent-data parse status <job-id>
agent-data parse events <job-id> --follow
agent-data parse cancel <job-id>

A build normally moves through building, deploying, publishing, probing and live. Acceptance is not completion: wait for live, inspect the returned API docs and use its verified call example. Builds can take several minutes.

If the build is blocked or fails, inspect its status and suggested next action. Do not start repeated builds to work around a source conflict. Some blockers require help from the team. Cancellation applies to unfinished builds.

If the next action offers continuation from a saved checkpoint, resume the existing job:

agent-data parse resume <job-id>

This continues the saved session with a fresh time budget. A blocked or failed job is not always resumable; follow the reported next action if continuation is unavailable.

Describe the endpoint you need

Include the relevant website URL, the records and fields you want, the inputs users should be able to supply, and how much data each call should return. For example: search books by title, accept a search term and page number, and return each book’s title, price and product URL.

Use --dry-run to preview the request. With --allow-interrupts required, the build can pause for a question. Answer using the interrupt identifier from its status:

agent-data parse respond <job-id> --interrupt-id <interrupt-id> --answer "Your answer"

To request an additive change to your own build, use agent-data parse revise <job-id> "<change>". To contribute endpoints to someone else’s public API, use the extension workflow.

Health and daily checks

Published APIs receive scheduled daily checks. Check API documentation and the dashboard for the latest recorded health and verification time. Coverage can vary by API; a successful check does not prove that every possible input or source works.

A failing parse API may enter repair, or may need manual attention. A degraded result means calls may fail; it does not promise that a repair has started. Daily operational results are summarized for the agent-data team.