Document Parsing

Document Parsing uses state-of-the-art research techniques to extract and visualize structured data from a wide range of document types, including but not limited to PDFs, images, Word documents (DOC/DOCX), and PowerPoint files (PPT/PPTX). It's designed to handle complex layouts such as tables, charts, and mixed text-image content.

Document parsing is available through the Agent Bricks UI, in SQL, with the REST API, and with the Asynchronous API (Beta).

Create a document parsing agent in the UI

Requirements

• A workspace that includes the following:
  • Serverless compute enabled. See Serverless compute requirements.
  • Unity Catalog enabled. See Enable a workspace for Unity Catalog.
  • Access to a serverless usage policy with a nonzero budget.
• This function is only available in some regions, see AI function availability.
  • The ai_parse_document function is also available to workspaces with the Enhanced Security and Compliance add-on.

Use Document Parsing to parse your documents and visualize their structure.

1. Go to Agents in the left navigation pane of your workspace.
2. Click Create Agent > Document Parsing.
3. Select your source document. You can choose to upload a file or select one from an existing Unity Catalog catalog. Supported formats include: PDF, images, DOC/DOCX, and PPT/PPTX.
4. Click Parse document.

Parsing your document can take a few minutes. When complete, Document Parsing shows the source document on the left and the parsed document on the right. You can choose to view the parsed document as Formatted text or Raw JSON.

Process and query results

To view the ai_parse_document query and run it on more documents, click Use Agent and choose either to run the query from the SQL Editor or Notebook. You can edit the query to point to the volume or table your documents live in.

To prepare the parsed output for retrieval (RAG), use ai_prep_search (Beta) downstream.

Parse documents in SQL

Call ai_parse_document in SQL to parse documents at scale:

SELECT
  ai_parse_document(
    content,
    map('version', '2.0')
  ) AS parsed
FROM READ_FILES('/Volumes/my_catalog/my_schema/my_documents', format => 'binaryFile');

See the ai_parse_document SQL reference.

Parse documents with the REST API

Use the REST API to parse documents from applications, services, and automation workflows. See the REST API docs for reference.

ai_parse_document REST API requests are limited to 100 pages per document.

The default rate limit for ai_parse_document REST API requests is 250 requests per minute per workspace. Contact your Databricks account team to request a higher limit.

ai_parse_document reads its input from a Unity Catalog volume. The following example uploads a local PDF to a volume with the Files API, then parses it.

First, upload the document to a volume:

curl -X PUT "$DATABRICKS_HOST/api/2.0/fs/files/Volumes/my_catalog/my_schema/my_volume/invoice.pdf?overwrite=true" \
     -H "Authorization: Bearer $DATABRICKS_TOKEN" \
     -H "Content-Type: application/octet-stream" \
     --data-binary @invoice.pdf

Then parse the uploaded document by passing its volume path as content:

curl -X POST "$DATABRICKS_HOST/api/2.0/ai-functions/ai-parse-document" \
     -H "Authorization: Bearer $DATABRICKS_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
  "content": "/Volumes/my_catalog/my_schema/my_volume/invoice.pdf"
}'

Parse documents with the Asynchronous API (Beta)

Use the AI Functions Asynchronous API (Beta) to parse many documents in one batch. For the request and response fields, and for the requests to cancel and list batches, see the Asynchronous API reference. A workspace admin must turn on the AI Functions Asynchronous API preview on the Previews page.

To parse a directory of documents with the Asynchronous API:

1. Upload the documents to a directory in a Unity Catalog volume. The batch parses every file in the directory and in its subdirectories, so put only the documents to parse there.

2. Create a batch that runs ai_parse_document on the directory. Set input_ref to the directory and output_ref to the volume directory for the results:

   curl -X POST "$DATABRICKS_HOST/api/2.0/ai-functions/batches" \
        -H "Authorization: Bearer $DATABRICKS_TOKEN" \
        -H "Content-Type: application/json" \
        -d '{
     "endpoint": "ai_parse_document",
     "input_ref": "/Volumes/my_catalog/my_schema/my_volume/invoices/",
     "output_ref": "/Volumes/my_catalog/my_schema/my_volume/output/"
   }'
   

   The response contains the batch name. Use it in place of <batch-name> in the next steps.

3. Check the status of the batch. The batch is finished when status is COMPLETED, CANCELLED, or FAILED:

   curl -X GET "$DATABRICKS_HOST/api/2.0/ai-functions/batches/<batch-name>" \
        -H "Authorization: Bearer $DATABRICKS_TOKEN"
   

   When the output file is ready, output_ref in the response contains its path.

4. Download the output file with the Files API:

   curl -X GET "$DATABRICKS_HOST/api/2.0/fs/files/Volumes/my_catalog/my_schema/my_volume/output/<batch-name>_output.jsonl" \
        -H "Authorization: Bearer $DATABRICKS_TOKEN" \
        -o output.jsonl
   

The output file has one line for each document. Each line has the custom_id of its request and one of the following fields:

• response: The result of the request. It has the same fields as the AI Parse Document REST API response.
• error: The type and message of the error, if the request failed.

For a directory, the custom_id of each line is the path of the document, and the lines can be in any order:

{"custom_id": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-001.pdf", "response": {...}}
{"custom_id": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-002.pdf", "error": {"type": "request_failed", "message": "..."}}

To parse only some of the documents, or to give each document your own custom_id, use a JSONL input file instead of a directory. Each line names one document in body.path:

{"custom_id": "invoice-001", "body": {"path": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-001.pdf"}}
{"custom_id": "invoice-002", "body": {"path": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-002.pdf"}}

Upload the input file to a volume, and set input_ref to its path. The output lines are then in the same order as the input file.

Limitations

See ai_parse_document limitations.