Read documents

A prospectus, a fee schedule or a custodian statement in PDF is not rows and columns, so the parse steps in Read a CSV or Excel file cannot read it. SPARK reads documents in two ways inside a workflow:

• Parse File AI, an action that reads the whole document with an AI model and hands the next step a table of its entities or its tables.
• An Ask Agent step whose agent reads the document with SPARK's document tools, including Extract with source locations, which keeps every value with its place on the page and a confidence, and can go on to read the document's entities.

Both read long documents in windows rather than in one pass, so a document of hundreds of pages is read whole rather than cut off part way.

Parse File AI

Parse File AI sits in the Data group of the action list, with the summary Extract a table from a file with AI. It takes a File and gives a Table.

1. Add it directly after a File Received trigger or a Load File action. It reads only the file one of those hands it (or a PDF made by a Canvas to PDF step, see Export a canvas to PDF), and fails with a message naming them if it is placed after anything else.
2. In the side panel, pick the Action Model:
   • Extract Entities, the default, reads the entities the document names: funds, sub-funds, share classes, companies and their attributes.
   • Extract Tables reads the document's tables.
3. Optionally write a User Instruction to steer the reading, for example which entities matter or how amounts should be normalized.

The step outputs the first table the reading returns, as rows. Column names arrive in title case, with underscores turned into spaces, so a field read as share_class becomes Share Class. Line them up with your attributes using Map Data Fields before a Save Record step if they differ.

Long documents

A PDF is read in page windows: SPARK maps the document, cuts it into windows that end at a table's edge, reads several windows at once and joins the results back into one answer. A table that runs over a window's edge is stitched back together, and an entity read in two windows is merged.

• Documents of up to 2,000 pages and 100 MB are read. A larger one is refused before any reading starts, with a message giving its page count.
• A long read can take several minutes. The step keeps going until the whole document is read, for up to 30 minutes, and fails with The document was not read within 30 minutes if it runs out of time.
• If the reading service is busy with other long documents, the step waits and asks again rather than failing.

Read a document with an agent

An Ask Agent step can read a document the same three ways an agent does in a thread. In a thread, SPARK asks the person which way to read before a whole-document read starts. A workflow run has nobody to ask, so the method goes in the step's instruction, and the agent uses the one it names.

To set it up:

1. Add a File Received trigger, filtered to the documents you expect, for example the pdf extension. See File-triggered workflows.
2. Add an Ask Agent step after it and pick an agent that has SPARK's file tools.
3. In the User instruction, name the method and what to do with the result, for example: Extract with source locations, with entities, into the Prospectus extractions object.

See Ask an agent for the rest of the step's settings.

Extract with source locations

This method turns the document into rows, one per table cell and one per block of text, each carrying the page it is on, its place on that page, the section it sits under and, for a table cell, its column header. Nothing in these rows is written by a model: they are what the file says, where it says it.

• PDFs and images (png, jpg, jpeg, webp) are read with OCR. Spreadsheets and CSV files are read directly, cell by cell, and each sheet's first non-empty row names its fields.
• Every value read with OCR carries a confidence from 0 to 1. Values under 0.8 are flagged so a reviewer checks them first.
• Up to 1,000 pages of a document are extracted. The agent reads 150 pages a call, so a long document takes several calls in one step.
• Each extraction is saved to a new data object named after the method, the time it ran and the file, such as Extract-OCR-Source-202610122305 (Prospectus.pdf), filed in the group Extract (OCR). Name an existing extraction object in the instruction to add to it instead. Extracting the same pages of a file again archives the earlier rows first.

Entities

Ask for entities, for example with entities in the instruction, and the agent goes on from the extracted rows to the document's entities: the umbrella, its sub-funds and share classes, the management company, the directors, and their attributes such as ISINs, fees and addresses. They are saved one value per row in a second data object, Extract-OCR-Entities-… (<file>), and each value points at the source row it was read from.

The reading is built to give the same answer every run:

• The rows are cut into small fixed windows, at most 80 rows, never splitting a table row, each carrying the headings printed above it. One pinned model reads them, many windows at once.
• A value is kept only if it is printed in the row it cites. A value the model cannot point to on the page is dropped.
• Once every page is read, SPARK tidies up: it gives each entity one spelling, fills in a parent the document leaves no doubt about, and adds a name row wherever an entity is named.
• Confidence is worked out, not asked of the model. It is the weaker of how well the OCR read the source row and where the entity's parent came from: 1.0 when the parent is printed with the value, 0.9 when it comes from an earlier heading or a running header, and 0.6 when it is printed nowhere it was read.

Check an extraction in the viewer

An extraction opens in the Document Extraction viewer, which shows the file beside the values read from it.

• From a run: open the run in Executions and click the Ask Agent step. When its agent extracted a document, the result shows Open extraction, which opens the viewer on that file.
• From a thread: an extraction made in a thread is listed in the thread's context under Document Extraction.

The viewer has four tabs:

A value read with low confidence is marked Read with low confidence: check it on the page. For a workbook, the sheet in view is named right after the file, in a picker that lists the sheets values were read from, so you can go straight to a sheet by its name.

When a space shared outside your workspace holds an extraction, its guests open it in the same viewer, with its pages, source rows, entities and meta. See Who can see a Space.

The Extract Unstructured Data action

The action list also offers Extract Unstructured Data. It is not available yet: it is drawn faded, and on hover reads This action is disabled in this workspace. Use Parse File AI or an Ask Agent step for documents today.

Where to go next

• Ask an agent
• Read a CSV or Excel file
• Save records
• File-triggered workflows
• Executions