snowflake.snowpark.DataFrameReader.documents

DataFrameReader.documents(path: str) → DataFrame[source]

Specify the path of the document(s) to read: the files are parsed to text, or a schema or a prompt turns the read into an extraction of structured values.

The reader chooses the model and how a long document is split across model calls; none of that is configurable. Options set with option() or options() (names are case-insensitive, any other option is ignored):

  • schema: the fields to extract. A JSON Schema dict (with a properties dict), a dict of {field: question}, or a list of field names or "field: question" strings. The result has one column per field, named in upper case.

  • prompt: instructions for the model. With a schema it guides the extraction. Without one the model’s answer is returned as text.

  • parse_mode: how files are parsed when there is no schema or prompt: "layout" (the default), "ocr" or "text".

  • row_boundary: "document" (the default) returns one row per file; "page" returns one row per page. Used with parse_mode only.

  • extract_images: True or False (the default). When True the images embedded in the documents are extracted too. It needs parse_mode "layout" and raises an error with any other mode.

parse_mode, row_boundary and extract_images are ignored when a schema or prompt is set: the file is then sent to the model directly, which keeps layout such as table columns that parsing would flatten.

The returned DataFrame has these columns:

  • no schema or prompt: SOURCE_FILE, TOTAL_PAGES and CONTENT (the parsed text); PAGE_INDEX is added when row_boundary is "page". IMAGES is added when extract_images is True: an array of objects with an id, the bounding-box coordinates of the image and its data as base64 text, so rows of image-heavy documents can be large.

  • schema: SOURCE_FILE, one column per schema field, and EXTRACTION_SCORES. EXTRACTION_SCORES is an object that maps each top-level schema field to a confidence score between 0 and 1 for its extracted value (one score for a whole array or object field). The model reports the scores itself, so they are not calibrated probabilities, and a field without a score is left out.

  • prompt only: SOURCE_FILE and RESPONSE.

Every read also has a case-sensitive "_document_error" column (quote the name when you select it). A document that fails does not fail the read: its row holds an array of {stage, message} entries describing what went wrong, and NULL when the document succeeded. A read stops early with an error when most of its documents fail, or when no supported model is available to the account.

Note

The reader registers temporary procedures and functions in the current session. When documents() is called from inside a stored procedure, that procedure must run with caller’s rights (execute_as="caller"): an owner’s-rights procedure cannot create temporary objects, and the read fails with “Unsupported statement type ‘temporary FUNCTION’”.

Note

The read is lazy: the documents are read (and the model calls are made, and billed) when an action such as collect() or show() runs, and every action runs the whole read again. To use the result more than once, call df.cache_result() or write the DataFrame to a table first.

Note

With a schema, the supported file types are .txt, .md, .pdf, .doc, .docx, .xls, .xlsx, .csv and .xhtml. Other files in the path are not sent to the model; each gets an error in its _document_error. Only PDFs are split into page ranges, so a large file of another type can exceed the model’s input size and fail as a whole. With a prompt only, every file in the path is sent to the model, and a file it cannot read gets an error in its _document_error.

Parameters:

path – The stage location of a document, or a stage location that has documents.

Example:

>>> # Parse every document in a stage to text.
>>> df = session.read.documents("@mystage/docs/")  
>>> df.select("SOURCE_FILE", "TOTAL_PAGES", "CONTENT").show()  

>>> # Extract fields from every document in a stage.
>>> schema = {  
...     "type": "object",
...     "properties": {
...         "invoice_number": {"type": "string"},
...         "total": {"type": "number"},
...     },
... }
>>> df = session.read.option("schema", schema).documents("@mystage/invoices/")  
>>> df.select("SOURCE_FILE", "INVOICE_NUMBER", "TOTAL", '"_document_error"').show()  

>>> # Read one row per page.
>>> df = session.read.option("row_boundary", "page").documents("@mystage/docs/")  

>>> # Also extract the images embedded in the documents.
>>> df = session.read.option("extract_images", True).documents("@mystage/docs/")  
>>> df.select("SOURCE_FILE", "IMAGES").show()

This function or method is in private preview since 1.56.0.