snowflake.ml.fileset.sfcfs.SFFileSystem

class snowflake.ml.fileset.sfcfs.SFFileSystem(*args, **kwargs)

Bases: AbstractFileSystem

A filesystem that allows user to access Snowflake stages and stage files with valid Snowflake locations.

The file system is is based on fsspec (https://filesystem-spec.readthedocs.io/). It is a file system wrapper built on top of SFStageFileSystem. It takes Snowflake stage file path as the input and supports read operation. A valid Snowflake location will have the form “@{database_name}.{schema_name}.{stage_name}/{path_to_file}”.

>>> conn = snowflake.connector.connect(**connection_parameters)
>>> sffs = SFFileSystem(sf_connection=conn)
>>> sffs.ls("@MYDB.public.FOO/nytrain")
['@MYDB.public.FOO/nytrain/data_0_0_0.csv', '@MYDB.public.FOO/nytrain/data_0_0_1.csv']
>>> with sffs.open('@MYDB.public.FOO/nytrain/nytrain/data_0_0_1.csv', mode='rb') as f:
>>>     print(f.readline())
b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2
Copy

‘

>>> conn = snowflake.connector.connect(**connection_parameters)
>>> sffs = fsspec.filesystem("sfc", sf_connection=conn)
>>> sffs.ls("@MYDB.public.FOO/nytrain")
['@MYDB.public.FOO/nytrain/data_0_0_0.csv', '@MYDB.public.FOO/nytrain/data_0_0_1.csv']
>>> with sffs.open('@MYDB.public.FOO/nytrain/nytrain/data_0_0_1.csv', mode='rb') as f:
>>>     print(f.readline())
b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2
Copy

‘

>>> conn = snowflake.connector.connect(**connection_parameters)
>>> with fsspec.open("sfc://@MYDB.public.FOO/nytrain/data_0_0_1.csv", mode='rb', sf_connection=conn) as f:
>>>     print(f.readline())
b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2
Copy

‘

Methods

info(path, **kwargs)

Override fsspec info method.

ls(path[, detail])

Override fsspec ls method.

optimize_read([files])

Prefetch and cache the presigned urls for all the given files to speed up the file opening.

Attributes

async_impl

blocksize

cachable

fsid

Persistent filesystem id that can be used to compare filesystems across sessions.

mirror_sync_methods

protocol

root_marker

sep

transaction

A context within which files are committed together upon exit
Linguagem: Português