snowflake.ml.fileset.sfcfs.SFFileSystem¶

class snowflake.ml.fileset.sfcfs.SFFileSystem(*args, **kwargs)¶

Bases: AbstractFileSystem

A filesystem that allows user to access Snowflake stages and stage files with valid Snowflake locations.

The file system is is based on fsspec (https://filesystem-spec.readthedocs.io/). It is a file system wrapper built on top of SFStageFileSystem. It takes Snowflake stage file path as the input and supports read operation. A valid Snowflake location will have the form “@{database_name}.{schema_name}.{stage_name}/{path_to_file}”.

>>> conn = snowflake.connector.connect(**connection_parameters)
>>> sffs = SFFileSystem(sf_connection=conn)
>>> sffs.ls("@MYDB.public.FOO/nytrain")
['@MYDB.public.FOO/nytrain/data_0_0_0.csv', '@MYDB.public.FOO/nytrain/data_0_0_1.csv']
>>> with sffs.open('@MYDB.public.FOO/nytrain/nytrain/data_0_0_1.csv', mode='rb') as f:
>>>     print(f.readline())
b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2

Copy

‘

>>> conn = snowflake.connector.connect(**connection_parameters)
>>> sffs = fsspec.filesystem("sfc", sf_connection=conn)
>>> sffs.ls("@MYDB.public.FOO/nytrain")
['@MYDB.public.FOO/nytrain/data_0_0_0.csv', '@MYDB.public.FOO/nytrain/data_0_0_1.csv']
>>> with sffs.open('@MYDB.public.FOO/nytrain/nytrain/data_0_0_1.csv', mode='rb') as f:
>>>     print(f.readline())
b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2

Copy

‘

>>> conn = snowflake.connector.connect(**connection_parameters)
>>> with fsspec.open("sfc://@MYDB.public.FOO/nytrain/data_0_0_1.csv", mode='rb', sf_connection=conn) as f:
>>>     print(f.readline())
b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2

Copy

‘

Methods

info(path, **kwargs)	Override fsspec info method.
ls(path[, detail])	Override fsspec ls method.
optimize_read([files])	Prefetch and cache the presigned urls for all the given files to speed up the file opening.

Attributes