snowflake.ml.fileset.sfcfs.SFFileSystem¶
- class snowflake.ml.fileset.sfcfs.SFFileSystem(*args, **kwargs)¶
Bases:
AbstractFileSystem
A filesystem that allows user to access Snowflake stages and stage files with valid Snowflake locations.
The file system is is based on fsspec (https://filesystem-spec.readthedocs.io/). It is a file system wrapper built on top of SFStageFileSystem. It takes Snowflake stage file path as the input and supports read operation. A valid Snowflake location will have the form “@{database_name}.{schema_name}.{stage_name}/{path_to_file}”.
>>> conn = snowflake.connector.connect(**connection_parameters) >>> sffs = SFFileSystem(sf_connection=conn) >>> sffs.ls("@MYDB.public.FOO/nytrain") ['@MYDB.public.FOO/nytrain/data_0_0_0.csv', '@MYDB.public.FOO/nytrain/data_0_0_1.csv'] >>> with sffs.open('@MYDB.public.FOO/nytrain/nytrain/data_0_0_1.csv', mode='rb') as f: >>> print(f.readline()) b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2
‘
>>> conn = snowflake.connector.connect(**connection_parameters) >>> sffs = fsspec.filesystem("sfc", sf_connection=conn) >>> sffs.ls("@MYDB.public.FOO/nytrain") ['@MYDB.public.FOO/nytrain/data_0_0_0.csv', '@MYDB.public.FOO/nytrain/data_0_0_1.csv'] >>> with sffs.open('@MYDB.public.FOO/nytrain/nytrain/data_0_0_1.csv', mode='rb') as f: >>> print(f.readline()) b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2
‘
>>> conn = snowflake.connector.connect(**connection_parameters) >>> with fsspec.open("sfc://@MYDB.public.FOO/nytrain/data_0_0_1.csv", mode='rb', sf_connection=conn) as f: >>> print(f.readline()) b'2014-02-05 14:35:00.00000054,13,2014-02-05 14:35:00 UTC,-74.00688,40.73049,-74.00563,40.70676,2
‘
Methods
info
(path, **kwargs)Override fsspec info method.
ls
(path[, detail])Override fsspec ls method.
optimize_read
([files])Prefetch and cache the presigned urls for all the given files to speed up the file opening.
Attributes