Snowpark Container Services: Additional considerations for services¶

Connecting to Snowflake¶

Snowflake provides the following environment variables for you to configure a Snowflake client in your service code:

• SNOWFLAKE_ACCOUNT: Provides the name of the Snowflake account the service is currently running under.

• SNOWFLAKE_HOST: Provides the hostname used to connect to Snowflake.

Snowflake also provides an OAuth token in the container in a file named /snowflake/session/token. When creating a connection, you provide this token to the connector.

When creating a connection to Snowflake from a container, you must use SNOWFLAKE_HOST, SNOWFLAKE_ACCOUNT, and the OAuth token. You cannot use the OAuth token without also using SNOWFLAKE_HOST, and you cannot use the OAuth token outside Snowpark Container Services. For more information, see Using an OAuth token to execute SQL.

For sample code using various Snowflake drivers, see Snowflake Connection Samples.

Examples

• In Tutorial 2 (see main.py), the code reads the environment variables as follows:

  SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT')
  SNOWFLAKE_HOST = os.getenv('SNOWFLAKE_HOST')
  

  Copy

  The code passes these variables to a connection creation code for the Snowflake client of choice. The container uses these credentials to create a new session, with the service role as the primary role, to execute queries. The following example code is the minimum needed to create a Snowflake connection in Python:

  def get_login_token():
    with open('/snowflake/session/token', 'r') as f:
      return f.read()
  
  conn = snowflake.connector.connect(
    host = os.getenv('SNOWFLAKE_HOST'),
    account = os.getenv('SNOWFLAKE_ACCOUNT'),
    token = get_login_token(),
    authenticator = 'oauth'
  )
  

  Copy

• When you use the default host (that is, you don’t include the host argument when creating a connection), connecting to Snowflake using other forms of authentication (such as username and password) is supported. For example, the following connection specifies the user name and password to authenticate:

  conn = snowflake.connector.connect(
    account = os.getenv('SNOWFLAKE_ACCOUNT'),
    user = '<user-name>',
    password = <password>
  )
  

  Copy

  Using a default host requires external access integration with a network rule allowing access from your service to the Snowflake hostname for your account. For example, if your account name is MyAccount, the hostname will be myaccount.snowflakecomputing.com. For more information, see Network egress.

  • Create a network rule that matches your account’s Snowflake API hostname:

    CREATE OR REPLACE NETWORK RULE snowflake_egress_access
      MODE = EGRESS
      TYPE = HOST_PORT
      VALUE_LIST = ('myaccount.snowflakecomputing.com');
    

    Copy

  • Create an integration using the preceding network rule:

    CREATE EXTERNAL ACCESS INTEGRATION snowflake_egress_access_integration
      ALLOWED_NETWORK_RULES = (snowflake_egress_access)
      ENABLED = true;
    

    Copy

Configuring the database and schema context for executing SQL¶

This section explains two concepts:

• The logic Snowflake uses to determine the database and schema in which to create your service.

• The method through which Snowflake conveys this information to your containers, thus enabling the container code to execute SQL in the same database and schema context.

Snowflake uses the service name to determine the database and schema in which to create a service:

• Example 1: In the following CREATE SERVICE and EXECUTE JOB SERVICE commands, the service name does not explicitly specify a database and schema name. Snowflake creates the service and the job service in the current database and schema.

  -- Create a service.
  CREATE SERVICE test_service IN COMPUTE POOL ...
  
  -- Execute a job service.
  EXECUTE JOB SERVICE
    IN COMPUTE POOL tutorial_compute_pool
    NAME = example_job_service ...
  

  Copy

• Example 2: In the following CREATE SERVICE and EXECUTE JOB SERVICE commands, the service name includes a database and schema name. Snowflake creates the service and job service in the specified database (test_db) and schema (test_schema), regardless of the current schema.

  -- Create a service.
  CREATE SERVICE test_db.test_schema.test_service IN COMPUTE POOL ...
  
  -- Execute a job service.
  EXECUTE JOB SERVICE
    IN COMPUTE POOL tutorial_compute_pool
    NAME = test_db.test_schema.example_job_service ...
  

  Copy

When Snowflake starts a service, it provides the database and schema information to the running containers using the following environment variables:

• SNOWFLAKE_DATABASE

• SNOWFLAKE_SCHEMA

Your container code can use environment variables in the connection code to determine which database and schema to use, as shown in this example:

conn = snowflake.connector.connect(
  host = os.getenv('SNOWFLAKE_HOST'),
  account = os.getenv('SNOWFLAKE_ACCOUNT'),
  token = get_login_token(),
  authenticator = 'oauth',
  database = os.getenv('SNOWFLAKE_DATABASE'),
  schema = os.getenv('SNOWFLAKE_SCHEMA')
)

Example

In Tutorial 2, you create a Snowflake job service that connects with Snowflake and executes SQL statements. The following steps summarize how the tutorial code uses the environment variables:

1. In the common setup (see the Common Setup section), you create resources, including a database and a schema. You also set the current database and schema for the session:

   USE DATABASE tutorial_db;
   ...
   USE SCHEMA data_schema;
   

   Copy

2. After you create a job service (by running EXECUTE JOB SERVICE), Snowflake starts the container and sets the following environment variables in the container to the current database and schema of the session:

   • SNOWFLAKE_DATABASE is set to “TUTORIAL_DB”

   • SNOWFLAKE_SCHEMA is set to “DATA_SCHEMA”

3. The job code (see main.py in Tutorial 2) reads these environment variables:

   SNOWFLAKE_DATABASE = os.getenv('SNOWFLAKE_DATABASE')
   SNOWFLAKE_SCHEMA = os.getenv('SNOWFLAKE_SCHEMA')
   

   Copy

4. The job code sets the database and schema as the context in which to execute the SQL statements (run_job() function in main.py):

   {
      "account": SNOWFLAKE_ACCOUNT,
      "host": SNOWFLAKE_HOST,
      "authenticator": "oauth",
      "token": get_login_token(),
      "warehouse": SNOWFLAKE_WAREHOUSE,
      "database": SNOWFLAKE_DATABASE,
      "schema": SNOWFLAKE_SCHEMA
   }
   ...
   

   Copy

   Note

   SNOWFLAKE_ACCOUNT, SNOWFLAKE_HOST, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA are environment variables that Snowflake generates for the application container, but SNOWFLAKE_WAREHOUSE is not (the Tutorial 2 application code created this variable because Snowflake does not pass a warehouse name to a container).

Specifying the warehouse for your container¶

If your service connects to Snowflake to execute a query in a Snowflake warehouse, you have the following options to specify a warehouse:

• Specify a warehouse in your application code. Specify a warehouse as part of the connection configuration when starting a Snowflake session to run queries in your code. For an example, see Tutorial 2.

• Specify a default warehouse when creating a service. Specify the optional QUERY_WAREHOUSE parameter in the CREATE SERVICE (or EXECUTE JOB SERVICE) command to provide a default warehouse. If your application code does not provide a warehouse as part of connection configuration, Snowflake uses the default warehouse. Use the ALTER SERVICE command to change the default warehouse.

If you specify a warehouse using both methods, the warehouse specified in the application code is used.

Using an OAuth token to execute SQL¶

All Snowflake-provided clients support OAuth as a way to authenticate. Service containers also use the OAuth mechanism to authenticate with Snowflake. For example, when a container wants to execute SQL, the container creates a connection to Snowflake, similar to any other Snowflake client:

def get_login_token():
  with open('/snowflake/session/token', 'r') as f:
    return f.read()

conn = snowflake.connector.connect(
  host = os.getenv('SNOWFLAKE_HOST'),
  account = os.getenv('SNOWFLAKE_ACCOUNT'),
  token = get_login_token(),
  authenticator = 'oauth'
)

When you create a service, Snowflake runs the containers and provides an Oauth token for containers to use at the following location within the container: /snowflake/session/token.

Note the following details about this OAuth token:

• You should read the contents of the /snowflake/session/token file immediately before use because the content expires in 10 minutes, and Snowflake refreshes this file every few minutes. After a container connects to Snowflake successfully, the expiration time does not apply to the connection (as with any sessions that users create directly).

• This OAuth token is valid only within the specific Snowflake service. You cannot copy the OAuth token and use it outside the service.

• Using the OAuth token, the containers connect with Snowflake as a user and service role as follows:

  • The user is the person who created the service.

  • The role is the service role for the service, and this is the only role supported. You cannot change the role.

• Using the OAuth token to connect will create a new session. The OAuth token is not associated with any existing SQL session.

  Note

  A significant difference between executing stored procedures and executing a service is that stored procedures run in the same session as the SQL that runs them. But every time a container establishes a new connection, you are creating a new session.

Network ingress¶

To allow anything to interact with your service from the internet, you declare the network ports on which your service is listening as endpoints in the service specification file. These endpoints control ingress.

By default, service endpoints are private. Only service functions and service-to-service communications can make requests to the private endpoints. You can declare an endpoint as public to allow requests to an endpoint from the internet. The role that creates the service (with a public endpoint) must have the BIND SERVICE ENDPOINT privilege on the account.

endpoints:
- name: <endpoint name>
  port: <port number>
  protocol : < TCP / HTTP / HTTPS >
  public: true

For an example, see Tutorial 1.

Ingress and web app security¶

You can create a Snowpark Container Services service for web hosting using the public endpoint support (network ingress). For added security, Snowflake employs a proxy service to monitor incoming requests from clients to your service and outgoing responses from your service to the clients. This section explains what the proxy does and how it impacts a service deployed to Snowpark Container Services.

For example:

• The proxy does not forward an incoming HTTP request if the request uses a banned HTTP method.

• The proxy sends a 403 response to the client if the Content-Type header in the response indicates that the response contains an executable.

Additionally, the proxy can also inject new headers and alter existing headers in the request and the response, with your container and data security in mind.

For example, upon receiving a request, your service might send HTML, JavaScript, CSS, and other content for a web page to the client browser in the response. The web page in the browser is part of your service, acting as the user interface. For security reasons, if your service has restrictions (such as a restriction on making network connections to other sites), you might also want the web page for your service to have the same restrictions.

By default, services don’t have permission to access the internet. The browser should also restrict the client app from accessing the internet and potentially sharing data. If you set up an External Access Integration (EAI) to allow your service to access example.com (see Network egress), the web page for your service should also be able to access example.com through your browser.

The Snowflake proxy applies the same network restrictions on the service and the web page by adding a Content-Security-Policy (CSP) header in the response. By default, the proxy adds a baseline CSP in the response to protect against common security threats. In addition, if your service is configured to use an EAI, the proxy applies the same network rules from the EAI to the CSP for the web page. This CSP enables the web page in the browser to access the same sites that the service can access.

The following sections explain how the Snowflake proxy handles incoming requests for your service and modifies the outgoing responses from your service to the clients.

Network egress¶

Your application code might require access to the internet. By default, application containers don’t have permission to access the internet. You need to enable internet access using External Access Integrations (EAIs).

Typically, you want an account administrator to create EAIs to manage external access allowed from services (including job services). Account administrators can then grant EAI usage to specific roles that developers use to run services.

The following example outlines the steps in creating an EAI that allows egress traffic to specific destinations specified using network rules. You then refer to the EAI when creating a service to allow requests to specific internet destinations.

Example

Suppose you want your application code to send requests to the following destinations:

• HTTPS requests to translation.googleapis.com

• HTTP and HTTPS requests to google.com

Follow these steps to enable your service to access these domains on the internet:

1. Create an External Access Integration (EAI). This requires appropriate permissions. For example, you can use ACCOUNTADMIN role to create an EAI. This is a two-step process:

   1. Use the CREATE NETWORK RULE command to create one or more egress network rules listing external destinations you want to allow access to. You can accomplish this example with one network rule, but for illustration, we create two network rules:

      1. Create a network rule named translate_network_rule:

         CREATE OR REPLACE NETWORK RULE translate_network_rule
           MODE = EGRESS
           TYPE = HOST_PORT
           VALUE_LIST = ('translation.googleapis.com');
         

         Copy

         This rule allows TCP connections to the translation.googleapis.com destination. The domain in the VALUE_LIST property does not specify the optional port number, so the default port 443 (HTTPS) is assumed. This allows your application to connect to any URL that starts with https://translation.googleapis.com/.

      2. Create a network rule named google_network_rule:

         CREATE OR REPLACE NETWORK RULE google_network_rule
           MODE = EGRESS
           TYPE = HOST_PORT
           VALUE_LIST = ('google.com:80', 'google.com:443');
         

         Copy

         This allows your application to connect to any URL that starts with http://google.com/ or https://google.com/.

      Note

      For the VALUE_LIST parameter, you must provide a full host name. Wildcards (for example, *.googleapis.com) are not supported.

      Snowpark Container Services supports only the network rules that allow ports 22, 80, 443, and 1024+. If a network rule referenced allows access to other ports, creation of the service will fail. Contact your account representative if you require use of additional ports.

      Note

      To allow your service to send HTTP or HTTPS requests to any destination on the internet, you specify “0.0.0.0” as the domain in the VALUE_LIST property. The following network rule allows sending both “HTTP” and “HTTPS” requests anywhere on the internet. Only ports 80 or 443 are supported with “0.0.0.0”.

      CREATE NETWORK RULE allow_all_rule
        TYPE = 'HOST_PORT'
        MODE= 'EGRESS'
        VALUE_LIST = ('0.0.0.0:443','0.0.0.0:80');
      

      Copy

   2. Create an external access integration (EAI) that specifies that the preceding two egress network rules are allowed:

      CREATE EXTERNAL ACCESS INTEGRATION google_apis_access_integration
        ALLOWED_NETWORK_RULES = (translate_network_rule, google_network_rule)
        ENABLED = true;
      

      Copy

      Now the account admin can grant usage of the integration to developers to allow them to run a service that can access specific destinations on the internet.

      GRANT USAGE ON INTEGRATION google_apis_access_integration TO ROLE test_role;
      

      Copy

2. Create the service by providing the EAI as shown in the following examples. The role that is creating the service needs the usage privilege on the EAI and read privilege on the secrets referenced. Note that you cannot use the ACCOUNTADMIN role to create a service.

   • Create a service:

     USE ROLE test_role;
     
     CREATE SERVICE eai_service
       IN COMPUTE POOL MYPOOL
       EXTERNAL_ACCESS_INTEGRATIONS = (GOOGLE_APIS_ACCESS_INTEGRATION)
       FROM SPECIFICATION
       $$
       spec:
         containers:
           - name: main
             image: /db/data_schema/tutorial_repository/my_echo_service_image:tutorial
             env:
               TEST_FILE_STAGE: source_stage/test_file
             args:
               - read_secret.py
         endpoints:
           - name: read
             port: 8080
       $$;
     

     Copy

     This example CREATE SERVICE request uses an inline service specification and specifies the optional EXTERNAL_ACCESS_INTEGRATIONS property to include the EAI. The EAI specifies the network rules that allow egress traffic from the service to the specific destinations.

   • Execute a job service:

     EXECUTE JOB SERVICE
       IN COMPUTE POOL tt_cp
       NAME = example_job_service
       EXTERNAL_ACCESS_INTEGRATIONS = (GOOGLE_APIS_ACCESS_INTEGRATION)
       FROM SPECIFICATION $$
       spec:
         container:
         - name: curl
           image: /tutorial_db/data_schema/tutorial_repo/alpine-curl:latest
           command:
           - "curl"
           - "http://google.com/"
       $$;
     

     Copy

     This example EXECUTE JOB SERVICE command specifies inline specification and the optional EXTERNAL_ACCESS_INTEGRATIONS property to include the EAI. This allows egress traffic from the job to destinations specified in the network rules the EAI allows.

Network communications between containers¶

There are two considerations:

• Communications between containers of a service instance: If a service instance runs multiple containers, these containers can communicate with each other over localhost (there is no need to define endpoints in the service specification).

• Communication between containers across multiple services or multiple service instances: Containers belonging to different services (or different instances of the same service) can communicate using endpoints defined in specification files. For more information, see Service-to-service communications.

Passing secrets as environment variables¶

To pass containers Snowflake secrets as environment variables, include a containers.secrets field:

containers:
- name: main
  image: <url>
  secrets:
  - snowflakeSecret: <snowflake-secret-name>
    secretKeyRef: <secret-key-reference>
    envVarName: <env-variable-name>

secretKeyRef: This value depends on the type of Snowflake secret. Possible values:

• username or password if the Snowflake secret is of the password type.

• secret_string if the Snowflake secret is of the generic_string type.

• access_token if the Snowflake secret is of the oauth2 type.

CREATE SECRET testdb.testschema.my_secret_object
  TYPE = password
  USERNAME = 'snowman'
  PASSWORD = '1234abc';

containers:
- name: main
  image: <url>
  secrets:
  - snowflakeSecret: testdb.testschema.my_secret_object
    secretKeyRef: username
    envVarName: LOGIN_USER
  - snowflakeSecret: testdb.testschema.my_secret_object
    secretKeyRef: password
    envVarName: LOGIN_PASSWORD

CREATE SECRET testdb.testschema.my_secret
  TYPE=generic_string
  SECRET_STRING='
       some_magic: config
  ';

containers:
- name: main
  image: <url>
  secrets:
  - snowflakeSecret: testdb.testschema.my_secret
    secretKeyRef: secret_string
    envVarName: GENERIC_SECRET

Passing secrets in local container files¶

To make Snowflake secrets available to your application container in local container files, include a containers.secrets field:

containers:
- name: <name>
  image: <url>
  ...
  secrets:
  - snowflakeSecret: <snowflake-secret-name>
    directoryPath: '<local directory path in the container>'

directoryPath: Snowflake populates one file for each secret key in this specified directory; specifying the secretKeyRef is not necessary.

CREATE SECRET testdb.testschema.my_secret_object
  TYPE = password
  USERNAME = 'snowman'
  PASSWORD = '1234abc';

containers:
- name: main
  image: <url>
  secrets:
  - snowflakeSecret: testdb.testschema.my_secret_object
    directoryPath: '/usr/local/creds'

CREATE SECRET testdb.testschema.oauth_secret
  TYPE = OAUTH2
  OAUTH_REFRESH_TOKEN = '34n;vods4nQsdg09wee4qnfvadH'
  OAUTH_REFRESH_TOKEN_EXPIRY_TIME = '2023-12-31 20:00:00'
  API_AUTHENTICATION = my_integration;

containers:
- name: main
  image: <url>
  secrets:
  - snowflakeSecret: testdb.testschema.oauth_secret
    directoryPath: '/usr/local/creds'