Azure Private Link and Snowflake Open Catalog¶
This topic describes how to configure Azure Private Link to directly connect your Snowflake Open Catalog account to your query engine by using inbound private connectivity.
Prerequisites¶
Your Snowflake Open Catalog account is hosted on Azure.
You have the necessary permissions to configure your DNS service with the private connectivity URL for your Open Catalog account.
Step 1: Retrieve your Open Catalog account settings¶
Retrieve the following settings for configuring access to Open Catalog with Azure Private Link.
Sign in to Snowflake Open Catalog.
In the navigation menu, select Settings.
On the Settings page, copy the values for the following settings into a text editor:
PrivateLink Account URL
Regionless PrivateLink Account URL
PrivateLink OCSP URL
Regionless PrivateLink OCSP URL
Private Link Service ID
You paste these values when you Configure access to Open Catalog with Azure Private Link and Connect to Open Catalog through Azure Private Link.
For descriptions of each setting, see Return values for the SYSTEM$GET_PRIVATELINK_CONFIG system function in the Snowflake documentation. In this topic, the names of the account settings are in JSON format.
Note
Remember that, where applicable, the description refers to a Snowflake account but your value is actually for your Snowflake Open
Catalog account. For example, the privatelink-account-url
is the URL for your Snowflake Open Catalog account.
Optional: To retrieve these values in JSON format, Create a Snowflake CLI connection for Open Catalog, and then call the SYSTEM$GET_PRIVATELINK_CONFIG system function.
Step 2: Configure access to Open Catalog with Azure Private Link¶
Attention
This section only covers the Open Catalog–specific details for configuring your VNet environment. Also, note that Snowflake is not responsible for the actual configuration of the required firewall updates and DNS records. If you have issues with any of these configuration tasks, contact Microsoft Support directly.
This section describes how to connect your VNet to the Open Catalog VNet using Azure Private Link.
To complete the instructions, you need to use the Azure CLI or Azure PowerShell. For installation help, see see the Microsoft documentation for the Azure CLI or Azure PowerShell.
After initiating the connection to Snowflake Open Catalog using Azure Private Link, you can determine the approval state of the connection in the Azure portal.
Create a private endpoint¶
Note
If you already created a private endpoint for your Snowflake account, and the account is in the same deployment as your Open Catalog account, creating a new private endpoint for your Open Catalog account isn’t required. You can optionally skip this step.
Retrieve your Azure account details. The following example uses the Azure CLI’s
az account list
command.Name CloudName SubscriptionId State IsDefault ------- ---------- ------------------------------------ ------- ---------- MyCloud AzureCloud 13c... Enabled True
In the Azure portal, search for Private Link, and then select Private Link in the results.
Click Private endpoints, and then click Add.
On the Basics tab, complete the Subscription, Resource group, Name, and Region fields for your environment and then click Next: Resource.
On the Resource tab, for Connection Method, select Connect to an Azure resource by resource ID or alias.
For Resource ID or alias, enter the value for
Private Link Service ID
that you obtained when you
retrieved your Open Catalog account settings for private connectivity.If you receive an error message regarding the alias value, contact Snowflake Support for the resource ID value, and then repeat this step using that value.
When the private endpoint is approved, the CONNECTION STATE in the Private endpoints section on the Private Link Center page displays the value Pending. This value changes to Approved when you complete the authorization in the next procedure.
Enable inbound private connectivity¶
In this procedure, you enable Azure Private Link for your Open Catalog account. This configuration allows the query engine to connect to Open Catalog through private connectivity. You will need your private endpoint resource ID, a subscription ID, and the federated token value that contains access credentials for a federated user.
To obtain these values, execute the following commands in the Azure CLI:
To obtain your private endpoint resource ID, execute the following command, and copy the value into a text editor:
az network private-endpoint show
To obtain the subscription ID, execute the following command, and note the value in the SubscriptionID column in the output:
az account list --output table
To obtain the federated token value, execute the following command, and copy the accessToken value into a text editor:
az account get-access-token --subscription <SubscriptionID>
Where:
SubscriptionID
is the unique identifier you obtained in the previous step.
Important
The user generating the Azure access Token must have Read permissions on the Subscription. The least privilege permission is Microsoft.Subscription/subscriptions/acceptOwnershipStatus/read. Alternatively, the default role
Reader
grants more coarse-grained permissions.The
accessToken
value is sensitive information and should be treated like a password value — do not share this value.If it is necessary to contact Snowflake Support, redact the access token from any commands and URLs before creating a support ticket.
Sign in to Snowflake Open Catalog.
In the navigation menu, select Settings.
Select Authorize.
In the Authorize Private Link dialog, enable private connectivity for your account:
For ID, enter the private endpoint resource ID that you copied to a text editor.
For Federated token, enter the federated token value that you copied to a text editor.
Select Save.
Set up DNS¶
All requests to Open Catalog must be routed through the private endpoint. To resolve the Open Catalog account and OCSP URLs to the private IP address of your private endpoint, update your DNS.
To get the endpoint IP address, in the Azure portal search bar, enter the name of the private endpoint you created.
Select the Network Interface result.
Copy the value for the Private IP address.
Configure your DNS to have the appropriate endpoint values from your Open Catalog account settings for private connectivity resolve to the private IP address.
Step 3: Connect to Open Catalog through Azure Private Link¶
To register a service connection and connect your query engine to Open Catalog through Azure Private Link, use the following code:
import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.appName('iceberg_lab') \ .config('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.1,<maven_coordinate>') \ .config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \ .config('spark.sql.defaultCatalog', 'opencatalog') \ .config('spark.sql.catalog.opencatalog', 'org.apache.iceberg.spark.SparkCatalog') \ .config('spark.sql.catalog.opencatalog.type', 'rest') \ .config('spark.sql.catalog.opencatalog.uri','https://<open_catalog_privatelink_account_url>/polaris/api/catalog') \ .config('spark.sql.catalog.opencatalog.header.X-Iceberg-Access-Delegation','vended-credentials') \ .config('spark.sql.catalog.opencatalog.credential','<client_id>:<client_secret>') \ .config('spark.sql.catalog.opencatalog.warehouse','<catalog_name>') \ .config('spark.sql.catalog.opencatalog.scope','PRINCIPAL_ROLE:<principal_role_name>') \ .getOrCreate()
Parameters¶
Note
Ensure that you set up your DNS service to match the value you specify for <open_catalog_account_identifier>
.
Parameter |
Description |
---|---|
|
Specifies the name of the catalog to connect to. |
|
Specifies the Maven coordinate for your external cloud storage provider:
|
|
Specifies the client ID for the service principal to use. |
|
Specifies the client secret for the service principal to use. |
|
Specifies the URL to connect to your Snowflake account using AWS PrivateLink or Azure Private Link.
|
|
Specifies the principal role that is granted to the service principal. |
Step 4 (Optional): Create a catalog integration for Snowflake¶
If you’re using Snowflake to query Open Catalog-managed tables, create a catalog for Snowflake that uses a private IP address. To create this catalog integration, your Snowflake account must be in the same deployment as your Open Catalog account.
For an example, see Example: Catalog integration that uses a private IP address in the Snowflake documentation.