Code examples: Apache Spark™¶
This section provides code examples for using Apache Spark™ to do the following in Snowflake Open Catalog:
Configure a service connection
Use catalog
List catalogs
List namespaces
Create a namespace
Use a namespace
Drop a namespace
Create a table
Query a table
Show table properties
List tables
Drop a table
Required privileges¶
To perform the commands included in the code examples, the following privileges need to be bestowed to the service principal you use to connect Spark to Open Catalog.
Command |
Required Privilege |
---|---|
Show Namespaces |
NAMESPACE_LIST |
Create namespace |
NAMESPACE_CREATE |
Use namespace |
NAMESPACE_READ_PROPERTIES |
Show tables |
TABLE_LIST |
Create or replace table |
|
Drop namespace |
NAMESPACE_DROP |
Drop table |
TABLE_DROP |
Insert into table |
TABLE_WRITE_DATA |
Select from table |
TABLE_READ_DATA |
Configure a service connection¶
Use catalog¶
Use the catalog1
catalog.
spark.sql("use catalog1").show()
List catalogs¶
List the catalogs you’re connected to.
spark.sql("show catalogs").show()
List namespaces¶
List the namespaces for the catalog you’re connected to.
spark.sql("show namespaces").show()
Create a namespace¶
Create the namespace1
namespace.
spark.sql("CREATE NAMESPACE namespace1")
Use a namespace¶
Use the namespace1
namespace.
spark.sql("use namespace1").show()
Drop a namespace¶
Drop the namespace1
namespace from the catalog.
spark.sql("DROP NAMESPACE namespace1")
Create a table¶
Create customers
table under the parent namespace namespace1
.
spark.sql ("use namespace1");
spark.sql("CREATE OR REPLACE TABLE customers (id int, custnum int) using iceberg")
Query a table¶
Query the customers
table.
spark.sql ("use namespace1");
spark.sql("SELECT * FROM customers").show()
Show table properties¶
Show the table properties for the customers
table.
spark.sql("SHOW TBLPROPERTIES customers").show(50, False)
List tables¶
List the tables for the catalog you’re connected to.
spark.sql("show tables").show()
Drop a table¶
Drop the customers
table under parent namespace namespace1
.
spark.sql ("use namespace1");
spark.sql("DROP TABLE customers")