Code examples: Apache Spark™¶

This section provides code examples for using Apache Spark™ to do the following in Polaris Catalog™:

Configure a service connection
Use catalog
List catalogs
List namespaces
Create a namespace
Use a namespace
Drop a namespace
Create a table
Query a table
Show table properties
List tables
Drop a table

Required privileges¶

To perform the commands included in the code examples, the following privileges need to be bestowed to the service principal you use to connect Spark to Polaris Catalog.

Command	Required Privilege
Show Namespaces	NAMESPACE_LIST
Create namespace	NAMESPACE_CREATE
Use namespace	NAMESPACE_READ_PROPERTIES
Show tables	TABLE_LIST
Create or replace table	TABLE_WRITE_DATA TABLE_CREATE
Drop namespace	NAMESPACE_DROP
Drop table	TABLE_DROP
Insert into table	TABLE_WRITE_DATA
Select from table	TABLE_READ_DATA

Configure a service connection¶

See examples of configuring a service connection in Spark.

Use catalog¶

Use the catalog1 catalog.

spark.sql("use catalog1").show()

Copy

List catalogs¶

List the catalogs you’re connected to.

spark.sql("show catalogs").show()

Copy

List namespaces¶

List the namespaces for the catalog you’re connected to.

spark.sql("show namespaces").show()

Copy

Create a namespace¶

Create the namespace1 namespace.

spark.sql("CREATE NAMESPACE namespace1")

Copy

Use a namespace¶

Use the namespace1 namespace.

spark.sql("use namespace1").show()

Copy

Drop a namespace¶

Drop the namespace1 namespace from the catalog.

spark.sql("DROP NAMESPACE namespace1")

Copy

Create a table¶

Create customers table under the parent namespace namespace1.

spark.sql ("use namespace1");
spark.sql("CREATE OR REPLACE TABLE customers (id int, custnum int) using iceberg")

Copy

Query a table¶

Query the customers table.

spark.sql ("use namespace1");
spark.sql("SELECT * FROM customers").show()

Copy

Show table properties¶

Show the table properties for the customers table.

spark.sql("SHOW TBLPROPERTIES customers").show(50, False)

Copy

List tables¶

List the tables for the catalog you’re connected to.

spark.sql("show tables").show()

Copy

Drop a table¶

Drop the customers table under parent namespace namespace1.

spark.sql ("use namespace1");
spark.sql("DROP TABLE customers")

Copy