Snowflake Connector¶
The Snowflake Connector enables seamless integration with Snowflake, a high-performance, cloud-based data warehousing platform renowned for its scalability, flexibility, and advanced analytics capabilities. You can learn more about Snowflake on its official documentation..
Sidra’s Snowflake connector allows you to configure a Data Intake Process (DIP) from a Snowflake database in a fast and easy way. The resulting configured Data Intake Process extracts data from any table and view in the source database and loads it into the specified Data Storage Unit at regular intervals. It relies on the Sidra Metadata model for mapping source data structures to Sidra as destination, and uses Azure Data Factory as the underlying data integration mechanism within Sidra.
When you configure and execute a DIP with this connector, Sidra will automatically:
- Create and populate the necessary metadata and data governance structures, including the Data Intake Process, Provider, Entities, Attributes, and required relationships.
- Generate, configure, and deploy the data integration infrastructure (ADF Pipeline).
You can configure the DIP in less than five minutes. Once you set up the parameters and start the deployment, the data ingestion duration will vary from a few minutes to several hours, depending on the data volume.
After you start the DIP, Sidra will notify you that the process has started and will continue running in the background. You can continue working within Sidra Web as usual while this process completes.
Once the deployment is finished, you will receive a notification in the Sidra Web Notifications widget. If the process is successful, the new data structures (Provider, Entities and Attributes) will automatically appear in the Data Catalog, and the DIP will show up in the Data Intake section, including this new data source.
Supported Snowflake versions¶
Snowflake operates on a continuous deployment model, meaning there are no traditional version numbers for its core platform. Instead, Snowflake automatically applies weekly updates, ensuring users always have access to the latest features and improvements without downtime. Sidra is designed to adapt to this model, maintaining compatibility with Snowflake’s latest updates and ensuring seamless integration as Snowflake evolves.
Pre-requisites¶
Before configuring a Data Intake Process (DIP) for Snowflake, you need to enable Snowflake to access Sidra DSU Stage Storage. Make sure that Snowflake supports the Azure region where your Sidra DSU Stage Storage is hosted. You can find more information on Supported Cloud Regions.
To ensure a successful integration, complete the following steps:
- Create a Snowflake Storage Integration: This object delegates authentication responsibility for accessing Sidra DSU Stage Storage Account to an Azure service principal.
- Grant the service principal access to Sidra DSU Stage Storage: Assign the 'Storage Blob Data Contributor' role to the service principal, allowing Snowflake to write into the Storage Account.
- Configure network rules for Snowflake: Add the subnets provided by Snowflake to the Sidra DSU Stage Storage.
For detailed configuration steps, visit the How to configure Snowflake to access Sidra DSU Stage Storage section in our documentation.
Supported Snowflake synchronization mechanisms¶
The Snowflake Connector for Sidra supports different data synchronization modes, allowing you to adapt to various needs and configurations in the source system or Sidra:
- Full load data synchronization:
Ideal for initial loads. This is also the default mode if no incremental load mechanism is configured. The first load will be a complete load by default.
- Incremental load data synchronization:
This method captures updates for any new or modified data from the Snowflake database, retrieving only the updates since the last synchronization. For incremental load to work, you need a defined mechanism to capture updates in the source system. This is detailed in depth in the Incremental Loads section of the documentation.
You can learn more about Sidra Data Intake Processes and how plugin works in the documentation.