Skip to main content

Integration Overview

Qarion connects to your existing data infrastructure to synchronize metadata, manage access, and track lineage.

Supported Platforms​

Supported Platforms​

Data Warehouses​

Qarion supports major cloud data warehouses including Snowflake, BigQuery, Databricks, and Redshift. For all these platforms, Qarion provides full metadata synchronization, quality checks, and lineage discovery by parsing SQL history and view definitions.

Databases​

You can connect to standard relational databases such as PostgreSQL, MySQL, and SQL Server. These integrations support metadata synchronization and quality checks but do not currently support automated lineage discovery.

Transformation Tools​

Qarion integrates with transformation engines like dbt and Airflow. The dbt integration synchornizes metadata and lineage from your dbt project, while Airflow integration captures lineage from DAG executions.

Cloud Storage​

For object storage, Qarion connects to AWS S3, Azure Blob Storage, and Google Cloud Storage. These integrations primarily support metadata synchronization to inventory your data files.

Integration Components​

Metadata Sync​

The system automatically imports your database structure, including schemas, tables, views, and columns. It captures technical details such as data types, primary keys, foreign key relationships, and existing descriptions.

Quality Checks​

You can run validations directly against your source data. This includes standard checks for Freshness (when was data last updated) and Volume (row counts), as well as Custom SQL checks for domain-specific validation rules.

Lineage Discovery​

Qarion traces data flow by analyzing various sources: it parses SQL in view definitions and query logs, interprets dbt model dependencies, and extracts metadata from ETL job configurations.

Getting Started​

Quick Setup Process​

To get started, navigate to Source Systems and register a new platform. You will need to configure secure connection details, test the connectivity, and run an initial sync to import your metadata. Finally, map the imported assets to your Data Products to begin governance.

Prerequisites​

Before connecting, ensure network access is configured (firewalls, VPNs), a service account with read permissions is available, and any necessary IP whitelisting or SSL certificates are in place.

Platform-Specific Guides​

Detailed setup instructions are available for specific platforms:

General Connection Settings​

Common Parameters​

Most integrations require standard connection details: the Host address or endpoint, the specific Port for the service, the default Database or catalog name, and optionally a default Schema.

Authentication Methods​

Supported authentication varies by platform. Username/Password is common for standard databases. Key Pair authentication is supported for Snowflake and Databricks. Service Accounts are used for Google Cloud services (BigQuery, GCS), and OAuth or API Tokens are available for platforms like Snowflake, dbt Cloud, and Airflow.

Security Best Practices​

Service Accounts​

Always create dedicated service accounts for Qarion (e.g., qarion_sync_service) rather than using personal credentials. This ensures continuity and allows you to grant only the minimum required permissions.

Credential Storage​

Credentials in Qarion are encrypted at rest. Access is strictly controlled—only administrators can view or edit connection details. You should rotate these credentials periodically according to your security policies.

Network Security​

Secure your connections by using SSL/TLS encryption. Where possible, restrict access to specific IP ranges to further lock down your database endpoints.

Troubleshooting​

Connection Failures​

If a connection fails, first check network connectivity (can you ping the host?). Then verify firewall rules and security groups, ensure credentials are valid, and check that SSL/TLS certificates are correctly configured.

Sync Issues​

Common causes:

  • Permission denied on specific schemas
  • Timeout on large metadata sets
  • Rate limiting from platform

Missing Metadata​

If tables or columns are missing:

  • Verify schema access permissions
  • Check if objects are excluded by filters
  • Trigger a manual resync