Integration Guide

Overview

DPF stores your transformed data as Apache Iceberg tables and exposes them through a standard Iceberg REST Catalog endpoint. Because Iceberg is an open table format with a standardized catalog API, you can query your data with any compatible engine — without moving it, copying it, or setting up proprietary connectors. The platform is API-first throughout, so the same data and catalog access used here is also what powers our MCP server for AI agents.

This guide covers how to connect analytics engines across AWS, Azure, GCP, Databricks, and Snowflake to your tables using catalog federation, shortcuts, and native catalog integrations. Once configured, your tables are discoverable and queryable through familiar SQL interfaces in each ecosystem.

For everything that speaks JDBC or ODBC — BI tools, SQL IDEs like DBeaver, ORMs, and existing database applications — DPF also provides a PostgreSQL wire-protocol gateway at gateway.dpf-it.com:5432. Clients connect with the stock PostgreSQL driver (no custom driver needed), and your own PostgreSQL database can mount DPF as a linked server via postgres_fdw. See PostgreSQL Gateway below.

Every path below starts at your analytics tools and ends at your Iceberg tables — what changes per platform is the piece in the middle: AWS, Azure, GCP's BigQuery, Databricks, and Snowflake connect with that vendor's own native driver through a federation/shortcut hop; Direct SQL Access uses the generic, unmodified PostgreSQL driver; and Open Engines connect straight to the REST catalog with the open-source Iceberg ecosystem's own client libraries. Pick a tab to see it.

PostgreSQL Gateway (JDBC / ODBC)

DPF operates a PostgreSQL wire-protocol gateway at gateway.dpf-it.com:5432. To any client it looks like a Postgres server, so you connect with the unmodified, official PostgreSQL JDBC or ODBC driver — there is no custom DPF driver to install. This is the integration path for BI tools, SQL IDEs (DBeaver, DataGrip), psql, ORMs, and for mounting DPF as a linked server inside an existing database application.

Every SQL statement you run is forwarded to the DPF query engine and executed directly against your Iceberg tables. Unlike the read-only federation paths above, the gateway supports both reads and writes (SELECT, INSERT, UPDATE, DELETE), with the same per-user authorization and credit metering as the DPF API.

Connection Settings

JDBC URL:

jdbc:postgresql://gateway.dpf-it.com:5432/<namespace>?sslmode=require

psql (or anything else built on libpq):

psql "host=gateway.dpf-it.com port=5432 dbname=<namespace> user=<email> sslmode=require"

Switching Workspaces

Either reconnect with a different database name, or run USE <namespace>; in an open session. Only namespaces your account is authorized for are allowed.

Current Limitations

Connecting DBeaver & BI Tools

Any tool with a PostgreSQL connector works with the gateway. DBeaver is shown step-by-step below; DataGrip, Tableau, and ODBC-based tools follow the same pattern with the connection settings from the previous section.

DBeaver Setup

1. Create a new PostgreSQL connection Database → New Database Connection → PostgreSQL. DBeaver will offer to download the official PostgreSQL JDBC driver automatically — accept it (any recent version works).
2. Fill in the Main tab

   Field	Value
   Host	gateway.dpf-it.com
   Port	5432
   Database	Your namespace (e.g. 633def9656c1)
   Username	Your DPF account email
   Password	Your DPF account password

3. Require SSL On the SSL tab, check Use SSL and set SSL mode to require.
4. Test and connect Click Test Connection, then Finish. Open a SQL editor (SQL Editor → New SQL Script) and query your tables directly:

   SELECT * FROM customers LIMIT 100;
   
   SELECT dpf_filename, COUNT(*) AS row_count
   FROM customers
   GROUP BY dpf_filename
   ORDER BY row_count DESC;

Other Tools

• DataGrip / JetBrains IDEs: create a PostgreSQL data source with the same values and set SSL to require. No driver property overrides are needed.
• Tableau / Power BI / other BI tools: use the generic PostgreSQL connector with the same host, port, database, and credentials, with SSL required. Schema browsing and custom SQL both work.
• ODBC (psqlODBC): use the stock PostgreSQL Unicode ODBC driver with SSLmode=require:

  Driver={PostgreSQL Unicode};Server=gateway.dpf-it.com;Port=5432;Database=<namespace>;Uid=<email>;Pwd=<password>;SSLmode=require;

  Passthrough SQL and the ODBC catalog functions (SQLTables, SQLColumns) both work — schema-browsing applications see your tables and columns. Keep UseDeclareFetch at its default of 0 (server-side cursors are not supported).

Linked Server from PostgreSQL (postgres_fdw)

Because the gateway speaks the Postgres wire protocol, your own PostgreSQL database can mount DPF as a foreign server using the built-in postgres_fdw extension — the Postgres equivalent of a SQL Server linked server. Your DPF tables then appear as foreign tables inside your existing database, queryable and joinable with your local data in plain SQL.

Setup

1. Enable the extension postgres_fdw ships with PostgreSQL — no third-party install needed.

   CREATE EXTENSION IF NOT EXISTS postgres_fdw;

2. Create the foreign server The dbname is your workspace namespace (last 12 characters of your workspaceId).

   CREATE SERVER dpf FOREIGN DATA WRAPPER postgres_fdw
       OPTIONS (host 'gateway.dpf-it.com', port '5432', dbname '633def9656c1');

3. Map your local user to your DPF account

   CREATE USER MAPPING FOR CURRENT_USER SERVER dpf
       OPTIONS (user 'you@example.com', password 'your-dpf-password');

4. Define foreign tables Declare the columns to match your DPF table (the schema on the DPF side is always default). You can see each table's columns in the DPF workspace UI.

   CREATE FOREIGN TABLE customers (
       customer_id   text,
       name          text,
       email         text,
       dpf_filename  text,
       dpf_job       text,
       dpf_ts        timestamp
   ) SERVER dpf
     OPTIONS (schema_name 'default', table_name 'customers');

5. Query DPF data from inside your database

   -- Read DPF data like any local table
   SELECT * FROM customers WHERE dpf_ts >= '2026-06-01' LIMIT 100;
   
   -- Join DPF data with your application's local tables
   SELECT
       c.customer_id,
       c.name,
       o.order_id,
       o.order_total
   FROM customers c              -- foreign table (DPF)
   JOIN app.orders o             -- local table
     ON o.customer_id = c.customer_id
   WHERE o.order_total > 1000;

Writing Through the Link

Foreign tables are writable. INSERTs and fully pushed-down (“direct modify”) UPDATE/DELETE statements are forwarded to the DPF engine and committed per statement:

-- Insert into a DPF table from local data
INSERT INTO customers (customer_id, name, email)
SELECT id, full_name, email FROM app.new_signups;

-- Direct-modify update (whole predicate pushed down)
UPDATE customers SET email = lower(email) WHERE email <> lower(email);

-- Direct-modify delete
DELETE FROM customers WHERE dpf_filename = 'bad_batch.csv';

Setting Up the Glue Data Catalog Integration

AWS Glue Data Catalog supports catalog federation for remote Iceberg REST catalogs. This feature connects Glue to the DPF catalog endpoint, synchronizing metadata at query time so that AWS analytics engines can discover your tables without any data movement.

Once configured, Athena and Redshift see your DPF tables as if they were native Glue tables — with Lake Formation providing fine-grained access control on top.

Prerequisites

• An active DPF workspace with at least one completed data load job
• Your DPF REST Catalog endpoint URL: https://api.dpf-it.com/iceberg/v1
• OAuth2 client credentials for catalog authentication (see Step 1 below to generate)
• AWS account with IAM permissions for Glue, Lake Formation, Athena, and Secrets Manager
• An IAM role with read access to the S3 location where DPF stores your data files

Step-by-Step Setup

1. Generate OAuth2 credentials from the DPF API Use the DPF Workspaces API to create a client ID and secret for your workspace. This is a one-time operation — save the clientSecret immediately, as it cannot be retrieved again.

   # Generate catalog credentials (one-time)
   curl -X POST https://api.dpf-it.com/workspaces \
     -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "action": "create-catalog-credential",
       "workspaceId": "YOUR_WORKSPACE_ID",
       "label": "Glue Data Catalog Federation"
     }'
   
   # Response contains clientId and clientSecret (save immediately!)
   # {
   #   "success": true,
   #   "data": {
   #     "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   #     "clientSecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...",
   #     "message": "Save the clientSecret now — it cannot be retrieved again."
   #   }
   # }

2. Store your OAuth2 credentials in Secrets Manager Store the client secret from Step 1 in AWS Secrets Manager using the key name USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET. Glue uses this key name to retrieve the secret during OAuth2 authentication.

   aws secretsmanager create-secret \
     --name "dpf/catalog-credentials" \
     --secret-string '{"USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET":"YOUR_CLIENT_SECRET"}' \
     --region us-east-1

3. Create a federated catalog in Glue First, create an IAM role that Glue can assume to access Secrets Manager and your Iceberg data. Then create the connection and federated catalog.

   # 1. Create IAM role for Glue federation
   aws iam create-role \
     --role-name DPFGlueFederationRole \
     --assume-role-policy-document '{
       "Version": "2012-10-17",
       "Statement": [{
         "Effect": "Allow",
         "Principal": {"Service": "glue.amazonaws.com"},
         "Action": "sts:AssumeRole"
       }]
     }'
   
   # 2. Attach policy granting access to Secrets Manager and S3 data files
   aws iam put-role-policy \
     --role-name DPFGlueFederationRole \
     --policy-name dpf-federation-access \
     --policy-document '{
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Action": [
             "secretsmanager:GetSecretValue",
             "secretsmanager:DescribeSecret",
             "secretsmanager:PutSecretValue"
           ],
           "Resource": ["arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:dpf/catalog-credentials*"]
         },
         {
           "Effect": "Allow",
           "Action": ["s3:GetObject", "s3:ListBucket"],
           "Resource": [
             "arn:aws:s3:::dpf-storage",
             "arn:aws:s3:::dpf-storage/*"
           ]
         },
         {
           "Effect": "Allow",
           "Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables"],
           "Resource": ["*"]
         }
       ]
     }'
   
   # 3. Create the Glue connection (wait ~10s for IAM propagation)
   aws glue create-connection \
     --connection-input '{
       "Name": "dpf-catalog-connection",
       "ConnectionType": "ICEBERGRESTCATALOG",
       "ConnectionProperties": {
         "INSTANCE_URL": "https://api.dpf-it.com/iceberg/v1",
         "ROLE_ARN": "arn:aws:iam::ACCOUNT:role/DPFGlueFederationRole"
       },
       "AuthenticationConfiguration": {
         "AuthenticationType": "OAUTH2",
         "OAuth2Properties": {
           "OAuth2GrantType": "CLIENT_CREDENTIALS",
           "TokenUrl": "https://api.dpf-it.com/iceberg/v1/oauth/tokens",
           "OAuth2ClientApplication": {
             "UserManagedClientApplicationClientId": "YOUR_CLIENT_ID"
           }
         },
         "SecretArn": "arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:dpf/catalog-credentials"
       }
     }' \
     --region us-east-1
   
   # 4. Create the federated catalog
   aws glue create-catalog \
     --name "dpf-data" \
     --catalog-input '{
       "FederatedCatalog": {
         "ConnectionName": "dpf-catalog-connection",
         "Identifier": "dpf"
       },
       "CreateDatabaseDefaultPermissions": [],
       "CreateTableDefaultPermissions": []
     }' \
     --region us-east-1

4. Grant Lake Formation permissions Grant your analytics IAM role access to the federated catalog so Athena and Redshift can query it.

   aws lakeformation grant-permissions \
     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::ACCOUNT:role/YourAnalyticsRole"}' \
     --resource '{"Catalog":{"Id":"dpf-data"}}' \
     --permissions "ALL" \
     --region us-east-1

5. Verify table discovery List the databases (namespaces) visible through the federated catalog. Each DPF workspace appears as a separate database.

   aws glue get-databases \
     --catalog-id "dpf-data" \
     --region us-east-1

Querying with Amazon Athena

Athena provides serverless, pay-per-query SQL access to your DPF tables through the federated catalog. There is no infrastructure to provision — you pay only for the bytes scanned.

Setup

1. Open the Athena console Navigate to the Athena query editor. Ensure you have a workgroup configured with an S3 results location for query output.
2. Select the federated catalog In the query editor, use the catalog/database selector to choose dpf-data and your workspace database. Alternatively, use three-part naming in your SQL.
3. Run your first query Reference the federated catalog, workspace namespace, and table name:

   SELECT *
   FROM "dpf-data".<workspace_namespace>.<table_name>
   LIMIT 100;

Supported Operations

Examples

Basic query with DPF audit fields:

SELECT
    customer_id,
    name,
    email,
    dpf_filename,
    dpf_job,
    dpf_ts
FROM "dpf-data".my_workspace.customers
LIMIT 100;

Time-travel query — view data as it existed at a specific point in time:

SELECT *
FROM "dpf-data".my_workspace.customers
FOR TIMESTAMP AS OF TIMESTAMP '2026-06-10 12:00:00';

Aggregation by source file:

SELECT
    dpf_filename,
    COUNT(*) AS row_count,
    MIN(dpf_ts) AS earliest_load,
    MAX(dpf_ts) AS latest_load
FROM "dpf-data".my_workspace.customers
GROUP BY dpf_filename
ORDER BY row_count DESC;

Query snapshot history:

SELECT *
FROM "dpf-data".my_workspace."customers$snapshots"
ORDER BY committed_at DESC;

Cost Model

• $5 per TB scanned — only charged for data read by your query
• $0 when idle — no charges when not querying
• Use column projections (select only needed columns) to minimize scanned data
• Partitioned tables automatically prune unneeded data files

Querying with Amazon Redshift

Amazon Redshift Serverless provides a managed SQL analytics engine that can query your DPF tables through the federated catalog. Redshift is ideal for sustained analytical workloads, BI tool integration, and scenarios requiring complex joins across multiple tables.

Setup

1. Create a Redshift Serverless workgroup If you don't already have one, create a workgroup and namespace in the Redshift console. Ensure the associated IAM role has Lake Formation permissions on the dpf-data federated catalog.
2. Query using three-part notation Reference the federated catalog directly in your SQL:

   SELECT *
   FROM "dpf-data".<workspace_namespace>.<table_name>
   LIMIT 100;

3. Alternatively, create an external schema For simpler two-part notation, create an external schema pointing to the federated database:

   CREATE EXTERNAL SCHEMA dpf_workspace
   FROM DATA CATALOG
   DATABASE '<workspace_namespace>'
   CATALOG_ID 'dpf-data'
   IAM_ROLE 'arn:aws:iam::ACCOUNT:role/RedshiftLakeFormationRole';

   Then query with two-part notation:

   SELECT * FROM dpf_workspace.customers LIMIT 100;

Supported Operations

Examples

Aggregation by load job:

SELECT
    dpf_job,
    COUNT(*) AS total_rows,
    COUNT(DISTINCT dpf_filename) AS file_count,
    MIN(dpf_ts) AS job_start,
    MAX(dpf_ts) AS job_end
FROM "dpf-data".my_workspace.customers
GROUP BY dpf_job
ORDER BY job_start DESC;

Join across DPF tables:

SELECT
    c.customer_id,
    c.name,
    a.account_id,
    a.balance
FROM "dpf-data".my_workspace.customers c
JOIN "dpf-data".my_workspace.accounts a
  ON c.customer_id = a.customer_id
WHERE a.balance > 10000;

Create a materialized view for frequent queries:

CREATE MATERIALIZED VIEW mv_customer_summary AS
SELECT
    dpf_job,
    COUNT(*) AS total_rows,
    COUNT(DISTINCT dpf_filename) AS file_count
FROM "dpf-data".my_workspace.customers
GROUP BY dpf_job;

Athena vs. Redshift — When to Use Which

Setting Up Microsoft Fabric Shortcuts

Microsoft Fabric connects to external Iceberg catalogs through OneLake Shortcuts. A shortcut creates a virtualized reference to your tables, allowing Fabric workloads (SQL Analytics Endpoint, Lakehouse, Notebooks) to query your data as if it were stored locally in OneLake — without copying or moving anything.

Once a shortcut is configured, your tables appear as native Fabric tables and can be queried with T-SQL from SQL Server, Synapse, or any tool connected to the Fabric SQL Analytics Endpoint.

Prerequisites

• An active DPF workspace with at least one completed data load job
• Your DPF REST Catalog endpoint URL: https://api.dpf-it.com/iceberg/v1
• OAuth2 client credentials for catalog authentication (generate via the DPF Workspaces API)
• A Microsoft Fabric workspace with at least Contributor access
• A Fabric Lakehouse created within the workspace

Step-by-Step Setup

1. Generate OAuth2 credentials from the DPF API If you haven't already created catalog credentials for this DPF workspace, generate them now. These are the same credentials used for the AWS integration.

   curl -X POST https://api.dpf-it.com/workspaces \
     -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "action": "create-catalog-credential",
       "workspaceId": "YOUR_WORKSPACE_ID",
       "label": "Microsoft Fabric Shortcut"
     }'

2. Open your Fabric Lakehouse Navigate to your Microsoft Fabric workspace, then open the Lakehouse where you want to surface your tables. Select the Tables section in the Explorer pane.
3. Create an Iceberg table shortcut Click New shortcut → select Apache Iceberg as the source type. Fill in the connection details:

   Field	Value
   Catalog URL	https://api.dpf-it.com/iceberg/v1
   Authentication	OAuth2 Client Credentials
   Token endpoint	https://api.dpf-it.com/iceberg/v1/oauth/tokens
   Client ID	Your DPF client ID from Step 1
   Client Secret	Your DPF client secret from Step 1
   Namespace	Your workspace namespace (last 12 characters of your workspaceId, e.g. 633def9656c1)

4. Select tables to shortcut After connecting, Fabric displays the available tables in your namespace. Select the tables you want to surface (e.g., customers, accounts) and click Create.
5. Verify in the Lakehouse Explorer Your tables now appear in the Tables section of the Lakehouse with a shortcut icon. They are queryable immediately via the SQL Analytics Endpoint.

Querying with SQL Server

SQL Server 2022 and Azure SQL Managed Instance can query your tables through linked servers pointing to a Fabric SQL Analytics Endpoint or Synapse Serverless SQL pool. This enables T-SQL access to your Iceberg data from within your existing SQL Server databases.

Architecture

SQL Server does not connect directly to Iceberg catalogs. Instead, it queries through an intermediary that natively supports Iceberg — either the Fabric SQL Analytics Endpoint or a Synapse Serverless SQL pool — using a linked server connection.

Option A: Via Fabric SQL Analytics Endpoint

Once you've created Iceberg shortcuts in a Fabric Lakehouse, the Lakehouse's SQL Analytics Endpoint exposes them as T-SQL-queryable tables.

1. Get the SQL Analytics Endpoint connection string In your Fabric Lakehouse, click SQL Analytics Endpoint in the top bar. Copy the server name (e.g., xxxxxxxx.datawarehouse.fabric.microsoft.com).
2. Create a linked server in SQL Server

   -- Create linked server to Fabric SQL Analytics Endpoint
   EXEC sp_addlinkedserver
       @server = N'DPF_FABRIC',
       @srvproduct = N'',
       @provider = N'MSOLEDBSQL',
       @datasrc = N'your-endpoint.datawarehouse.fabric.microsoft.com',
       @catalog = N'your_lakehouse';
   
   -- Configure authentication (Azure AD / Entra ID)
   EXEC sp_addlinkedsrvlogin
       @rmtsrvname = N'DPF_FABRIC',
       @useself = N'FALSE',
       @rmtuser = N'your-azure-ad-user@domain.com',
       @rmtpassword = N'your-password-or-token';

3. Query your tables through the linked server

   -- Four-part naming: LinkedServer.Database.Schema.Table
   SELECT *
   FROM DPF_FABRIC.your_lakehouse.dbo.customers
   WHERE dpf_job = '550e8400-e29b-41d4-a716-446655440005';
   
   -- Aggregation across tables
   SELECT
       dpf_filename,
       COUNT(*) AS row_count,
       MIN(dpf_ts) AS earliest_load,
       MAX(dpf_ts) AS latest_load
   FROM DPF_FABRIC.your_lakehouse.dbo.customers
   GROUP BY dpf_filename;

Option B: Via Synapse Serverless SQL

Alternatively, create a linked server pointing to a Synapse Serverless SQL pool that has access to your tables.

-- Create linked server to Synapse Serverless SQL
EXEC sp_addlinkedserver
    @server = N'DPF_SYNAPSE',
    @srvproduct = N'',
    @provider = N'MSOLEDBSQL',
    @datasrc = N'your-synapse-workspace-ondemand.sql.azuresynapse.net',
    @catalog = N'dpf_external';

-- Query your tables through Synapse
SELECT
    customer_id,
    name,
    email,
    dpf_filename,
    dpf_job
FROM DPF_SYNAPSE.dpf_external.dpf.customers
WHERE dpf_ts >= '2026-06-01';

Cross-Database Joins

A key advantage of the linked server approach is that you can join your Iceberg tables with existing SQL Server data in a single query:

-- Join Iceberg data with local SQL Server tables
SELECT
    c.customer_id,
    c.name,
    c.email,
    o.order_id,
    o.order_total
FROM DPF_FABRIC.your_lakehouse.dbo.customers c
INNER JOIN dbo.orders o
    ON c.customer_id = o.customer_id
WHERE c.dpf_job = '550e8400-e29b-41d4-a716-446655440005'
ORDER BY o.order_total DESC;

Querying with Azure Synapse Analytics

Azure Synapse Serverless SQL pool can query your Iceberg tables via Fabric Lakehouse shortcuts. This provides pay-per-query T-SQL access without provisioning compute resources.

Setup: Via Fabric Lakehouse Shortcuts

Once you've configured Fabric shortcuts, Synapse can query your tables through the Lakehouse's SQL Analytics Endpoint. Fabric handles catalog authentication and metadata resolution automatically.

1. Connect Synapse to the Fabric workspace In Synapse Studio, add a linked service pointing to your Fabric SQL Analytics Endpoint, or query it directly using a Serverless SQL pool with cross-workspace access.
2. Query your tables

   -- Query through Fabric Lakehouse (three-part name)
   SELECT *
   FROM [your_lakehouse].[dbo].[customers]
   LIMIT 100;

Supported Operations

Examples

Aggregation by source file and job:

SELECT
    dpf_job,
    dpf_filename,
    COUNT(*) AS row_count,
    MIN(dpf_ts) AS load_start,
    MAX(dpf_ts) AS load_end
FROM [your_lakehouse].[dbo].[customers]
GROUP BY dpf_job, dpf_filename
ORDER BY load_start DESC;

Data lineage query — trace rows back to source:

SELECT
    customer_id,
    name,
    dpf_filename AS source_file,
    dpf_line AS source_row_number,
    dpf_job AS load_job_id,
    dpf_ts AS loaded_at
FROM [your_lakehouse].[dbo].[customers]
WHERE customer_id = 'CUST-12345'
ORDER BY dpf_ts DESC;

Cross-source join — Iceberg data with Azure SQL tables:

-- Join your shortcut table with a local Synapse table
SELECT
    c.customer_id,
    c.name,
    c.email,
    s.subscription_tier,
    s.renewal_date
FROM [your_lakehouse].[dbo].[customers] c
INNER JOIN dbo.subscriptions s
    ON c.customer_id = s.customer_id
WHERE c.dpf_job = '550e8400-e29b-41d4-a716-446655440005';

SQL Server vs. Synapse — When to Use Which

Setting Up GCP Integration

BigQuery reaches your tables through BigQuery Omni, Google's cross-cloud connector for data sitting in AWS. Because your table data lives in DPF's S3 storage, this requires a one-time cross-account IAM authorization, similar to the Glue integration above.

Prerequisites

• An active DPF workspace with at least one completed data load job
• Your DPF REST Catalog endpoint URL: https://api.dpf-it.com/iceberg/v1
• A BigQuery Omni connection to AWS (aws-us-east-1 region) and an AWS account to authorize for cross-account access

Step-by-Step Setup

1. Create an AWS IAM role for BigQuery Omni Follow Google's BigQuery Omni setup to create the role and trust policy, then create the connection in BigQuery:

   bq mk --connection --connection_type=AWS \
     --properties='{"accessRole":{"iamRoleId":"arn:aws:iam::ACCOUNT:role/BigQueryOmniAccessRole"}}' \
     --location=aws-us-east-1 \
     dpf-omni-connection

2. Create a BigLake external table pointing at the Iceberg metadata BigQuery Omni reads Iceberg tables from a metadata pointer rather than talking to the REST catalog live, so you supply the current metadata_location for the table (visible in the DPF workspace UI or via the REST catalog's LoadTable response):

   bq mk --table \
     --external_table_definition='@iceberg_def.json' \
     my_dataset.customers
   
   # iceberg_def.json
   {
     "icebergOptions": {
       "metadataLocation": "s3://dpf-storage/<workspace_namespace>/customers/metadata/00003-xxxx.metadata.json"
     },
     "connectionId": "projects/YOUR_PROJECT/locations/aws-us-east-1/connections/dpf-omni-connection"
   }

Querying from GCP

Once the BigLake external table is configured, query it like any other BigQuery table:

SELECT
    customer_id,
    name,
    email,
    dpf_filename,
    dpf_job,
    dpf_ts
FROM `my_project.my_dataset.customers`
LIMIT 100;

Setting Up Databricks Unity Catalog Integration

Unity Catalog supports Iceberg REST catalog federation: a foreign catalog object that points at an external Iceberg REST endpoint. Once configured, your DPF workspace appears inside Unity Catalog as a foreign catalog, queryable from Databricks SQL warehouses and notebooks alike — on any cloud your Databricks workspace runs on.

Prerequisites

• An active DPF workspace with at least one completed data load job
• Your DPF REST Catalog endpoint URL: https://api.dpf-it.com/iceberg/v1
• OAuth2 client credentials for catalog authentication (see Step 1 below to generate)
• A Databricks workspace with Unity Catalog enabled
• CREATE CONNECTION and CREATE CATALOG privileges on the metastore

Step-by-Step Setup

1. Generate OAuth2 credentials from the DPF API These are the same credentials used for the other integrations on this page.

   curl -X POST https://api.dpf-it.com/workspaces \
     -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "action": "create-catalog-credential",
       "workspaceId": "YOUR_WORKSPACE_ID",
       "label": "Databricks Unity Catalog"
     }'

2. Create the connection In a Databricks SQL editor or notebook, create a connection describing how to reach the DPF REST catalog:

   CREATE CONNECTION dpf_rest_connection TYPE ICEBERG_REST
   OPTIONS (
     uri 'https://api.dpf-it.com/iceberg/v1',
     token_refresh_url 'https://api.dpf-it.com/iceberg/v1/oauth/tokens',
     client_id 'YOUR_CLIENT_ID',
     client_secret 'YOUR_CLIENT_SECRET'
   );

3. Create the foreign catalog

   CREATE FOREIGN CATALOG dpf_data
   USING CONNECTION dpf_rest_connection
   OPTIONS (catalog 'dpf');

4. Verify table discovery Each DPF workspace namespace appears as a schema under the foreign catalog:

   SHOW SCHEMAS IN dpf_data;
   SHOW TABLES IN dpf_data.<workspace_namespace>;

Querying with Databricks

Once the foreign catalog is set up, query your tables the same way from a Databricks SQL warehouse or a notebook — both share the same Unity Catalog metadata.

Examples

Basic query with DPF audit fields:

SELECT
    customer_id,
    name,
    email,
    dpf_filename,
    dpf_job,
    dpf_ts
FROM dpf_data.my_workspace.customers
LIMIT 100;

Time-travel query:

SELECT *
FROM dpf_data.my_workspace.customers
TIMESTAMP AS OF '2026-06-10 12:00:00';

Join a DPF (foreign) table with a native Delta table:

SELECT
    c.customer_id,
    c.name,
    o.order_id,
    o.order_total
FROM dpf_data.my_workspace.customers c
JOIN main.sales.orders o
  ON o.customer_id = c.customer_id
WHERE o.order_total > 1000;

Supported Operations

Setting Up the Snowflake Catalog Integration

Snowflake reads external Iceberg tables through a catalog integration of type ICEBERG_REST, paired with an external volume that describes how to reach the underlying storage. Once configured, your DPF workspace's tables can be exposed as Snowflake Iceberg tables and queried with standard SQL.

Prerequisites

• An active DPF workspace with at least one completed data load job
• Your DPF REST Catalog endpoint URL: https://api.dpf-it.com/iceberg/v1
• OAuth2 client credentials for catalog authentication (see Step 1 below to generate)
• A Snowflake account on Enterprise edition or higher (required for Iceberg Tables)
• ACCOUNTADMIN, or a role with CREATE INTEGRATION and CREATE EXTERNAL VOLUME privileges

Step-by-Step Setup

1. Generate OAuth2 credentials from the DPF API

   curl -X POST https://api.dpf-it.com/workspaces \
     -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "action": "create-catalog-credential",
       "workspaceId": "YOUR_WORKSPACE_ID",
       "label": "Snowflake Catalog Integration"
     }'

2. Create the catalog integration

   CREATE CATALOG INTEGRATION dpf_catalog_integration
     CATALOG_SOURCE = ICEBERG_REST
     TABLE_FORMAT = ICEBERG
     REST_CONFIG = (
       CATALOG_URI = 'https://api.dpf-it.com/iceberg/v1'
       CATALOG_NAME = 'dpf'
     )
     REST_AUTHENTICATION = (
       TYPE = OAUTH
       OAUTH_TOKEN_URI = 'https://api.dpf-it.com/iceberg/v1/oauth/tokens'
       OAUTH_CLIENT_ID = 'YOUR_CLIENT_ID'
       OAUTH_CLIENT_SECRET = 'YOUR_CLIENT_SECRET'
     )
     ENABLED = TRUE;

3. Create an external volume for vended credentials Omit STORAGE_AWS_ROLE_ARN so Snowflake relies on the catalog integration to vend short-lived storage credentials per query, rather than a static IAM role — no cross-account authorization is required.

   CREATE EXTERNAL VOLUME dpf_ext_volume
     STORAGE_LOCATIONS =
     (
       (
         NAME = 'dpf-vended'
         STORAGE_PROVIDER = 'S3'
         STORAGE_BASE_URL = 's3://dpf-storage/'
       )
     );

4. Create a catalog-linked database for your workspace This auto-populates a Snowflake database from your DPF workspace namespace — no need to declare each table individually.

   CREATE DATABASE dpf_data
     LINKED_CATALOG = (
       CATALOG = 'dpf_catalog_integration',
       CATALOG_NAMESPACE = '<workspace_namespace>'
     )
     EXTERNAL_VOLUME = 'dpf_ext_volume';

5. Verify table discovery

   SHOW ICEBERG TABLES IN DATABASE dpf_data;

Querying with Snowflake

Query the catalog-linked database like any other Snowflake database — your workspace namespace appears as a schema, and each DPF table appears as a Snowflake Iceberg table.

Examples

Basic query with DPF audit fields:

SELECT
    customer_id,
    name,
    email,
    dpf_filename,
    dpf_job,
    dpf_ts
FROM dpf_data.my_workspace.customers
LIMIT 100;

Aggregation by load job:

SELECT
    dpf_job,
    COUNT(*) AS total_rows,
    COUNT(DISTINCT dpf_filename) AS file_count,
    MIN(dpf_ts) AS job_start,
    MAX(dpf_ts) AS job_end
FROM dpf_data.my_workspace.customers
GROUP BY dpf_job
ORDER BY job_start DESC;

Join a DPF Iceberg table with a native Snowflake table:

SELECT
    c.customer_id,
    c.name,
    s.subscription_tier,
    s.renewal_date
FROM dpf_data.my_workspace.customers c
JOIN app_db.public.subscriptions s
  ON s.customer_id = c.customer_id;

Supported Operations

AI Agents (MCP)

DPF publishes a Model Context Protocol (MCP) server so AI agents can discover the DPF API, onboard new users, and run end-to-end data integration workflows — register an account, create a workspace, load a file, build a data spec, and query the results, all through natural-language tool calls. There are two ways to connect it, and which one to use depends entirely on your client.

{
  "mcpServers": {
    "dpf": {
      "url": "https://api.dpf-it.com/mcp"
    }
  }
}