Examples

Six concrete objectives, each worked end-to-end as a sequence of DPF API calls.

Overview

On This Page

Each objective below is self-contained but builds on the ones before it — together they walk a single workspace from account creation through a scheduled ingestion pipeline and into aggregated reporting. Run them in order the first time through:

1. Register & verify OTPget a JWT and create a workspace
↓
2. Load a file into a new tableschema inferred by AI (Flow A)
↓
3. Load & transform into an existing tablefield mapping + business rules via additionalPrompt
↓
4. SFTP connection + daily triggerre-runs the Objective 2 spec automatically (Flow B under the hood)
↓
5. Query the fully loaded dataSQL over your Iceberg tables
↓
6. Merge incremental data into an aggregate tableread changes since the last run (dpf_ts window), MERGE them in via /query's write path

Conventions Used Below

  • Base URL is https://api.dpf-it.com. Every request/response is JSON.
  • Success responses are shaped { "success": true, "data": {...} }; errors are { "error": { "code": "...", "message": "..." } }.
  • YOUR_DPF_JWT_TOKEN is the token from Objective 1's login call. Pass it as Authorization: Bearer YOUR_DPF_JWT_TOKEN on every authenticated request below.
  • YOUR_WORKSPACE_ID is the workspaceId created in Objective 1 — reused by every other objective on this page.
  • Most endpoints (/data-specs, /connections, /job-triggers, /workspaces) are multi-action: one URL, with an action field in the JSON body selecting the operation.
  • Full request/response schemas are in the API Docs; this page is the "how do these calls chain together" companion to that reference.

1Register a User & Verify OTP

Create an account, verify it with the one-time code emailed on registration, log in for a JWT, and create the workspace that every other objective on this page will load data into and query.

  1. Register termsAccepted must be true or registration is rejected. This sends a 6-digit OTP to the given email. 
    curl -X POST https://api.dpf-it.com/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane@example.com",
    "password": "SecurePassword123",
    "firstName": "Jane",
    "lastName": "Doe",
    "termsAccepted": true
  }'

# 201 Created
# { "success": true, "data": { "userId": "...", "email": "jane@example.com", ... } }
  2. Verify the OTP Use the 6-digit code from the verification email. If it expired or never arrived, call POST /auth/resend-otp with just {"email": "..."}. 
    curl -X POST https://api.dpf-it.com/auth/verify-otp \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane@example.com",
    "otp": "123456"
  }'

# 200 OK
# { "success": true, "data": { "message": "Email verified successfully" } }
  3. Log in Returns a 24h-lived access token plus a ~90-day refreshToken (exchange it at POST /auth/refresh once the access token expires). Save the token — every call below sends it as Authorization: Bearer .... 
    curl -X POST https://api.dpf-it.com/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane@example.com",
    "password": "SecurePassword123"
  }'

# 200 OK
# {
#   "success": true,
#   "data": {
#     "userId": "550e8400-e29b-41d4-a716-446655440000",
#     "token": "eyJhbGciOiJIUzI1NiIs...",
#     "refreshToken": "q8c-BbPEpIF7ai-FrYUZ_a4acEk7finhMGHQ5vUvBvg",
#     ...
#   }
# }
  4. Create a workspace Creating a workspace also provisions its Iceberg catalog namespace — you'll need that workspaceId for every remaining objective on this page. 
    curl -X POST https://api.dpf-it.com/workspaces \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create",
    "name": "Sales Analytics",
    "description": "Workspace used throughout the DPF examples"
  }'

# 201 Created
# { "success": true, "data": { "workspaceId": "550e8400-e29b-41d4-a716-446655440099", ... } }
The Iceberg Namespace
Query and catalog access (Objectives 5–6) address this workspace by namespace, which is just the trailing 12 characters of workspaceId — for 550e8400-e29b-41d4-a716-446655440099 that's 446655440099. No lookup call needed; it's a pure string operation on the ID you already have.

2Load a File into a New Table, Schema Inferred

This is Flow A from the /data-specs endpoint with targetOption: "auto-infer": give DPF a sample file, and AI infers the target schema, generates parsing/transform code, and loads the sample — creating a brand-new Iceberg table in the process. No pre-existing table or schema file needed.

  1. Create the spec Returns a presigned upload URL for the sample file (valid 1 hour, 100 KB max — a representative sample is enough, DPF re-runs this same logic against full files later). 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-spec",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Customer Signups",
    "sampleFileName": "customers.csv",
    "targetOption": "auto-infer",
    "computeSize": "small",
    "description": "Load new customer signups into a fresh Iceberg table"
  }'

# 201 Created
# {
#   "success": true,
#   "data": {
#     "specId": "550e8400-e29b-41d4-a716-446655440003",
#     "signedUrls": { "customers.csv": "https://dpf-specs.s3.us-east-1.amazonaws.com/...?X-Amz-Signature=..." },
#     "expiresIn": 3600,
#     ...
#   }
# }
  2. Upload the sample file customers.csv: 
    id,name,email,signup_date
1,Acme Corp,ops@acme.test,2026-07-01
2,Beta Inc,hello@beta.test,2026-07-01
3,Globex LLC,contact@globex.test,2026-07-02
    PUT it straight to the signed URL from step 1 (it's a standard S3 presigned PUT, not a DPF endpoint — no Authorization header): 
    curl -X PUT "SIGNED_URL_FROM_STEP_1" \
  --upload-file customers.csv
  3. Start analysis Kicks off schema inference + code generation, then (by default) loads the sample and starts the Glue data-load job. Returns immediately — poll get-status for progress. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start-analysis",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Customer Signups"
  }'

# 202 Accepted
# { "success": true, "data": { "specId": "...", "status": "processing", ... } }
  4. Poll the spec's status Poll with specId until status is ready (or failed). Once ready, lastJobId points at the data-load run itself. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "get-status",
    "specId": "550e8400-e29b-41d4-a716-446655440003"
  }'

# 200 OK
# { "success": true, "data": { "status": "ready", "lastJobId": "550e8400-e29b-41d4-a716-446655440005", ... } }
  5. Poll the load itself Poll the same endpoint with jobId for the Glue run's own progress and, once complete, its metrics. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "get-status",
    "jobId": "550e8400-e29b-41d4-a716-446655440005"
  }'

# 200 OK
# {
#   "success": true,
#   "data": {
#     "status": "completed",
#     "metrics": { "recordsRead": 3, "recordsWritten": 3, "filesProcessed": 1 },
#     ...
#   }
# }
What You Now Have
A new Iceberg table (named by the AI from the sample — customers in this example) with 4 audit columns appended automatically: dpf_line, dpf_filename, dpf_job, dpf_ts. Confirm the table name with SHOW TABLES in Objective 5 if you're not sure what it inferred.

3Load & Transform into an Existing Table

Still Flow A, but with targetOption: "existing-tables" instead of auto-infer: the target schema is the customers table created in Objective 2, and the source file has a different shape entirely — differently-named columns, a different date format, mixed casing. additionalPrompt carries the field-mapping and transformation instructions, and merge: true makes the load an upsert (by inferred primary key) instead of a plain append, so re-running it with overlapping IDs doesn't create duplicates.

  1. Create the spec against the existing table 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-spec",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Legacy CRM Backfill",
    "sampleFileName": "legacy_crm_export.csv",
    "targetOption": "existing-tables",
    "targetTables": ["customers"],
    "merge": true,
    "computeSize": "small",
    "additionalPrompt": "Source columns are CUST_ID, COMPANY_NAME, EMAIL_ADDR, SIGNUP_MMDDYYYY. Map CUST_ID -> id, COMPANY_NAME -> name, EMAIL_ADDR -> email (lowercased), and parse SIGNUP_MMDDYYYY (MM/DD/YYYY) into signup_date as an ISO 8601 date. Upsert on id.",
    "description": "Backfill customers from the legacy CRM export"
  }'

# 201 Created — same signedUrls / specId shape as Objective 2
  2. Upload the sample file legacy_crm_export.csv — note the different column names, casing, and date format vs. the target customers table: 
    CUST_ID,COMPANY_NAME,EMAIL_ADDR,SIGNUP_MMDDYYYY
1,ACME CORP,OPS@ACME.TEST,07/01/2026
4,Initech,billing@initech.test,07/03/2026
    curl -X PUT "SIGNED_URL_FROM_STEP_1" \
  --upload-file legacy_crm_export.csv
  3. Start analysis 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start-analysis",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Legacy CRM Backfill"
  }'
  4. Poll until ready, then poll the load Same two-step polling as Objective 2 (get-status by specId, then by the resulting lastJobId). Because id: 1 already exists in customers from Objective 2, that row is updated in place (company casing normalized, email lowercased) — id: 4 is inserted as new.
existing-tables vs. auto-infer
existing-tables only determines column mapping against a schema that already exists — it does not create tables. If customers didn't already exist, this call would fail; that's exactly what Objective 2's auto-infer flow is for.

4Connect an SFTP Server & Trigger a Daily Load

Automate Objective 2's spec so new files dropped on an SFTP server load themselves every day, with no manual create-job/start-job calls. A connection holds how DPF authenticates to the server; a trigger pairs that connection with a spec and a schedule.

  1. Create the connection DPF generates an RSA-4096 keypair and returns the public half — it never returns the private key. 
    curl -X POST https://api.dpf-it.com/connections \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-connection",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "type": "sftp",
    "hostname": "sftp.example.com",
    "username": "sftpuser"
  }'

# 201 Created
# { "success": true, "data": { "connectionId": "...", "publicKey": "ssh-rsa AAAA...", ... } }
  2. Install the public key on the SFTP server As sftpuser on the server (see the Integration Guide for the full authorized_keys permissions walkthrough): 
    mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo 'ssh-rsa AAAA... ' >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
  3. Test the connection A connection must pass this before it can be used in a trigger. 
    curl -X POST https://api.dpf-it.com/connections \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "test-connection",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "connectionId": "YOUR_CONNECTION_ID"
  }'

# 200 OK — { "success": true, "data": { "success": true, ... } }
  4. Create the daily trigger Points at the Objective 2 spec (Customer Signups) — it must already have transformation code from a prior start-analysis, which it does. preRules/postRules are plain English; DPF compiles them into executable JS and returns it read-only as preCode/postCode. 
    curl -X POST https://api.dpf-it.com/job-triggers \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-trigger",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "type": "sftp",
    "specName": "Customer Signups",
    "connectionId": "YOUR_CONNECTION_ID",
    "frequency": { "unit": "daily", "hourOfDay": 6 },
    "dedupe": true,
    "preRules": "Process only *.csv files under /outbound",
    "postRules": "Rename each processed file with a .done suffix"
  }'

# 201 Created — { "success": true, "data": { "triggerId": "...", "preCode": "...", "postCode": "...", ... } }
  5. (Optional) Fire it once immediately to test Don't want to wait for 06:00 UTC to confirm it works end-to-end: 
    curl -X POST https://api.dpf-it.com/job-triggers \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "run-trigger-now",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "triggerId": "YOUR_TRIGGER_ID"
  }'

# 202 Accepted — the run shows up under list-trigger-runs on /workspaces
From here on, it's automatic
An hourly dispatcher checks every enabled trigger against its frequency — no more manual calls needed for day-to-day loads. Each run stamps its rows with dpf_ts, which is exactly what Objective 6 filters on to find "what's new."

5Query the Fully Loaded Data

POST /query runs SQL directly against your Iceberg tables, scoped to a single namespace — the trailing 12 characters of workspaceId (see the callout in Objective 1). This objective covers reads — SELECT, DESCRIBE, and SHOW TABLES; Objective 6 covers the write side.

  1. Confirm the table name 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "SHOW TABLES"
  }'
  2. Select rows 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "SELECT * FROM customers ORDER BY dpf_ts DESC LIMIT 10"
  }'

# 200 OK
# {
#   "success": true,
#   "data": {
#     "columns": ["id", "name", "email", "signup_date", "dpf_line", "dpf_filename", "dpf_job", "dpf_ts"],
#     "rows": [ { "id": "4", "name": "Initech", "email": "billing@initech.test", ... } ],
#     "rowCount": 1
#   }
# }
  3. Count and aggregate dpf_filename/dpf_job make it easy to see how much each load contributed: 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "SELECT dpf_filename, COUNT(*) AS rows_loaded FROM customers GROUP BY dpf_filename ORDER BY rows_loaded DESC"
  }'
Heavier queries
POST /query-large is a drop-in replacement with 4× the memory/CPU and a 120s timeout (vs. 60s), for large scans or heavy joins/aggregations. It costs 4× the credits per call.

6Query Incrementally Loaded Data & Merge into an Aggregate Table

This objective uses a capability the rest of this page hasn't needed: POST /query accepts real write SQL too — INSERT, UPDATE, DELETE, and MERGE — for any workspace member with full access (not read-only), committed durably to the Iceberg table. That makes the whole incremental-aggregate pattern a single statement: read the rows added to customers since the last run, and MERGE them straight into the aggregate table.

"Since the last run" is a plain dpf_ts BETWEEN window on the audit column every load already stamps.

  1. One-time: create the aggregate table No need to route this through /data-specs — since you already know the target shape, create it directly against the Iceberg REST catalog. It's synchronous, no sample file or AI step involved, and comes back with zero snapshots (empty). 
    curl -X POST https://api.dpf-it.com/iceberg/v1/namespaces/446655440099/tables \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "customer_signups_daily",
    "schema": {
      "type": "struct",
      "fields": [
        { "id": 1, "name": "signup_date", "type": "date", "required": true },
        { "id": 2, "name": "total_signups", "type": "long", "required": true }
      ]
    }
  }'

# 200 OK — { "metadata-location": "...", "metadata": { "current-snapshot-id": -1, "snapshots": [], ... } }
    Requires full permission on the workspace, same as create-spec.
  2. Set the run window as shell variables From here on, this objective is a short chain of commands, so pull the moving parts into shell variables instead of repeating literals. RUN_TS is captured fresh each run, before the merge executes; LAST_RUN_TS is whatever RUN_TS was set to at the end of the previous run (see the last step below) — first run only, seed it with a floor from before Objective 2's initial load. 
    export DPF_TOKEN="YOUR_DPF_JWT_TOKEN"
export NAMESPACE="446655440099"

RUN_TS=$(date -u +"%Y-%m-%d %H:%M:%S")
LAST_RUN_TS="2026-07-01 00:00:00"   # first run only; later runs reuse the previous RUN_TS
  3. Merge the change window into the aggregate table One statement: a plain dpf_ts BETWEEN filter isolates the rows loaded in this window, grouped by signup_date and joined against the target — WHEN MATCHED adds to an existing day's total, WHEN NOT MATCHED inserts a new day. Sent to /query as the query field, with $NAMESPACE, $LAST_RUN_TS, and $RUN_TS interpolated in via an unquoted heredoc; \n inside the query string keeps the statement readable without repeating it outside the JSON: 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer $DPF_TOKEN" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "namespace": "$NAMESPACE",
  "query": "MERGE INTO customer_signups_daily AS tgt\nUSING (\n    SELECT signup_date, COUNT(*) AS new_signups\n    FROM customers\n    WHERE dpf_ts BETWEEN TIMESTAMP '$LAST_RUN_TS' AND TIMESTAMP '$RUN_TS'\n    GROUP BY signup_date\n) AS src\nON tgt.signup_date = src.signup_date\nWHEN MATCHED THEN UPDATE SET total_signups = tgt.total_signups + src.new_signups\nWHEN NOT MATCHED THEN INSERT (signup_date, total_signups)\n  VALUES (src.signup_date, src.new_signups)"
}
EOF

# 200 OK — { "success": true, "data": { "columns": [], "rows": [], "rowCount": 0 } }, write committed

LAST_RUN_TS=$RUN_TS   # ready for the next run
    BETWEEN is inclusive on both ends — a row landing exactly on RUN_TS would be picked up again by the next run's LAST_RUN_TS. In practice dpf_ts carries sub-second precision and each run captures its own fresh RUN_TS, so an exact collision is vanishingly unlikely — use >/<= instead of BETWEEN if you need that edge airtight. DPF doesn't track per-caller checkpoints for you — persisting LAST_RUN_TS across runs (a file, a small DynamoDB row, whatever) is on whatever schedules the next run.
Full access required
DML (INSERT/UPDATE/DELETE/MERGE) via /query is rejected with 403 for workspace members whose permission is read-only — only owner/full members can write. The same per-request, namespace-scoped credentials that confine reads to your workspace also confine writes, so there's no separate write-access boundary to reason about.
The recurring version
In production, steps 2–4 are the shape of a small scheduled job (cron, Lambda, Airflow task — outside DPF) that runs after each of Objective 4's trigger fires: capture the run window, MERGE the delta, then persist RUN_TS as the next run's LAST_RUN_TS. Step 1 only ever runs once.