Examples

Six concrete objectives, each worked end-to-end as a sequence of DPF API calls or MCP prompts.

Overview

Each objective below is self-contained but builds on the ones before it — together they walk a single workspace from account creation through a scheduled ingestion pipeline and into aggregated reporting. Run them in order the first time through — click any step to jump to it:

1. Register & verify OTPget a JWT and create a workspace
↓
2. Load a file into a new tableschema inferred by AI
↓
3. Load & transform into an existing tablefield mapping + business rules via additionalPrompt
↓
4. SFTP connection + daily triggerre-runs the Objective 2 spec's existing transformation code automatically
↓
5. Query the fully loaded dataSQL over your Iceberg tables
↓
6. Aggregate into a summary tabletable-source spec windowed automatically, fired by a spec_success trigger on Objective 4's load

Conventions Used Below

  • Base URL is https://api.dpf-it.com. Every request/response is JSON.
  • Success responses are shaped { "success": true, "data": {...} }; errors are { "error": { "code": "...", "message": "..." } }.
  • YOUR_DPF_JWT_TOKEN is the token from Objective 1's login call. Pass it as Authorization: Bearer YOUR_DPF_JWT_TOKEN on every authenticated request below.
  • YOUR_WORKSPACE_ID is the workspaceId created in Objective 1 — reused by every other objective on this page.
  • Most endpoints (/data-specs, /connections, /job-triggers, /workspaces) are multi-action: one URL, with an action field in the JSON body selecting the operation.
  • Full request/response schemas are in the API Docs; this page is the "how do these calls chain together" companion to that reference.

1Register a User & Verify OTP

Create an account, verify it with the one-time code emailed on registration, log in for a JWT, and create the workspace that every other objective on this page will load data into and query.

API
  1. Register termsAccepted must be true or registration is rejected. This sends a 6-digit OTP to the given email. 
    curl -X POST https://api.dpf-it.com/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane@example.com",
    "password": "SecurePassword123",
    "firstName": "Jane",
    "lastName": "Doe",
    "termsAccepted": true
  }'

# 201 Created
# { "success": true, "data": { "userId": "...", "email": "jane@example.com", ... } }
  2. Verify the OTP Use the 6-digit code from the verification email. If it expired or never arrived, call POST /auth/resend-otp with just {"email": "..."}. 
    curl -X POST https://api.dpf-it.com/auth/verify-otp \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane@example.com",
    "otp": "123456"
  }'

# 200 OK
# { "success": true, "data": { "message": "Email verified successfully" } }
  3. Log in Returns a 24h-lived access token plus a ~90-day refreshToken (exchange it at POST /auth/refresh once the access token expires). Save the token — every call below sends it as Authorization: Bearer .... 
    curl -X POST https://api.dpf-it.com/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "email": "jane@example.com",
    "password": "SecurePassword123"
  }'

# 200 OK
# {
#   "success": true,
#   "data": {
#     "userId": "550e8400-e29b-41d4-a716-446655440000",
#     "token": "eyJhbGciOiJIUzI1NiIs...",
#     "refreshToken": "q8c-BbPEpIF7ai-FrYUZ_a4acEk7finhMGHQ5vUvBvg",
#     ...
#   }
# }
  4. Create a workspace Creating a workspace also provisions its Iceberg catalog namespace — you'll need that workspaceId for every remaining objective on this page. 
    curl -X POST https://api.dpf-it.com/workspaces \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create",
    "name": "Sales Analytics",
    "description": "Workspace used throughout the DPF examples"
  }'

# 201 Created
# { "success": true, "data": { "workspaceId": "550e8400-e29b-41d4-a716-446655440099", ... } }
MCP
  1. Add DPF's MCP server to Claude Settings → Connectors → Add custom connector → paste https://api.dpf-it.com/mcp as the remote server URL. Claude registers itself as a client automatically — there's no separate credential to create first.
  2. Log in when Claude redirects you On first use, Claude opens a browser to DPF's own login page. No account yet? The same page's sign-up form handles registration and OTP verification before sending you back to Claude — your password is typed there, never into a chat message.
  3. Nothing to copy or save Once you're redirected back, Claude stores its own access and refresh token and attaches it to every tool call automatically. There's no JWT to paste anywhere, and you won't be asked to log in again unless you revoke access or the refresh token itself expires.
  4. Create a workspace Ask, in plain English: "create a workspace called Sales Analytics." Claude calls create_workspace and every other objective on this page reuses it automatically.
The Iceberg Namespace
Query and catalog access (Objectives 5–6) address this workspace by namespace, which is just the trailing 12 characters of workspaceId — for 550e8400-e29b-41d4-a716-446655440099 that's 446655440099. No lookup call needed; it's a pure string operation on the ID you already have.

2Load a File into a New Table, Schema Inferred

Give DPF a sample file and set targetOption: "auto-infer" on the spec: AI infers the target schema, generates parsing/transform code, and loads the sample — creating a brand-new Iceberg table in the process. No pre-existing table or schema file needed.

API
  1. Create the spec Returns a presigned upload URL for the sample file (valid 1 hour, 100 KB max — a representative sample is enough, DPF re-runs this same logic against full files later). 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-spec",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Customer Signups",
    "sampleFileName": "customers.csv",
    "targetOption": "auto-infer",
    "computeSize": "small",
    "description": "Load new customer signups into a fresh Iceberg table"
  }'

# 201 Created
# {
#   "success": true,
#   "data": {
#     "specId": "550e8400-e29b-41d4-a716-446655440003",
#     "signedUrls": { "customers.csv": "https://dpf-specs.s3.us-east-1.amazonaws.com/...?X-Amz-Signature=..." },
#     "expiresIn": 3600,
#     ...
#   }
# }
  2. Upload the sample file customers.csv: 
    id,name,email,signup_date
1,Acme Corp,ops@acme.test,2026-07-01
2,Beta Inc,hello@beta.test,2026-07-01
3,Globex LLC,contact@globex.test,2026-07-02
    PUT it straight to the signed URL from step 1 (it's a standard S3 presigned PUT, not a DPF endpoint — no Authorization header): 
    curl -X PUT "SIGNED_URL_FROM_STEP_1" \
  --upload-file customers.csv
  3. Start analysis Kicks off schema inference + code generation, then (by default) loads the sample and starts the Glue data-load job. Returns immediately — poll get-status for progress. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start-analysis",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Customer Signups"
  }'

# 202 Accepted
# { "success": true, "data": { "specId": "...", "status": "processing", ... } }
  4. Poll the spec's status Poll with specId until status is ready (or failed). Once ready, lastJobId points at the data-load run itself. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "get-status",
    "specId": "550e8400-e29b-41d4-a716-446655440003"
  }'

# 200 OK
# { "success": true, "data": { "status": "ready", "lastJobId": "550e8400-e29b-41d4-a716-446655440005", ... } }
  5. Poll the load itself Poll the same endpoint with jobId for the Glue run's own progress and, once complete, its metrics. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "get-status",
    "jobId": "550e8400-e29b-41d4-a716-446655440005"
  }'

# 200 OK
# {
#   "success": true,
#   "data": {
#     "status": "completed",
#     "metrics": { "recordsRead": 3, "recordsWritten": 3, "filesProcessed": 1 },
#     ...
#   }
# }
MCP

Describe the outcome in plain English. Claude picks the tools, uploads the sample file, and polls the job to completion — below is the real tool-call sequence from one such session.

Prompt Using DPF, load new customer signups into a fresh Iceberg table, use the customers.csv file as an example.
Thought for 6s
Dpf-it[list_my_workspaces]
workspace confirmed0.2s

Located customer.csv in the project folder (150-row sample) and checked existing specs — WebLogDaily and WebLogs are unrelated web-log pipelines, nothing for customer data yet.

Thought for 7s
Dpf-it[list_data] resource: "specs"
2 unrelated specs found2.6s
Thought for 2s
Dpf-it[onboard_data_source] specName: "CustomerSignups", sourceType: "file", sampleFileName: "customer.csv", targetOption: "auto-infer"
spec created, presigned upload URL returned56s
Uploaded customer.csv to that URL — a plain S3 PUT, not a DPF call — then continued.
Thought for 2s
Dpf-it[finish_data_source_onboarding] specId: "8a4e4fad-...", specName: "CustomerSignups"
processing — schema inference under way
Dpf-it[get_status] specId: "8a4e4fad-..."
processing, 65% — "Generating schema transformation code"5s
⏱ waited 15s
Dpf-it[get_status] specId: "8a4e4fad-..."
ready — analysis complete, load job started automatically0.3s

The load job kicked off on its own — polled that next.

Dpf-it[get_status] jobId: "dd14d3ea-..."
processing, 85% — "Starting data processing environment"0.4s
⏱ waited 15s
Dpf-it[get_status] jobId: "dd14d3ea-..."
complete — data transformation completed successfully0.4s

Verified the result with SQL, directly over MCP:

Thought for 1s
Dpf-it[submit_query] sql: "SELECT COUNT(*) AS row_count FROM customer"
150 rows1.1s
Dpf-it[submit_query] sql: "SELECT * FROM customer LIMIT 5"
sample rows returned0.9s
Done. One prompt → spec created, sample uploaded, schema inferred, and 150 rows landed in a brand-new customer Iceberg table, confirmed by query.
What You Now Have
A new Iceberg table (named by the AI from the sample — customers in this example) with 4 audit columns appended automatically: dpf_line, dpf_filename, dpf_job, dpf_ts. Confirm the table name with SHOW TABLES in Objective 5 if you're not sure what it inferred.

3Load & Transform into an Existing Table

Same /data-specs endpoint as Objective 2, but with targetOption: "existing-tables" instead of auto-infer: the target schema is the customers table created in Objective 2, and the source file has a different shape entirely — differently-named columns, a different date format, mixed casing. additionalPrompt carries the field-mapping and transformation instructions, and merge: true makes the load an upsert instead of a plain append, so re-running it with overlapping IDs doesn't create duplicates — that requires the target table to already have a primary key set (Step 1 below), which is what DPF upserts on.

API
  1. Set a primary key on the target table Because we're about to do merge: true loads against customers, it needs a primary key for DPF to match rows on — a plain Iceberg table property (dpf.primary-keys), not a data-specs call. One-time per table — skip this if the table already has one (e.g. it was created via auto-infer with merge: true, which sets this automatically; see Objective 2). Without it, the next step fails with 400 MISSING_PRIMARY_KEY instead of silently appending duplicate rows. The easiest way is ALTER TABLE ... ADD PRIMARY KEY through the /query endpoint — DuckDB and Iceberg don't support that syntax natively, so DPF handles it as a special case and sets the property directly: 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "ALTER TABLE customers ADD PRIMARY KEY (id)"
  }'

# 200 OK — { "success": true, "data": { "rows": [{ "result": "ALTER TABLE" }], ... } }
    (ALTER TABLE customers DROP PRIMARY KEY removes it the same way.) The same property can also be set at a lower level directly through the Iceberg REST Catalog (see API Docs) if you're already scripting against that endpoint: 
    curl -X POST https://api.dpf-it.com/iceberg/v1/namespaces/446655440099/tables/customers \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "requirements": [],
    "updates": [
      { "action": "set-properties", "updates": { "dpf.primary-keys": "id" } }
    ]
  }'

# 200 OK — table metadata updated
  2. Create the spec against the existing table 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-spec",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Legacy CRM Backfill",
    "sampleFileName": "legacy_crm_export.csv",
    "targetOption": "existing-tables",
    "targetTables": ["customers"],
    "merge": true,
    "computeSize": "small",
    "additionalPrompt": "Source columns are CUST_ID, COMPANY_NAME, EMAIL_ADDR, SIGNUP_MMDDYYYY. Map CUST_ID -> id, COMPANY_NAME -> name, EMAIL_ADDR -> email (lowercased), and parse SIGNUP_MMDDYYYY (MM/DD/YYYY) into signup_date as an ISO 8601 date. Upsert on id.",
    "description": "Backfill customers from the legacy CRM export"
  }'

# 201 Created — same signedUrls / specId shape as Objective 2
  3. Upload the sample file legacy_crm_export.csv — note the different column names, casing, and date format vs. the target customers table: 
    CUST_ID,COMPANY_NAME,EMAIL_ADDR,SIGNUP_MMDDYYYY
1,ACME CORP,OPS@ACME.TEST,07/01/2026
4,Initech,billing@initech.test,07/03/2026
    curl -X PUT "SIGNED_URL_FROM_STEP_1" \
  --upload-file legacy_crm_export.csv
  4. Start analysis 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start-analysis",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "Legacy CRM Backfill"
  }'
  5. Poll until ready, then poll the load Same two-step polling as Objective 2 (get-status by specId, then by the resulting lastJobId). Because id: 1 already exists in customers from Objective 2, that row is updated in place (company casing normalized, email lowercased) — id: 4 is inserted as new.
MCP

Ask Claude to load the file into the existing table with the field mapping you want. If the target table doesn't have a primary key set yet for the upsert, Claude sets one via the Iceberg REST endpoint before continuing — one prompt covers both steps.

Prompt Using the DPF MCP, load legacy_crm_export.csv into the existing customer table. Map CUST_ID -> id, COMPANY_NAME -> name, EMAIL_ADDR -> email (lowercased), and parse SIGNUP_MMDDYYYY into signup_date as an ISO 8601 date. Upsert on id.

Checked existing specs first — the customer table from Objective 2 already has signup_date/email columns, so this is a field mapping, not a schema change.

Thought for 3s
Dpf-it[list_data] resource: "specs"
customer table schema confirmed0.5s
Thought for 2s
Dpf-it[onboard_data_source] specName: "LegacyCRMImport", targetOption: "existing-tables", targetTables: ["customer"], merge: true
Blocked: customer has no primary key set, so merge/upsert has no key to match rows on — the same requirement as Step 1 in the API column above. The error came back with the exact call_dpf_api params to fix it.

Set the primary key through the Iceberg REST endpoint — same operation as the curl example above, via call_dpf_api (the MCP escape hatch for endpoints without a dedicated tool):

Thought for 1s
Dpf-it[call_dpf_api] path: "/iceberg/v1/namespaces/.../tables/customer", params: { updates: [{ action: "set-properties", updates: { "dpf.primary-keys": "id" } }] }
table metadata updated0.7s
Thought for 2s
Dpf-it[onboard_data_source] specName: "LegacyCRMImport", targetOption: "existing-tables", targetTables: ["customer"], merge: true, additionalPrompt: "Map CUST_ID -> id; COMPANY_NAME -> name; EMAIL_ADDR -> email, lowercased; SIGNUP_MMDDYYYY -> signup_date, parsed MM/DD/YYYY into ISO 8601..."
spec created, presigned upload URL returned1.1s
Uploaded legacy_crm_export.csv to that URL — a plain S3 PUT, not a DPF call — then continued.
Thought for 2s
Dpf-it[finish_data_source_onboarding] specId: "2722d276-...", specName: "LegacyCRMImport"
processing, 45% — "Determining target schema"11.8s
This run's analysis took longer than the call's polling window — the response is explicit that calling finish_data_source_onboarding again would re-attempt starting it (and could trigger a second load job), and to poll get_status instead. The agent did exactly that below — no repeat call, no duplicate job.
Dpf-it[get_status] specId: "2722d276-..."
processing, 65% — "Generating schema transformation code"0.4s
⏱ waited 10s
Dpf-it[get_status] specId: "2722d276-..."
ready — analysis complete (28.6s total), load job started automatically0.4s

One load job (b9b085f2-...) — polled it to completion:

Dpf-it[get_status] jobId: "b9b085f2-..."
processing, 85% — "Starting data processing environment"0.7s
⏱ waited 15s
Dpf-it[get_status] jobId: "b9b085f2-..."
still processing, 85%0.8s
⏱ waited 20s
Dpf-it[get_status] jobId: "b9b085f2-..."
complete — data transformation completed successfully ($0.075 charged)0.4s

Verified the mapping with SQL, directly over MCP:

Thought for 1s
Dpf-it[submit_query] sql: "SELECT id, name, email, signup_date FROM customer WHERE id BETWEEN 1 AND 9 ORDER BY id"
9 rows — e.g. 07132026 → 2026-07-13, emails lowercased1.3s
Done. One primary-key fix, one spec, one upload, one analysis run, one load job — even though analysis alone ran longer than a single poll window, the agent never re-triggered it. 9 rows upserted into customer with the requested field mapping, confirmed by query.
existing-tables vs. auto-infer
existing-tables only determines column mapping against a schema that already exists — it does not create tables. If customers didn't already exist, this call would fail; that's exactly what Objective 2's auto-infer approach is for.

4Connect an SFTP Server & Trigger a Daily Load

Automate Objective 2's spec so new files dropped on an SFTP server load themselves every day, with no manual create-job/start-job calls. A connection holds how DPF authenticates to the server; a trigger pairs that connection with a spec and a schedule.

API
  1. Create the connection DPF generates an RSA-4096 keypair and returns the public half — it never returns the private key. 
    curl -X POST https://api.dpf-it.com/connections \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-connection",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "type": "sftp",
    "hostname": "test-sftp.dpf-it.com",
    "username": "sftpuser"
  }'

# 201 Created
# { "success": true, "data": { "connectionId": "...", "publicKey": "ssh-rsa AAAA...", ... } }
  2. If sftpuser doesn't already exist, create it Skip this if the connection's username is an existing SFTP account on the server — it already has a .ssh directory. 
    sudo useradd -m sftpuser
sudo -iu sftpuser
mkdir -p ~/.ssh && chmod 700 ~/.ssh
  3. Install the public key on the SFTP server As the connection's username on the server. 
    # As an administrator, become that user first:
sudo -iu sftpuser

echo 'ssh-rsa AAAA... ' >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
  4. Test the connection A connection must pass this before it can be used in a trigger. 
    curl -X POST https://api.dpf-it.com/connections \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "test-connection",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "connectionId": "YOUR_CONNECTION_ID"
  }'

# 200 OK — { "success": true, "data": { "success": true, ... } }
  5. Create the daily trigger Points at the Objective 2 spec (Customer Signups) — it must already have transformation code from a prior start-analysis, which it does. preRules/postRules are plain English; DPF compiles them into executable JS and returns it read-only as preCode/postCode. postRules is optional — omitted here, meaning "do nothing after load"; combined with dedupe: false below, that's a deliberate trade-off (an untouched file reloads every run) rather than an oversight — see the MCP tab for the reasoning DPF walks through before making that same choice. 
    curl -X POST https://api.dpf-it.com/job-triggers \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-trigger",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "type": "sftp",
    "specName": "Customer Signups",
    "connectionId": "YOUR_CONNECTION_ID",
    "frequency": { "unit": "daily", "hourOfDay": 2 },
    "dedupe": false,
    "preRules": "Only pick up *.csv files"
  }'

# 201 Created — { "success": true, "data": { "triggerId": "...", "preCode": "...", "postCode": "...", ... } }
  6. (Optional) Fire it once immediately to test Don't want to wait for 02:00 UTC to confirm it works end-to-end: 
    curl -X POST https://api.dpf-it.com/job-triggers \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "run-trigger-now",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "triggerId": "YOUR_TRIGGER_ID"
  }'

# 202 Accepted — the run shows up under list-trigger-runs on /workspaces
MCP

Describe the connection and schedule in plain English — server, username, and what "the customer load" refers to. If a detail is ambiguous, Claude asks before calling any DPF tool rather than guessing; when it needs the public key installed on the server, install it and reply to continue.

Prompt Using DPF, help me create a connection to my sftp server "test-sftp.dpf-it.com" connecting as the user "sftpuser" and scheduling a daily trigger for the customer load.
Thought for 12s
Dpf-it[list_my_workspaces]
Dpf-it[list_data] resource: "specs"
workspace confirmed; 2 candidate specs found — CustomerSignups and LegacyCRMImport, both plausible matches for "the customer load"1.8s

"The customer load" was ambiguous between two specs, and the request didn't say what time to run or whether re-pulled files should be skipped — asked all three before touching any DPF API.

Thought for 7s
Asked
Which data spec is "the customer load" that the SFTP trigger should feed?
CustomerSignups · LegacyCRMImport
What time (UTC) should the daily pull run?
02:00 UTC · 06:00 UTC · 00:00 UTC · Other
Should repeat pulls skip files already loaded (dedupe by file name)? dedupe=true silently skips a same-named file that reappears; dedupe=false reloads an unchanged file every run if it's left on the server.
Yes, dedupe on file name (Recommended) · No, always reload
⏱ waiting on your answer
Answered CustomerSignups · 02:00 UTC · No, always reload

"Always reload" with no post-load action means an untouched file would load again every day — followed up on what should happen to a file after it loads, plus the file filter, before creating anything.

Thought for 6s
Asked
After a file is successfully loaded, what should happen to it on the SFTP server? (With dedupe off, an untouched file gets reloaded every day.)
Rename with .done suffix (Recommended) · Leave files untouched · Other
Any restriction on which files to pick up (file type or directory), or pull everything?
Only *.csv files · Pull everything · Other
⏱ waiting on your answer
Answered Leave files untouched · Only *.csv files

Every parameter was now explicit — spec, schedule, dedupe, post-load action, file filter — so it created the connection and trigger together in one call.

Thought for 4s
Dpf-it[setup_scheduled_pull] specName: "CustomerSignups", hostname: "test-sftp.dpf-it.com", username: "sftpuser", frequency: {unit: "daily", hourOfDay: 2}, preRules: "Only pick up *.csv files", dedupe: false
connection ba438616-... created; test failed — public key not yet installed on the server3.4s
Returned the generated public key and asked me to install it in sftpuser's authorized_keys on test-sftp.dpf-it.com, then say when ready — it didn't retry or abandon the setup, just paused there with everything configured so far summarized.
⏸ Paused ~53s here while the public key was installed on the server
Me continue
Thought for 2s
Dpf-it[setup_scheduled_pull] same arguments as above — reuses the existing connection by matching hostname/username
reused connection ba438616-...; test succeeded (4,200 files listed); trigger 64c90033-... created on a daily 02:00 UTC schedule2.4s

Offered to fire the trigger immediately instead of waiting for tomorrow's 02:00 UTC run, to confirm the whole path works end-to-end.

Me yes, run now
Thought for 3s
Dpf-it[manage_trigger] action: "run-now", triggerId: "64c90033-..."
run started0.4s
Dpf-it[manage_trigger] action: "run-history", triggerId: "64c90033-...", pageSize: 5
success — 1 file pulled, job 740706ca-... started0.6s
Done. Two rounds of clarifying questions up front (spec, schedule, dedupe, then post-load handling and file filter), a real pause for you to install the SSH key mid-setup, and a check-in before firing the trigger early — no parameter left for DPF to silently guess at.
From here on, it's automatic
An hourly dispatcher checks every enabled trigger against its frequency — no more manual calls needed for day-to-day loads. Each run stamps its rows with dpf_ts, which is exactly what the table-source spec in Objective 6 uses under the hood to window each re-run to "what's new."

5Query the Fully Loaded Data

POST /query runs SQL directly against your Iceberg tables, scoped to a single namespace — the trailing 12 characters of workspaceId (see the callout in Objective 1). This objective covers reads — SELECT, DESCRIBE, and SHOW TABLES; Objective 6 covers the write side.

API
  1. Confirm the table name 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "SHOW TABLES"
  }'
  2. Select rows 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "SELECT * FROM customers ORDER BY dpf_ts DESC LIMIT 10"
  }'

# 200 OK
# {
#   "success": true,
#   "data": {
#     "columns": ["id", "name", "email", "signup_date", "dpf_line", "dpf_filename", "dpf_job", "dpf_ts"],
#     "rows": [ { "id": "4", "name": "Initech", "email": "billing@initech.test", ... } ],
#     "rowCount": 1
#   }
# }
  3. Count and aggregate dpf_filename/dpf_job make it easy to see how much each load contributed: 
    curl -X POST https://api.dpf-it.com/query \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "446655440099",
    "query": "SELECT dpf_filename, COUNT(*) AS rows_loaded FROM customers GROUP BY dpf_filename ORDER BY rows_loaded DESC"
  }'
MCP

Just ask, in plain English — Claude finds the workspace and writes the SQL, so you never need to look up a namespace. Ask a follow-up and it keeps going from there.

Prompt using dpf, show me the tables in my workspace
Thought for 9s
Dpf-it[submit_query] sql: "SHOW TABLES"
4 tables found — aws_edge_locations, cloudfront_log, cloudfront_log_daily, customer5.0s
Summarized each table's likely contents from its name alone (nothing else queried yet) and offered to show schema or sample rows next.
Me provide me a sample of the customer table
Thought for 2s
Dpf-it[submit_query] sql: "SELECT * FROM customer LIMIT 10"
10 rows returned1.5s
Rendered as a table, dropping the free-text comment field and the dpf_line/dpf_filename/dpf_job/dpf_ts lineage columns for readability — noted they were available on request.
Me give me a count of records by the filename which loaded it
Thought for 4s
Dpf-it[submit_query] sql: "SELECT dpf_filename, COUNT(*) AS record_count FROM customer GROUP BY dpf_filename ORDER BY record_count DESC"
customer.csv: 141 · legacy_crm_export.csv: 9 — 150 total2.9s
Done. Three plain-English questions, three submit_query calls — schema explored, sample rows inspected, and per-file load counts confirmed, without knowing a namespace ahead of time.
Heavier queries
POST /query-large is a drop-in replacement with 4× the memory/CPU and a 120s timeout (vs. 60s), for large scans or heavy joins/aggregations. It costs 4× the credits per call.

6Aggregate Loaded Data Automatically with a Table-Source Spec

This spec's source is an existing Iceberg table instead of an uploaded file: set sourceType: "tables", and start-analysis generates a SQL query (grouping/aggregating from sourceTables) plus a deterministic INSERT/MERGE into targetTables — DPF windows each run to rows added since the spec's own last successful run automatically, so there's no dpf_ts bookkeeping to do by hand.

Paired with a spec_success trigger pointed at Objective 4's spec, the aggregate re-runs itself the moment that day's load finishes — no external cron, Lambda, or Airflow task required.

API
  1. Create the table-source spec No sample or format file to upload here — sourceTables supplies the data instead. targetOption: "auto-infer" has the AI design and create customer_signups_daily from the query's own output shape (pass targetTables instead to pin an existing table's name, same as Objective 3). merge: true makes re-runs upsert existing days instead of double-counting them. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-spec",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "daily-signups-aggregate",
    "sourceType": "tables",
    "sourceTables": ["customers"],
    "targetOption": "auto-infer",
    "merge": true,
    "additionalPrompt": "Count signups per day from the customers table, as columns signup_date and total_signups",
    "description": "Roll up customer signups into a daily summary table"
  }'

# 201 Created
# { "success": true, "data": { "specId": "550e8400-e29b-41d4-a716-446655440007", ... } }
# No signedUrls — sourceType: tables never uploads sample/format files.
  2. Start analysis Generates the aggregation query and the target table (first run), then executes it via the query engine directly — no Glue job involved. The query is windowed to rows with dpf_ts after the spec's last successful run; on this first run that's the epoch, so it covers every row Objective 2–4 have loaded so far. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start-analysis",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "specName": "daily-signups-aggregate"
  }'

# 202 Accepted
# { "success": true, "data": { "specId": "...", "status": "processing", ... } }
  3. Poll until ready, then poll the run Same two-step polling as every other objective on this page — get-status by specId, then by the resulting lastJobId. 
    curl -X POST https://api.dpf-it.com/data-specs \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "get-status",
    "specId": "550e8400-e29b-41d4-a716-446655440007"
  }'

# 200 OK — { "success": true, "data": { "status": "ready", "lastJobId": "...", "sourceQuerySql": "...", "targetLoadSql": "...", ... } }
    sourceQuerySql/targetLoadSql are always visible on the spec — the AI-generated query, same view-only guarantee as the Python code Objectives 2–3 generate.
  4. Create a spec_success trigger on Objective 4's spec Fires daily-signups-aggregate automatically every time Customer Signups (Objective 4's SFTP-triggered spec) finishes a successful job — no frequency/connectionId, since it isn't polling anything itself. 
    curl -X POST https://api.dpf-it.com/job-triggers \
  -H "Authorization: Bearer YOUR_DPF_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create-trigger",
    "workspaceId": "YOUR_WORKSPACE_ID",
    "type": "spec_success",
    "specName": "daily-signups-aggregate",
    "upstreamSpecName": "Customer Signups"
  }'

# 201 Created — { "success": true, "data": { "triggerId": "...", "type": "spec_success", "upstreamSpecName": "Customer Signups", ... } }
MCP

Describe the rollup you want in plain English. Claude checks existing tables for a naming convention to follow, then asks how the summary should refresh and what it should contain — a table-source spec needs both decided explicitly, so answer those before it creates anything.

Prompt using dpf, Roll up the data by signup_date from the customer table into a daily summary table
Thought for 8s
Dpf-it[list_my_workspaces]
Dpf-it[submit_query] sql: "SHOW TABLES"
1 workspace; 4 tables including customer and an existing cloudfront_log_daily rollup4.8s

Checked customer's columns, then looked at how the workspace's one existing daily rollup was shaped, to follow the same convention rather than invent a new one.

Thought for 2s
Dpf-it[submit_query] sql: "DESCRIBE customer"
14 columns, incl. signup_date, acctbal, mktsegment1.3s
Thought for 4s
Dpf-it[submit_query] sql: "DESCRIBE cloudfront_log_daily"
Dpf-it[list_data] resource: "specs"
confirmed the day-grain rollup pattern already used in this workspace1.1s

Two things the pattern here doesn't answer by itself — how the summary stays current, and what it actually contains — asked before creating anything.

Thought for 41s
Asked
How should the new customer_daily summary table stay up to date?
Manual only · Daily schedule · After signups load
What should each daily row summarize? (count of new signups is a given)
Count + acctbal stats (Recommended) · Count only · Count + acctbal + segment breakdown
⏱ waiting on your answer
Answered After signups load · Count + acctbal + segment breakdown

Both answers map directly onto onboard_data_source's own parameters — no separate manage_trigger call needed afterward.

Thought for 17s
Dpf-it[onboard_data_source] specName: "CustomerDailySummary", sourceType: "tables", sourceTables: ["customer"], targetTables: ["customer_daily"], merge: true, autoRefresh: "spec_success", autoRefreshUpstreamSpecName: "CustomerSignups", additionalPrompt: "Group by signup_date and mktsegment; new_customers = count, total_acctbal = sum(acctbal), derive avg from the stored sum/count rather than overwriting it, since later incremental runs add more rows for the same historical day..."
spec 9d885534-... created; wired to re-run whenever CustomerSignups finishes (trigger 04fa7bfb-...)3.1s
Thought for 2s
Dpf-it[finish_data_source_onboarding] specId: "9d885534-...", specName: "CustomerDailySummary"
ready — query generated, initial load job 72b9279b-... started11.3s
Dpf-it[get_status] jobId: "72b9279b-..."
ready, not yet started0.5s
⏱ waited 5s
Dpf-it[get_status] jobId: "72b9279b-..."
processing, 10% — "Running statement 1 of 2"0.5s
⏱ waited 8s
Dpf-it[get_status] jobId: "72b9279b-..."
complete — 143 row(s) affected0.5s

Verified the resulting table, then went a step further and read back the AI-generated targetLoadSql itself to confirm re-runs would add to existing day/segment rows instead of overwriting them.

Thought for 2s
Dpf-it[submit_query] sql: "SELECT * FROM customer_daily ORDER BY signup_date, mktsegment LIMIT 10"
6 columns: signup_date, mktsegment, new_customers, total_acctbal, dpf_ts, dpf_job6.1s
Thought for 12s
Dpf-it[list_data] resource: "specs", pageSize: 1
confirmed targetLoadSql does t.new_customers + s.new_customers (additive), not an overwrite0.4s
Done. Built customer_daily — grouped by signup_date × mktsegment, 143 rows on the initial load, wired to re-run automatically whenever CustomerSignups finishes loading. Average balance wasn't stored as its own column: the generator derives total_acctbal / new_customers at query time instead, since a stored average can't be merged additively the way a sum and a count can — confirmed by reading the generated SQL back, not just assumed.
From here on, it's automatic
Every time Objective 4's daily trigger loads new rows into customers, this trigger fires daily-signups-aggregate right after — each run's window picks up exactly the rows the previous run hadn't seen yet. A schedule trigger (plain frequency, no upstream spec) is the alternative if you'd rather re-run on a timer than chain off a specific load.
Table-source specs only
Both spec_success and schedule triggers require their own (downstream) specName to already be a sourceType: "tables" spec — neither one supplies a file for a file-source spec to load, so daily-signups-aggregate has to exist first. upstreamSpecName has no such restriction; it can be any spec, file- or table-source.