{ "openapi": "3.1.1", "info": { "title": "Data Lineage Explorer API", "version": "1.0.0", "summary": "Paste SQL or dbt models: table lineage graph, column lineage and upstream/downstream of any field.", "description": "Every endpoint performs the same calculation as the human page and returns inputs, result, units, assumptions, sources, warnings and computedAt. MCP endpoint: /mcp." }, "servers": [ { "url": "https://datalineageexplorer.usefulweb.site" } ], "paths": { "/api/v1/calculate": { "post": { "operationId": "calculate", "summary": "Paste SQL or dbt models: table lineage graph, column lineage and upstream/downstream of any field.", "description": "Paste SQL (CREATE VIEW/TABLE AS SELECT, INSERT … SELECT, MERGE, UPDATE, CTEs, JOINs, UNION) or dbt models with {{ ref() }} and {{ source() }}. A built-in tokenizer and parser derives table-level lineage and column-level lineage for direct references and aliases, marks expressions as derived or aggregated, and traces everything upstream and downstream of a chosen table or column.\n\nMethodology:\n- dbt Jinja is resolved first: {{ ref(\"m\") }} → m, {{ ref(\"pkg\",\"m\") }} → m, {{ source(\"s\",\"t\") }} → s.t, {{ this }} → the current model, {{ config(materialized=…) }} is read; other Jinja is stripped with a note.\n- A hand-written tokenizer handles quoted identifiers (\"x\", `x`, [x]), strings, comments, dollar-quoting and operators; statements are split at top-level semicolons.\n- A recursive-descent parser understands WITH/CTEs (column lists, RECURSIVE), SELECT lists with aliases, FROM/JOIN … ON/USING, comma joins, derived tables, scalar/EXISTS/IN subqueries, UNION/INTERSECT/EXCEPT, SELECT */t.* (incl. EXCEPT/EXCLUDE), CREATE … AS, INSERT … SELECT (positional mapping to the column list or known DDL), MERGE (UPDATE SET / INSERT VALUES) and UPDATE … SET … FROM.\n- Each output column is resolved through CTEs and subqueries to base-table columns. A bare column reference is \"direct\" (or \"renamed\" when aliased); anything else is \"derived\", \"aggregate\" if it uses an aggregate function, or \"constant\" without column inputs. Columns used only in JOIN/WHERE/GROUP BY are listed as filter columns of the statement.\n- Unqualified columns are resolved against the tables in scope using known columns (from CREATE TABLE DDL or earlier statements); if several tables could supply a column it is attributed to all and flagged as ambiguous.\n- Upstream/downstream tracing is a breadth-first walk over column edges (or table edges for a table focus) across all statements; the reported transform is the strongest transformation along the path.\n- Not supported: dynamic SQL, stored procedures, PIVOT/UNPIVOT column lineage, column lineage of table functions (UNNEST/FLATTEN/LATERAL VIEW), Jinja loops and macros, and SELECT * from tables whose columns are unknown (kept as a \"*\" star edge).", "tags": [ "devcloud" ], "x-mcp-tool": "data_lineage_explorer", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/DataLineageExplorerInput" }, "examples": { "example1": { "summary": "dbt models + MERGE: where does revenue come from?", "value": { "sql": "-- model: stg_orders\n{{ config(materialized='view') }}\nselect\n o.id as order_id,\n o.customer_id,\n o.amount_cents / 100.0 as amount,\n o.currency,\n o.status,\n o.created_at\nfrom {{ source('shop', 'orders') }} o\nwhere o.status <> 'test'\n\n-- model: stg_customers\nselect id as customer_id, email, country_code as country, created_at as signed_up_at\nfrom {{ source('shop', 'customers') }}\n\n-- model: fct_orders\n{{ config(materialized='table') }}\nwith paid as (\n select order_id, customer_id, amount, currency, created_at\n from {{ ref('stg_orders') }}\n where status = 'paid'\n),\nfx as (\n select currency, rate_to_eur from {{ source('finance', 'fx_rates') }}\n)\nselect\n p.order_id,\n p.customer_id,\n c.country,\n p.amount * coalesce(fx.rate_to_eur, 1) as amount_eur,\n date_trunc('month', p.created_at) as order_month\nfrom paid p\njoin {{ ref('stg_customers') }} c on c.customer_id = p.customer_id\nleft join fx on fx.currency = p.currency\n\n-- model: rpt_revenue_by_country\nselect country, order_month, sum(amount_eur) as revenue, count(distinct customer_id) as customers\nfrom {{ ref('fct_orders') }}\ngroup by country, order_month\n\n-- file: warehouse/merge_dim_customer.sql\nMERGE INTO analytics.dim_customer t\nUSING stg_customers s ON t.customer_id = s.customer_id\nWHEN MATCHED THEN UPDATE SET email = s.email, country = s.country\nWHEN NOT MATCHED THEN INSERT (customer_id, email, country) VALUES (s.customer_id, s.email, upper(s.country));\n", "focus": "rpt_revenue_by_country.revenue", "direction": "both" } }, "example2": { "summary": "Plain SQL: impact of raw.users.plan", "value": { "sql": "CREATE TABLE raw.events (event_id bigint, user_id bigint, event_type text, ts timestamp, payload json);\nCREATE TABLE raw.users (id bigint, email text, plan text);\n\nCREATE OR REPLACE VIEW mart.daily_active AS\nSELECT CAST(e.ts AS date) AS day, u.plan, COUNT(DISTINCT e.user_id) AS dau\nFROM raw.events e\nJOIN raw.users u ON u.id = e.user_id\nWHERE e.event_type IN ('login', 'page_view')\nGROUP BY 1, 2;\n\nINSERT INTO mart.plan_kpis (day, plan, active_users, source)\nSELECT day, plan, dau, 'web' FROM mart.daily_active\nUNION ALL\nSELECT d.day, d.plan, d.active, 'mobile' FROM (SELECT day, plan, users AS active FROM raw.mobile_daily) d;\n", "focus": "raw.users.plan", "direction": "downstream" } } } } } }, "responses": { "200": { "description": "Calculation result envelope.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/DataLineageExplorerResult" } } } }, "400": { "description": "Malformed JSON body.", "content": { "application/problem+json": { "schema": { "$ref": "#/components/schemas/Problem" } } } }, "422": { "description": "Invalid input (see errors[]).", "content": { "application/problem+json": { "schema": { "$ref": "#/components/schemas/Problem" } } } }, "429": { "description": "Rate limit exceeded (see Retry-After).", "content": { "application/problem+json": { "schema": { "$ref": "#/components/schemas/Problem" } } } } } } }, "/api/v1/tool": { "get": { "operationId": "getToolDefinition", "summary": "Inputs, output schema and examples for this tool", "responses": { "200": { "description": "Tool definition.", "content": { "application/json": { "schema": { "type": "object" } } } } } } } }, "components": { "schemas": { "Problem": { "type": "object", "description": "RFC 9457 problem details. 422 responses include per-field `errors`.", "required": [ "type", "title", "status", "detail" ], "properties": { "type": { "type": "string" }, "title": { "type": "string" }, "status": { "type": "integer" }, "detail": { "type": "string" }, "code": { "type": "string" }, "errors": { "type": "array", "items": { "type": "object", "required": [ "field", "message" ], "properties": { "field": { "type": "string" }, "message": { "type": "string" } } } } } }, "DataLineageExplorerInput": { "type": "object", "additionalProperties": false, "properties": { "sql": { "title": "SQL / dbt models", "description": "Separate statements with \";\". Mark dbt models with a line \"-- model: name\" (or \"-- file: models/name.sql\"); the model’s SELECT becomes that table.", "type": "string", "maxLength": 400000 }, "file": { "title": "Or upload a .sql file", "type": "object", "description": "Upload as {filename, contentType?, contentBase64}. Max 2048 KiB. Accepts .sql,.txt,text/plain.", "additionalProperties": false, "required": [ "filename", "contentBase64" ], "properties": { "filename": { "type": "string" }, "contentType": { "type": "string" }, "contentBase64": { "type": "string" } } }, "focus": { "title": "Focus table or column", "description": "table or table.column. Defaults to the last target in the input.", "type": "string", "maxLength": 200 }, "direction": { "title": "Trace", "type": "string", "enum": [ "both", "upstream", "downstream" ], "default": "both" } }, "required": [] }, "DataLineageExplorerResult": { "type": "object", "additionalProperties": false, "required": [ "schemaVersion", "tool", "computedAt", "inputs", "summary", "result", "units", "assumptions", "sources", "warnings", "presentation", "links" ], "properties": { "schemaVersion": { "type": "string", "const": "1.0" }, "tool": { "type": "object", "additionalProperties": false, "required": [ "id", "name", "version" ], "properties": { "id": { "type": "string" }, "name": { "type": "string" }, "version": { "type": "string" } } }, "computedAt": { "type": "string", "format": "date-time" }, "inputs": { "type": "object", "description": "Normalised inputs actually used, including applied defaults." }, "summary": { "type": "string" }, "result": { "type": "object", "additionalProperties": false, "required": [ "statements", "tables", "tableEdges", "columns", "focus", "notes", "errors" ], "properties": { "statements": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "index", "kind", "target", "sources", "columnCount", "filterColumns" ], "properties": { "index": { "type": "integer" }, "kind": { "type": "string" }, "target": { "type": "string" }, "sources": { "type": "array", "items": { "type": "string" } }, "columnCount": { "type": "integer" }, "filterColumns": { "type": "array", "items": { "type": "string" } } } } }, "tables": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "name", "role", "upstream", "downstream", "columns" ], "properties": { "name": { "type": "string" }, "role": { "type": "string", "enum": [ "source", "model", "report" ] }, "upstream": { "type": "array", "items": { "type": "string" } }, "downstream": { "type": "array", "items": { "type": "string" } }, "columns": { "type": "array", "items": { "type": "string" } } } } }, "tableEdges": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "from", "to", "kinds" ], "properties": { "from": { "type": "string" }, "to": { "type": "string" }, "kinds": { "type": "array", "items": { "type": "string" } } } } }, "columns": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "target", "column", "transform", "sources", "expr", "via" ], "properties": { "target": { "type": "string" }, "column": { "type": "string" }, "transform": { "type": "string", "enum": [ "direct", "renamed", "derived", "aggregate", "constant", "star" ] }, "sources": { "type": "array", "items": { "type": "string" } }, "expr": { "type": "string" }, "via": { "type": "array", "items": { "type": "string" } } } } }, "focus": { "type": [ "object", "null" ], "additionalProperties": false, "required": [ "input", "type", "resolved", "upstream", "downstream" ], "properties": { "input": { "type": "string" }, "type": { "type": "string", "enum": [ "table", "column" ] }, "resolved": { "type": "string" }, "upstream": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "node", "depth", "transform" ], "properties": { "node": { "type": "string" }, "depth": { "type": "integer" }, "transform": { "type": "string" } } } }, "downstream": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "node", "depth", "transform" ], "properties": { "node": { "type": "string" }, "depth": { "type": "integer" }, "transform": { "type": "string" } } } } } }, "notes": { "type": "array", "items": { "type": "string" } }, "errors": { "type": "array", "items": { "type": "string" } } } }, "units": { "type": "object", "additionalProperties": { "type": "string" } }, "assumptions": { "type": "array", "items": { "type": "string" } }, "sources": { "type": "array", "items": { "type": "object", "additionalProperties": false, "required": [ "title" ], "properties": { "title": { "type": "string" }, "url": { "type": "string" }, "publisher": { "type": "string" }, "retrievedAt": { "type": "string" }, "note": { "type": "string" } } } }, "warnings": { "type": "array", "items": { "type": "string" } }, "presentation": { "type": "array", "items": { "type": "object" }, "description": "Display blocks used by the human page (same data)." }, "links": { "type": "object", "additionalProperties": false, "required": [ "human", "api", "openapi", "methodology" ], "properties": { "human": { "type": "string", "format": "uri" }, "api": { "type": "string", "format": "uri" }, "openapi": { "type": "string", "format": "uri" }, "methodology": { "type": "string", "format": "uri" } } } } } } } }