Introduction

A few weeks ago we wrote about upgrading the dbt AI stack with FalkorDB, vector embeddings, full-text search, and Bedrock reranking. It works well — semantic queries, concept-based discovery, multi-layer retrieval all in one system. But it also requires a running graph database, an embeddings pipeline, and cloud infrastructure before you get your first answer.

That’s the right setup for a production data platform. It’s too much for most situations.

graphify — built by Safi — is a tool we use constantly for the lighter cases. Point it at any folder of code or docs, and it extracts a knowledge graph you can query in plain English: graphify query "what calls this function?", graphify path "AuthModule" "Database", graphify explain "SwinTransformer". It runs locally, needs no API key for code projects, and integrates directly into Claude Code as a skill. On most repos it gives an AI agent a navigable map of the codebase instead of making it read files one at a time. Our Airflow repo is a good example — it's mostly Python and heavily functional, and graphify's call-graph queries make sense of it in a way that grepping DAG files never quite does.

dbt projects are the exception. Run graphify on a dbt project and you get a broken graph. The problem is Jinja2: dbt SQL isn’t standard SQL — it’s a template language where dependencies are expressed as {{ ref('stg_orders') }} and {{ source('raw', 'orders') }}. graphify’s extractor sees those as opaque macro calls, not edges. The lineage it produces is incomplete at best; usually it’s just a bag of disconnected SQL files with no meaningful relationships between them. dbt’s layer semantics — source → staging → intermediate → mart — don’t exist in the output at all. The result costs more tokens to query than just grepping the files.

Ask an agent “what breaks if I change stg_students?” and it has to read every .sql file in the project, reason over raw text, and hope it doesn’t miss an indirect dependency. On a 200-model project, that burns more tokens than it saves.

The thing is, dbt already computed the answer. Every dbt compile or dbt parse writes the fully-resolved DAG to target/manifest.json — every node, every edge, every upstream and downstream relationship, fully materialized. It’s sitting there in your target directory.

dbt-graphify bridges the two. It reads the manifest and reshapes it into exactly the format graphify expects: flat nodes with materialization type, file location, database, schema, column lists, and full lineage maps — layer semantics included. No database. No embeddings. No cloud account. Drop it into any dbt project, run one command, and graphify queries work the way they should have from the start.

The full AI stack is still the right answer when you need semantic search across business concepts. This is for everything else.

Check the repo: github.com/ponderedw/graphify-dbt

Getting it running

Two tools, one pipeline. dbt-graphify reads the manifest and builds the graph; graphify queries it.

Install both:

# Parse dbt manifests into a knowledge graph
pip install dbt-graphify

# With topology-based community detection (recommended for large projects)
pip install "dbt-graphify[clustering]"

# Query the graph and generate the HTML visualization
pip install graphifyy

Note the package name: it’s graphifyy (double-y) on PyPI, though the CLI command you actually run is graphify. We’d recommend uv tool install graphifyy or pipx install graphifyy over plain pip install — both isolate the package in their own environment, which avoids a ModuleNotFoundError that shows up with plain pip if your shell resolves a different Python than the one graphify was installed into.

pip install graphifyy only gets you the CLI. To make Claude Code (or Codex, Cursor, Gemini CLI, and a dozen others) reach for graphify automatically instead of reading files one at a time, run:

graphify install

This registers graphify as a skill in your AI assistant’s config — it nudges the assistant to run graphify query before it starts grepping through raw source. If you want it enforced rather than just suggested, graphify install --project --strict blocks the assistant’s first raw file read of a session and redirects it to the graph instead, then reverts to a soft nudge for the rest of the session. Platform-specific installs work the same way — graphify install --platform codex, --platform cursor, --platform gemini, and so on.

The [clustering] extra installs networkx and unlocks Louvain community detection — an algorithm that finds tightly-connected subgraphs by looking at the shape of the DAG rather than any labels. On the demo project in the repo, a 45-model educational analytics project, it finds four functional clusters without any configuration: student outcomes and at-risk analytics, activities and resource allocation, scheduling and teaching load, and courses and curriculum — purely from which models share upstream and downstream neighbors. On a real project with hundreds of models it surfaces domain boundaries that aren't explicit anywhere in the project structure. Without it, dbt-graphify falls back to grouping by layer — source, staging, intermediate, mart. Fully queryable either way.

Run dbt-graphify against your manifest:

# Auto-detect manifest.json in standard dbt target/ locations
dbt-graphify

# Explicit path (useful in monorepos)
dbt-graphify path/to/target/manifest.json

This writes everything to graphify-out/: a graph.json in NetworkX node-link format, a lineage.json with full ancestor and descendant maps, a GRAPH_REPORT.md summarizing the architecture, and a graph.html interactive D3 visualization. dbt-graphify does no querying — its only job is producing those files correctly from the manifest.

From there, graphify takes over. The repo includes a working demo — clone it, compile the dbt_project/ with dbt compile, run dbt-graphify, and you have a queryable educational analytics graph in under a minute:

graphify query "which models depend on stg_students?"
graphify query "what breaks if I change stg_enrollments?"
graphify path "stg_courses" "course_success_predictors"
graphify explain "int_department_analytics"

All BFS traversal on the local graph.json — no LLM calls, no API key, no network. The answers come from the graph structure dbt-graphify produced from the manifest.

One more thing: CLAUDE.md

Running dbt-graphify also writes a CLAUDE.md at your repo root — or updates its own section in-place if one already exists. The file contains a single instruction block:

## dbt lineage queries

This project has a graphify knowledge graph at graphify-out/graph.json.

Always use graphify commands to answer questions about models,
dependencies, lineage, or blast radius — do not grep SQL files.

    graphify explain "<model>"
    graphify path "<source>" "<target>"
    graphify query "<question>"

To rebuild: dbt-graphify path/to/manifest.json

Claude Code reads CLAUDE.md at session start and treats its contents as standing instructions. With this in place, a new session in the repo doesn’t need a prompt like “use graphify for lineage questions” — it just does. The conversation earlier in this post would have gone straight to graphify explain instead of spending several tool calls figuring out what to reach for.

The write is scoped: if CLAUDE.md already exists and doesn’t contain our block, dbt-graphify leaves it alone. If it does contain a block from a previous run or you manually added the following tags:

<!-- dbt-graphify start -->                                                                                                                       
   ...
<!-- dbt-graphify end --> 

only the manifest path gets updated. Nothing else in the file is touched.

What it’s actually useful for

The most immediate use case is blast-radius analysis. Before changing a staging model, you want to know how many downstream models it feeds and which marts it ultimately lands in. lineage.json has the full ancestor and descendant maps computed via BFS — read it directly in Python or ask graphify:

graphify query "what breaks if I change stg_enrollments?"

Traversal: BFS depth=2 | Start: ['stg_enrollments'] | 11 nodes found
NODE stg_enrollments [community=int_department_analytics]
NODE int_department_analytics [community=int_department_analytics]
NODE int_student_enrollment_history [community=stg_departments]
...
EDGE stg_enrollments --> int_department_analytics
EDGE int_department_analytics --> department_overview
EDGE int_student_enrollment_history --> student_academic_summary

On the demo project that returns the full chain: both intermediate models that read from stg_enrollments, and the three marts that depend on those intermediates. No scanning, no guessing.

The second case is onboarding. A new engineer joins a dbt project with 150 models and no documentation. GRAPH_REPORT.md gives them the full architecture in one file. graphify explain goes further, showing a single node’s full context:

graphify explain "int_department_analytics"

Node: int_department_analytics
  ID:        model.DbtEducationalDataProject.int_department_analytics
  Source:    models/intermediate/int_department_analytics.sql
  Type:      concept
  Community: stg_enrollments
  Degree:    7

Connections (7):
  <-- stg_departments [] []
  <-- stg_courses [] []
  <-- stg_students [] []
  <-- stg_enrollments [] []
  <-- stg_teachers [] []
  --> department_efficiency_report [] []
  --> teacher_performance_dashboard [] []

The third case is Claude Code integration. Install the graphify skill and lineage questions go through the graph instead of the files:

• “What feeds teacher_performance_dashboard?”

  

  

• “If I rename a column in stg_courses, what downstream models break?”

  

  One caveat: column-level lineage isn't in the graph yet — it's on the roadmap. Ask about a model and graphify answers from the graph; ask about a specific column and it still has to fall back to reading the file.

  

  

• “Which marts depend on int_department_analytics?”

  

• “Trace stg_attendance_records all the way to student_retention_analysis.”

Each becomes one traversal instead of Claude reading every model to find the joins. On a large project this typically cuts 70–90% of the tokens spent on lineage questions.

The fourth is CI. dbt-graphify has no side effects beyond writing graphify-out/, so it slots cleanly into any pipeline: compile the manifest, run dbt-graphify, commit the output. We added exactly this job to our own dbt CI, triggered on every merge to develop.

Under the hood

The package is two modules. parser.py reads the manifest; builder.py turns the parsed data into graphify’s expected format. No LLM calls anywhere — the whole pipeline is deterministic.

The parser’s main job is extracting three things from manifest.json: the node list (models, sources, macros), the edge list (parent_map and child_map, which dbt pre-computes during compilation), and per-node metadata — materialization from config.materialized, file path from original_file_path, columns from the columns dict, database and schema, and raw SQL from raw_code. Layer classification is a small inference function: stg_ prefix → staging, int_ prefix → intermediate, then fqn path segments as a tiebreaker, then mart as the default. If manifest.json is empty — dbt wasn’t compiled, just parsed — the tool falls back to graph_summary.json and reads the raw .sql files directly, using regex to extract {{ ref() }} and {{ source() }} calls and a best-effort SELECT-clause parser to recover column names.

The builder produces four outputs. graph.json is NetworkX node-link format — the exact structure nx.node_link_graph() expects, with all node attributes flat at the top level (graphify reads them via G.nodes[id].get("attr"), so anything nested inside a properties key is invisible to it). lineage.json holds two BFS maps, ancestors and descendants, computed from the adjacency lists the parser built. GRAPH_REPORT.md is a plain-text architecture summary with a per-layer node inventory and a blast-radius table sorted by downstream count. HTML generation is delegated entirely to graphify cluster-only via subprocess — no D3 code lives in this package.

Every node carries two different grouping properties that are easy to conflate. Layer is its position in the transformation pipeline — source, staging, intermediate, mart — always derived from the node’s name and fqn. It answers “where in the pipeline does this model sit?” Community is which functional domain the node belongs to, and it’s what the visualization colors by. A staging model and a mart can share the same community color if they serve the same business domain, even though they sit at opposite ends of the pipeline.

Community detection runs a priority chain: dbt groups first, then tags, then — if [clustering] is installed — Louvain modularity on the undirected projection of the DAG. Direction is dropped because Louvain works on undirected graphs, and structural proximity matters more than flow direction for grouping. Each community is named after its highest-degree node; if Louvain isn’t available in the installed networkx version, it falls back to greedy_modularity_communities. Without [clustering] at all, community falls back to layer — the two properties collapse to the same value, and the visualization colors by pipeline stage instead of domain.

The only required dependency is the Python standard library. PyYAML is optional, networkx is optional. The package has no opinion about which dbt adapter you use.

Wrapping up

dbt-graphify is a small, deterministic bridge: it takes the DAG dbt already computed and reshapes it into something graphify — and by extension, Claude Code — can query directly. No graph database, no embeddings, no cloud account, just a manifest and one CLI command. For blast-radius checks, onboarding, and day-to-day lineage questions in an agent session, that’s enough.

It’s not a replacement for the full AI stack — column-level lineage isn’t there yet, and semantic search across business concepts still needs embeddings and a real graph database. But most lineage questions aren’t semantic, they’re structural, and dbt already has the answer sitting in target/manifest.json. This just stops throwing that away.

Try it on your own project, or start with the demo in the repo: github.com/ponderedw/graphify-dbt. Issues and PRs welcome.

At Ponder, we build the Ponder DWE (Data Warehouse for Education) — data infrastructure that automates, cleans, and standardizes data from across instructional and operational platforms for K-12 schools and networks, shaped by our partnership with leaders in education reform such as Uncommon Schools. dbt sits at the center of that stack, and over the past few years we’ve run it in both flavors: Core, Cloud, and — this is the actual point of this article — both at once, at different layers.

Our main takeaway: the comparison table itself is the mistake.

A Quick Introduction: dbt, Core, and Cloud

For those outside the bubble: dbt is the de facto standard for the T in ELT. You write SQL models with Jinja templating, dbt compiles them into a dependency graph and runs them against your warehouse in the right order, with tests and documentation attached.

It comes in two forms. dbt Core is the open-source engine — a CLI you run yourself, wherever and however you want. dbt Cloud is the commercial platform around that engine: web IDE, job scheduler, CI, metadata catalog, all managed for you, all billed accordingly.

Same models run on both. Keep that fact in mind — it’s the load-bearing fact of this whole article.

The Framing Problem

“Core or Cloud” assumes dbt is one decision. It isn’t. It’s at least five separate decisions — development environment, execution, CI, orchestration, metadata — and nothing forces you to answer them all the same way. The vendors’ framing collapses these layers into a single either/or because that’s how you sell a platform. Your actual job is to decompose them back.

Here’s how that decomposition went for us in practice.

Development: Cloud Won, for a Mundane Reason

We started on Core, like most teams that come up through the engineering side. Then we moved development to Cloud, and the reason was mundane: the IDE. Not the scheduler, not the enterprise security slide — the IDE. Autocomplete over your actual project, instant compile previews, analysts contributing without a local Python environment fight.

Whatever your opinion on web IDEs, the onboarding difference is real. In an education data context, where the people who best understand the data are not always the people most comfortable with a terminal, that difference compounds. Pretending otherwise is how “team maturity” becomes a euphemism for “we enjoy gatekeeping.”

That said, the Cloud IDE may not be the final word here. We’ve been standardizing development environments across the stack with Coder — reproducible, isolated workspaces that spin up in minutes (we wrote about the Airflow version here). A dbt workspace built the same way could give us the onboarding benefits without the platform dependency — we’re evaluating exactly that, and will tell you about this decision shortly. Maybe that next solution is what we picked for the Ponder DWE package. Stay tuned.

Execution: Split Environments Are Worse Than Either Environment

Moving development to Cloud while running production on Core taught us the first lesson quickly: two runtimes, two behaviors, one permanent source of drift. This is the class of bug where a model works in the IDE and fails at 3 AM for reasons neither set of logs will explain.

So execution moved to Cloud too. Jobs run there now. One runtime, one behavior. Fine.

But that’s where the migration stopped — and the stopping points are the actual content of this article.

CI: Stayed on Core, Deliberately

Cloud’s CI is convenient precisely because it’s opinionated — and opinionated tools are great right until your workflow disagrees with the opinion. Ours does: selective builds based on what actually changed, custom data quality gates, environment-specific checks before merge.

A concrete example: we’ve written before about a setup where dbt handles both ingestion and transformation — Trino-based ingestion in one project, warehouse transformations in another, wired together as packages, with CI that tests modified models across both using defer state. You can read the full walkthrough here: Trino + dbt: Simplifying ELT with Pure SQL.

The relevant constraint: dbt can’t run two adapters in a single project. So CI has to be elastic — two projects, two profiles, ordered execution, shared state between them. In Core, that’s a shell script. In Cloud, that shape simply doesn’t exist. And bending a managed product against its own opinions is the most expensive kind of engineering there is.

One related warning from the same story: custom Core solutions can surface odd behavior when they meet Cloud, because Cloud implements some things slightly differently. After we integrated that Trino setup, the Cloud IDE started demanding dbt deps on every refresh — press F5, rerun deps, something strange in how it handles local packages. Fairness requires the next sentence, though: dbt Cloud support actually works. Real humans, real answers — they helped us through exactly this kind of problem, which is not something we can say about every SaaS vendor in the data stack.

Orchestration: Keep It Outside

Real pipelines don’t start and end with dbt. There’s ingestion upstream, reporting and reverse ETL downstream, sensors, retries that need context about what failed and why. Cloud’s scheduler knows about dbt jobs; your orchestrator knows about your platform.

Triggering Cloud jobs from Airflow via API is one operator. Wiring non-dbt dependencies around Cloud’s scheduler is a distributed system built out of webhooks and hope. We kept orchestration in Airflow and let dbt Cloud be an execution layer, not the brain.

Two Footnotes That Never Make the Comparison Tables

The catalog. Column-level lineage out of the box is quietly one of the strongest arguments for Cloud — the feature nobody asks about during evaluation and everybody uses six months in. When a district asks why a number on a dashboard changed, tracing a column from the report back through every transformation to the source system is not a nice-to-have.

Networking. SSH tunnels aren’t supported for all adapters. We couldn’t connect a private Trino cluster a few months ago — maybe it’s fixed by now, but check your networking story before you commit. “Enterprise-grade integrations” and “can reach your database” are different claims.

The Fine Print: Keep a dbt Core Escape Hatch

Even if you go all-in on Cloud, keep a backup image with dbt Core and your run logic. Remember the load-bearing fact — Core and Cloud run the same code from the same repo, which means a container with dbt Core is a disaster recovery plan that costs you one Dockerfile.

When dbt Cloud is down, “fully managed” means fully waiting. When it’s down and you have that image, it means running dbt build from a container until it’s back. The whole option costs nothing. Not having it costs an incident review.

Conclusion: It’s Not Maturity. It’s Decomposition.

So where did we land at Ponder? Development and execution in Cloud. CI in Core. Orchestration outside both, in Airflow. Catalog from Cloud, disaster recovery from Core.

That’s not indecision — that’s the answer. The maturity question was never “which tool can your team handle.” It’s whether you know which layers of your platform you actually need to own, and have the discipline to rent the rest.

The comparison tables get one thing right: it’s not about the tool. They just stop one step too early. It’s not about maturity either. It’s about decomposition.

A few months ago, we published Graph-Powered AI Chat for dbt — and the response genuinely surprised us. The idea of treating a dbt project as a knowledge graph, loading it into FalkorDB, and chatting with it in natural language resonated with a lot of people. We ended up talking about it at the FalkorDB community event and at SQLBits 2026 in Newport, and the conversations there only convinced us there was more to explore.

So we explored.

The original pipeline was already powerful — you could ask complex dependency questions, traverse the graph recursively, and get intelligent answers about your dbt project’s structure. But it had a blind spot: it understood relationships, not meaning. If you asked something like “find models related to revenue reporting,” the graph could only help if you happened to traverse the right edges. It couldn’t understand what a model was about.

That’s what this post fixes.

All the changes described here are added to the same repository — dbt-kg — so you can follow along, clone it, and have everything in one place. We’re enhancing the same pipeline with three upgrades that, together, make the system dramatically more capable:

• Semantic search via embeddings — we’ll embed dbt node descriptions directly into FalkorDB, so you can now search by meaning, not just by graph structure. Dependency traversal and semantic similarity, working side by side.

• Full-text search — fast, exact-match keyword search that complements semantic search in ways that matter in practice. We’ll show you why you want both.

• Reranking — once you have multiple retrieval signals, how do you decide what’s actually most relevant? We’ll explain what reranking is, why it matters, and how to wire it in.

By the end, you’ll have a retrieval pipeline that can find what you’re looking for whether you know the exact name, a related concept, or just a vague description. Let’s build it.

Semantic Search via Embeddings

The original pipeline stored your dbt project as a graph — nodes for models, sources, seeds, and snapshots, connected by edges representing dependencies, tests, and macro usage. That structure is fantastic for traversal questions: what depends on what, blast radius analysis, lineage chains. But the nodes themselves were just identifiers. They had no sense of what they meant.

To understand the fix, it helps to understand RAG — Retrieval-Augmented Generation. The core idea is simple: instead of asking an LLM to answer from memory alone, you first retrieve relevant context from your data, then augment the prompt with that context before generating an answer. This is what makes AI systems actually useful on private, domain-specific data — the LLM doesn’t need to have seen your dbt project during training; it just needs the right context handed to it at query time. The retrieval step is usually powered by vector similarity: you embed both your documents and the user’s question into the same vector space, then find the documents whose embeddings are closest to the question.

That’s exactly what we’ve added here. For each node, we take everything meaningful about it — its name, resource type, schema, materialization, description, and all its column descriptions — concatenate it into a single piece of text, embed it, and store that embedding directly as an attribute on the node in FalkorDB. No separate vector store, no extra nodes, no parallel data structure to keep in sync. The embedding lives on the same node as everything else.

This happens automatically when you upload your manifest.json and catalog.json — the upload endpoint calls build_node_embeddings() right after the graph is loaded, so by the time you open the chat, every node is already searchable by meaning.

At query time, the FalkorDBNodeRetriever takes your question, embeds it using the same model, and runs a KNN vector search across all node labels simultaneously. The results are ranked by cosine similarity — no Cypher required, no need to know any model names upfront.

This is the key shift. Questions like “which models track early warning signals for students at academic risk?” or “find me everything related to teacher workload and teaching effectiveness” now work — even if none of the model names contain those words. The graph knows the structure of your project; the embeddings know what it’s about. Together, they cover both.

And the two modes compose naturally. You can ask “find the model that tracks at-risk students, then show me all the models it depends on” — the agent uses semantic search to locate the right node, then switches to graph traversal to answer the structural follow-up. That combination is where this really shines.

One honest caveat, though. Traditional RAG pipelines deal with large documents — think long PDFs, sprawling wiki pages, dense technical docs — and a standard best practice there is to chunk them: split each document into smaller pieces, embed each chunk separately, and retrieve only the most relevant chunks at query time. This gives you fine-grained retrieval and avoids stuffing irrelevant content into the LLM’s context.

In our graph setup, we don’t have that luxury in the same way. Each node is a single unit — one model, one embedding. If a node’s combined text (description + all column descriptions) is very long and covers multiple distinct concepts, you can’t retrieve just the relevant part of it; you get the whole node or nothing. You could split a node’s content into multiple embeddings stored as additional attributes, but that quickly becomes unintuitive — you’d be fighting the grain of the graph model rather than working with it. For most dbt projects, where node descriptions are concise and focused, this isn’t a problem in practice. But it’s worth keeping in mind if you’re working with unusually rich metadata or very wide models with dozens of heavily documented columns.

One practical note on embedding providers: the system automatically picks the right model based on your LLM_MODEL_ID prefix — Titan for Bedrock or Anthropic, text-embedding-3-small for OpenAI — so no extra configuration needed.

Full-Text Search

Full-text search has a long history in software engineering. Elasticsearch popularized it at scale, and the core algorithm behind most implementations — BM25 — has been the de facto standard for keyword relevance ranking for decades. The idea is straightforward: score documents by how often the search terms appear in them, adjusted for document length and how rare those terms are across the entire corpus. It’s fast, battle-tested, and remarkably effective when you know what you’re looking for.

FalkorDB takes a different approach. Its full-text indexing is powered by RediSearch and uses TF-IDF scoring rather than BM25 — similar in spirit, but simpler: it rewards nodes where the search term appears frequently relative to the total number of terms, without BM25’s additional length normalization and term saturation adjustments. For our use case — searching across concise dbt node descriptions rather than long documents — this is a perfectly natural fit. The nodes aren’t essays; the scoring difference between TF-IDF and BM25 rarely matters at this scale.

What does matter is the query expressiveness. And RediSearch delivers. When you upload your manifest, build_fulltext_index() creates a full-text index across six properties on every node — name, description, schema, alias, materialized, and resource_type — for all four label types: Model, Source, Seed, and Snapshot. The FalkorDBFulltextRetriever then queries that index at runtime, deduplicates results across labels, sorts by TF-IDF score, and returns the top matches.

The query syntax is what makes this tool genuinely useful day-to-day:

• Prefix search — enroll* matches “enrollment”, “enrollments”, “enrollee”

• Boolean operators — graduation AND NOT exited, teacher AND workload, risk OR warning

• Fuzzy matching — handles typos via Levenshtein distance, so performence still finds the right models

• Stemming — searching “running” finds “run” and “runs” automatically

Semantic search is powerful, but it has a blind spot: sometimes you know exactly what you’re looking for. If you need every model that mentions “GPA”, or you want to find anything referencing “attendance” and “retention” together, embedding similarity is overkill — and can actually hurt you by surfacing loosely related models when what you wanted was a precise match.

Full-text search is the right tool for that. The agent knows when to reach for it: use full-text when the question contains specific terms, name fragments, or boolean logic; use semantic search when the question is about a concept or business meaning.

The two tools complement each other cleanly — and as you’ll see next, there’s a third signal we layer on top of semantic search to make sure the most relevant results actually rise to the top.

Reranking

Vector similarity search is good, but it has a subtle problem. When you embed a query and find the closest nodes by cosine distance, you’re measuring how similar two vectors are in a high-dimensional space — and that’s a proxy for relevance, not relevance itself. Embedding models are trained to capture general semantic similarity, not to deeply reason about which result actually best answers a specific question. The result is that KNN search often gets the ballpark right but the ordering wrong. The most relevant node might be ranked 4th or 7th, buried under results that are thematically nearby but not quite what you needed.

This is where reranking comes in.

A reranker is a separate model that takes the query and a set of candidate documents — already retrieved by your vector search — and scores each candidate specifically against the query, paying attention to the full meaning of both. Unlike embedding models, which encode the query and each document independently into vectors and then compare them, rerankers look at the query and a candidate together in a single pass. This cross-attention mechanism is much more expensive computationally, which is why you don’t use it to search across your entire dataset — you use it to re-score a small shortlist that vector search already pulled out. The typical pattern is: retrieve 20–50 candidates with KNN, then rerank them and keep the top 5.

We use Amazon Bedrock’s Cohere Rerank v3.5 for this. After FalkorDBNodeRetriever retrieves KNN candidates from FalkorDB, the _rerank() function sends them to Cohere’s reranking model along with the original query, gets back a new relevance ordering, and returns the reranked top results alongside all the original KNN candidates — so you can see exactly what the reranker chose from and why the ordering changed.

To make this tangible, consider a query like “which models track students who are falling behind academically?” Vector search might return 20 candidates that all vaguely relate to students or academics. But the top KNN result by cosine similarity might be a model about GPA trends in general — semantically close to “academic”, but not really about falling behind or at-risk detection. The reranker reads the full query and each candidate together and recognizes that a model specifically described as tracking early warning indicators for academic risk is a much better match — and promotes it accordingly.

In the Streamlit UI, you can actually watch this happen. We surface both layers: the full set of KNN candidates with their cosine similarity scores, and then the reranked top results with their Cohere relevance scores. Comparing the two orderings is often the most instructive part — you can see precisely where vector similarity and true relevance diverge.

One practical note: reranking only applies to semantic search, not to full-text search. Full-text results are already ranked by TF-IDF term frequency, which is a direct signal of keyword relevance rather than a vector approximation — reranking on top of that would add latency without meaningfully improving precision.

To enable it, just add two environment variables to your .env:

BEDROCK_RERANKER_MODEL_ID=cohere.rerank-v3-5:0
BEDROCK_RERANKER_MODEL_ARN=arn:aws:bedrock:{region}::foundation-model/cohere.rerank-v3-5:0

If Bedrock credentials aren’t configured or the reranker call fails for any reason, the system falls back silently to the original KNN ordering — so it degrades gracefully rather than breaking your pipeline.

Putting It All Together

Let’s zoom out and look at what we’ve built.

The original pipeline gave you a graph. You could traverse it, query it with Cypher, and ask complex structural questions about your dbt project that no other tool could answer. That was already powerful — and judging by the response to the first post, it resonated with a lot of people working on real, complicated dbt projects.

This post added three layers on top of that foundation, and each one fills a gap the previous couldn’t cover:

Graph traversal answers structural questions — lineage, blast radius, dependency chains, test coverage. It speaks the language of relationships. Ask it “what breaks if I drop stg_students?” and it will tell you exactly.

Full-text search answers keyword questions — precise, fast, exact-match retrieval across model names, descriptions, schemas, and more. When you know the term you’re looking for, this is the right tool. Boolean operators, prefix matching, fuzzy tolerance for typos — it handles the cases where you want precision, not approximation.

Semantic search with reranking answers meaning questions — business concepts, vague descriptions, things you can’t quite name but know when you see them. The embeddings live directly on the graph nodes, so there’s no separate vector store to maintain. The reranker then ensures the most genuinely relevant results rise to the top, not just the ones that happen to be closest in vector space.

Together, these three tools give the LangGraph agent a complete retrieval toolkit. A question about lineage goes to the graph. A question about a specific term goes to full-text. A question about a concept goes to semantic search, with reranking to sharpen the results. And the most interesting questions — “find the model that tracks at-risk students, then show me everything it depends on” — combine all three in a single conversation turn.

All of this lives in the same dbt-kg repository. Upload your manifest.json and catalog.json, and the pipeline builds the graph, creates the embeddings, and sets up the full-text index automatically. Everything is ready by the time you open the chat.

That said, there are two things we haven’t tackled yet — and they’re worth being honest about. The system understands your dbt project’s structure and metadata deeply, but it doesn’t query your actual data. You can ask which models track student attendance patterns, but you can’t yet ask what the numbers look like right now. And while dbt is the transformation layer, many teams also have a semantic layer sitting on top — tools like dbt Semantic Layer, Cube, or Looker — that define business metrics, dimensions, and how data should be interpreted by end users. Integrating that layer into the same conversational interface would be the natural next step.

So that’s what we’re thinking about next. Stay tuned.

If you’ve ever onboarded a new data engineer onto an Airflow project, you know the drill. Half a day disappears into Python version mismatches, missing system dependencies, conflicting virtual environments, and the classic “it works on my machine” shrug. Local Airflow development is painful — and it doesn’t have to be.

This post walks through how we use Coder to give every developer a fully isolated, reproducible Airflow environment that spins up in minutes. We’ll use the airflow-mcp-plugins repo as our example — a project that adds a conversational AI interface directly into the Airflow UI – https://newsletter.ponder.co/p/airflow-chat-conversational-ai-built .

The Problem with Local Airflow

Just yesterday I ran into someone who told me they’d been a Dagster advocate for years — not because of the technology itself, but because local development just felt easier there. Airflow had a reputation for being painful to run locally, and that reputation stuck even as Airflow improved significantly over time. I love Airflow, and it bothered me that something as fixable as local dev experience was pushing people toward other tools. This post is partly my answer to that.

Running Airflow locally sounds simple until it isn’t. The typical setup involves Docker Compose with multiple services — webserver, scheduler, worker, Redis, Postgres — all needing to talk to each other, share volumes, and pick up your DAG files. On top of that:

• Everyone needs the same Python version and the same packages

• Port conflicts are common if you’re running multiple projects

• When you stop and restart, state can get corrupted

• New team members spend hours just getting to a working baseline

The usual answer is “just use Docker Compose directly.” But that still leaves every developer managing their own environment, their own .env files, their own credentials, and their own debugging when something breaks.

And then there’s the invisible problem: standardization drift. If you figure out a better way to set up the local environment — a smarter Docker Compose config, a new environment variable that everyone needs, a helper script that saves thirty minutes — how do you roll that out? You send an email. Which probably won’t be read. Or will be read, bookmarked, and forgotten. Six months later you discover half the team is still running the old setup because there was never a centralized, enforced way to define what “local development” actually means.

This challenge becomes even more apparent when supporting a broader internal development team. At Uncommon Schools, we work with engineers and developers across the data stack — from Airflow pipelines to dbt transformations and analytics tooling — with different roles, workflows, and levels of infrastructure familiarity. Before introducing Coder-driven workspaces, keeping everyone’s local setup aligned with the latest infrastructure changes was a recurring challenge.

The goal was to move away from onboarding docs, Slack messages, and tribal knowledge toward a development environment that is itself the source of truth: a reproducible workspace definition that every developer can spin up and keep in sync.

It gets worse when your stack grows beyond a simple Docker Compose. What if some components need to run directly on the laptop — a local Python process, a dbt command, a custom script that talks to both your local Airflow and a cloud service? Now you’re coordinating Docker networks, local processes, environment variables, and credentials across multiple tools, and the “just run this command” onboarding doc is already three pages long and six months out of date.

The Solution: Coder

Coder is an open-source platform for self-hosted remote development environments. Think of it as a control plane for developer workspaces — similar in spirit to Dev Containers, but more flexible. Where Dev Containers are tied to a specific editor and a specific container, Coder lets you define full environments that can run not just containers but also VMs or even bare-metal OS instances. You get a browser-based IDE, a terminal, and proxied access to any app running inside the workspace, all through a single URL.

You define a workspace as code using Terraform, and Coder provisions it on demand — locally or in the cloud. The key insight is that a “workspace” is just a compute resource (a Docker container, a VM, a Kubernetes pod) with a Coder agent running inside it. The agent connects back to the Coder server, which proxies your IDE, your apps, and your terminal outward. Every developer gets their own isolated environment, defined by the same Terraform template, with their own credentials injected at workspace creation time.

Coder supports a wide range of workspace types — AWS EC2 instances, Google Cloud VMs, Kubernetes pods, and more. For this tutorial we’ll use Docker, which is the simplest setup for local development and needs nothing beyond Docker Desktop on your Mac.

For Airflow development this is a perfect fit. Each workspace gets its own Docker daemon (Docker-in-Docker), its own Postgres, its own Redis, and its own Airflow stack — all isolated inside a single container that Coder manages. Want to change the default environment for the whole team? Update the Terraform template. Everyone’s next workspace picks it up automatically — no email required.

Running Coder Locally on Mac

First, install Coder:

curl -L https://coder.com/install.sh | sh

Then start the server. You need to pass both an access URL and a wildcard access URL — the wildcard is what allows Coder to proxy your Airflow UI through a subdomain:

coder server --access-url http://localhost:3000 --wildcard-access-url “*.localhost:3000”

If you’re using our repository, we’ve wrapped this in a just command:

just start_coder

Once it starts, open http://localhost:3000 in your browser. You’ll be prompted to create an admin account on the first run.

Creating Your First Workspace

After logging in:

Go to Templates and create a new template — paste in the coder_template.tf from the repository, then click on Build and Publish

Click Create Workspace

Fill in the parameters — your Git name, your Anthropic/OpenAI/AWS API key depending on which LLM backend you want to use

Hit Create and watch the build logs

The first time takes a few minutes because Docker images need to be pulled. On subsequent starts it’s fast — the Docker image cache is persisted in a named volume so nothing gets re-downloaded.

Once the workspace is running you’ll see two buttons: code-server (VS Code in the browser) and Airflow.

Click Airflow and you’ll land directly on the Airflow UI at http://airflow--yourusername--coder.localhost:3000/. Log in with airflow / airflow.

How Terraform Works in Coder

Every Coder workspace is defined by a Terraform template. When you create a workspace, Coder runs terraform apply with your parameters, which provisions the Docker container, volumes, and agent. When you stop the workspace, it runs terraform destroy on the container (but keeps the volumes). When you restart, it re-applies — same volumes, same data, new container.

The template we use here does a few things:

• Declares input parameters (coder_parameter) that become form fields in the UI — your API keys, your Git name, your LLM model choice

• Creates Docker volumes for /home/coder (your code and env files) and /var/lib/docker (the inner Docker image cache)

• Spins up a privileged workspace container with its own Docker daemon inside (DinD)

• Runs a startup script that installs Docker Engine, clones the repo, writes airflow-plugin.env from your parameters, and launches just airflow

• Registers a coder_app that tells Coder where to proxy the Airflow UI

For private repositories, you add a gitlab_token or github_token parameter and embed it in the clone URL:

git clone “https://oauth2:${TOKEN}@gitlab.com/yourorg/yourrepo.git”

The token gets injected at workspace creation time and never lives anywhere outside the container.

Walking Through the Template

Let’s look at what the template actually does, piece by piece.

Parameters are declared at the top as coder_parameter blocks. Each one becomes a form field when a developer creates a workspace. Here we collect the Git name, the LLM model ID, and whichever API keys are needed depending on whether you’re using Anthropic, OpenAI, or Amazon Bedrock. Parameters can have defaults, descriptions, and validation — they’re the clean interface between “what Coder knows” and “what the developer provides.”

Volumes come next. We create two named Docker volumes per workspace. home_volume mounts at /home/coder and holds everything the developer touches — the cloned repo, the generated .env file, any local changes. docker_data_volume mounts at /var/lib/docker inside the workspace container, which is where the inner Docker daemon stores its image cache, container state, and named volumes (including Postgres data). Both volumes survive workspace stops and restarts, so your data is never lost between sessions.

The workspace container is a codercom/enterprise-base:ubuntu image running with privileged = true. The privileged flag is what makes Docker-in-Docker possible — it allows the container to run its own kernel namespaces and mount its own filesystems. The entrypoint creates the repo directory, fixes ownership, and then hands off to the Coder agent init script. Notice the replace() call that rewrites localhost to host.docker.internal — this ensures the agent can phone home to the Coder server running on your Mac, not try to reach itself.

The startup script runs once the agent is up. It installs Docker Engine from the official Docker apt repository (not the system package, which is often outdated), adds the coder user to the docker group, starts the daemon, clones the repo, and then writes airflow-plugin.env by copying the test env file and injecting each parameter value using a delete-then-append pattern:

sed -i ‘/^ANTHROPIC_API_KEY=/d’ airflow-plugin.env
echo “ANTHROPIC_API_KEY=${data.coder_parameter.anthropic_api_key.value}” >> airflow-plugin.env

This is more reliable than sed replace because it works regardless of whether the key already exists or is blank in the source file. We also add a blank line after the copy step — without it, if the source file has no trailing newline, your first appended key gets concatenated onto the last existing line.

Finally, the script runs exec sg docker “cd $HOME/$REPO_DIR && just airflow”. The sg docker part activates the docker group membership immediately without needing a new login session — a common gotcha with Docker-in-Docker setups where usermod -aG docker doesn’t take effect until the next login.

The coder_app resource registers the Airflow UI with Coder. With subdomain = true, Coder proxies it through http://airflow--yourusername--coder.localhost:3000/ — the wildcard DNS entry you configured when starting the server. The URL points to localhost:8088 because the repo’s docker-compose maps the webserver’s internal port 8080 to 8088 on the workspace container’s network.

One thing worth noting: this setup runs the full Docker daemon inside every workspace container, and each workspace pulls and caches its own images. For a small team of engineers it’s perfectly fine. For a setup with hundreds of developers, you’d want to think about shared image registries, resource limits, and a proper Coder deployment on a server rather than everyone’s laptop. We’ll cover exactly that in a future post — different workspace architectures, how to right-size environments for different roles, and how to deploy Coder in production so your whole team can use it without running anything locally at all.

Workspace Architecture

Here’s what’s actually running when your workspace is up:

Mac (host)
└── Docker Desktop
    └── coder-yourusername-myworkspace (workspace container, privileged)
        └── Inner Docker daemon (/var/lib/docker on persistent volume)
            ├── airflow-webserver  → exposed on workspace's localhost:8088
            ├── airflow-scheduler
            ├── airflow-worker
            ├── postgres           → state persists across restarts
            └── redis:7

The workspace container runs with privileged = true, which lets it run its own Docker daemon. All the Airflow containers are siblings inside that inner daemon — they share a bridge network and can reach each other by service name. From the outside, Coder’s agent proxies localhost:8088 on the workspace container through the wildcard subdomain to your browser.

Because /var/lib/docker is a named Docker volume, the inner daemon’s image cache survives workspace restarts. The first cold start pulls ~1GB of Airflow images. Every restart after that is fast.

Because /home/coder is also a named volume, your cloned repo, your airflow-plugin.env, and any local changes persist between restarts too.

What’s Coming Next

This is the first post in a series on managing data engineering development environments with Coder. Coming up:

• dbt workspaces — same pattern, different stack, with dbt docs served as a Coder app

• AI-assisted development with LiteLLM — routing Claude Code through a self-hosted LiteLLM proxy so your API keys stay centralized and usage is tracked per developer

• Multiple workspace types — how to run lightweight workspaces for analysts alongside heavy workspaces for engineers, all from the same Coder deployment

• Production deployment — moving from coder server on your laptop to a proper deployment behind a reverse proxy, with wildcard DNS, persistent Postgres, and workspace autostop policies to keep costs down

The full template is in the repository. If you run into issues, the most common ones are covered in the troubleshooting section of the README.

Rebuilding the same semantic layer over and over across different BI tools has always felt unnecessarily painful. Most dashboards still reach directly into databases, each with their own idea of what a “metric” is, so you end up maintaining the same logic in three or four places.

Late last year, out of frustration, we built dbt-to-cube to see whether dbt, Cube.js, and Superset could be wired together into a single automated flow. It worked: define metrics once in dbt, let a small tool translate them to Cube.js, and push them into your BI tool of choice—giving you a unified semantic layer across Superset, Tableau, or PowerBI without manual duplication.

But as we started relying heavily on this code-first workflow at Ponder Labs, a glaring logical gap emerged.

It’s great that downstream BI tools can query these unified metrics. But what about using those metrics inside dbt itself? If I wanted to build an aggregated reporting mart or feed a downstream process within my dbt pipeline, I still had to manually rewrite the SUM() or AVG() aggregations in my SQL, breaking the “define it once” rule. I wanted the metrics defined for Cube to be queryable natively in my dbt DAG.

That’s exactly why I built the next piece of the puzzle: dbt_simple_metrics.

The Solution: Metrics as Code Inside dbt

dbt_simple_metrics is a lightweight dbt package that lets you generate aggregated SQL from metric definitions sitting directly inside your model YAML files. Instead of navigating a heavy, standalone semantic layer setup, your columns and metrics live together, and you reuse them across your marts with a single macro call.

In our simplified format, we keep things incredibly close to the data. You define the dimensions under columns, and your measures under config.meta.metrics. It looks like this:

version: 2
models:
  - name: course_performance_summary
    columns:
      - name: course_name
      - name: academic_year
      - name: total_enrollments
    config:
      meta:
        metrics:
          total_course_enrollments:
            type: sum
            label: “Total Course Enrollments”
            sql: ${total_enrollments}

Now, instead of manually writing aggregations in your downstream models, you simply call the macro. The package validates the dimensions and metrics at compile time and generates the proper GROUP BY SQL:

{{ config(materialized=’table’) }}
{{ dbt_simple_metrics.metrics(
    ‘course_performance_summary’,
    [’course_name’, ‘academic_year’],
    [’total_course_enrollments’]
) }}

If you misspell a dimension or an unknown metric is requested, you get a clear compile-time error rather than a runtime failure.

The Elephant in the Room: MetricFlow

If you are immersed in the dbt ecosystem, you are likely wondering: Doesn’t dbt already have a Semantic Layer with MetricFlow?

It does. In fact, if you look at the semantic_models.yml file in the dbt_simple_metrics repository, you’ll see we actually included the MetricFlow definitions. We added them specifically so you can still see your metrics natively in dbt docs and the dbt Cloud catalog.

However, that file is an example of the legacy MetricFlow format. And honestly, it highlights exactly why we built our package. The old spec forces you to define a top-level semantic_models block entirely separate from your actual data models, with verbose, disconnected metrics configurations that look like this:

# The Legacy MetricFlow Format
metrics:
  - name: total_semester_headcount
    type: simple
    type_params:
      measure: total_unique_students
  - name: deans_list_rate
    type: ratio
    type_params:
      numerator: total_deans_list
      denominator: total_semester_headcount

It works for populating a catalog, but it feels detached from the code-first, model-centric workflow developers actually want.

Ahead of the Curve: Preparing for MetricFlow v2.0

Here is the good news: dbt Labs recognizes this friction.

With the upcoming release of dbt 1.12, they are stabilizing MetricFlow spec version 2.0. If you look at the differences, dbt is completely overhauling their approach to look... exactly like what we are already doing. They are abandoning the disconnected top-level wrappers and moving to embed semantic annotations (like semantic_model and simple metrics) directly under the models: block.

By adopting dbt_simple_metrics today, you aren’t fighting against the dbt ecosystem; you are skating to where the puck is going. You get the clean, embedded syntax that dbt is moving toward, with the immediate benefit of native dbt querying and automated syncing to Cube.js today.

And naturally, once dbt 1.12 becomes stable, we will be updating the package to ensure full, seamless compatibility with the final v2.0 spec.

Quickstart & Installation

To start using it in your dbt project and close the loop on your semantic layer, add the git repository and the specific revision to your packages.yml:

packages:
  - git: “https://github.com/ponderedw/dbt_simple_metrics.git”
    revision: v1.0.3

Run dbt deps, and you’re ready to start calling the macro in your marts.

At Ponder Labs, we are constantly pushing the boundaries of what code-first orchestration can do, and eliminating redundant metric definitions has been a massive win for our data engineering workflows.

Give the package a try, and let me know what you think on GitHub—PRs, feedback, and feature requests are always welcome!

Introduction

Data ingestion is rarely glamorous — but getting it right is everything. In a previous post, we explored batch data ingestion using Trino and dbt with a relational database as the source. But real-world data pipelines are messier: files land in S3 buckets from various sources, in various formats, and you need a reliable, manageable way to bring them into your Data Lake.

The natural question is: why not just create an external schema and table directly? You absolutely can — and in many cases, you should. But there are scenarios where that approach introduces more friction than it solves:

• Managing permissions between database users, IAM roles, and S3 buckets can become a tangled, error-prone mess

• Each warehouse (Redshift, BigQuery, Snowflake) handles external tables differently, making your pipeline harder to standardize

• Ingestion logic scattered across SQL scripts and cloud consoles is difficult to version, test, and maintain

• Onboarding new data sources requires touching infrastructure, not just code

We’re also familiar with the dbt-redshift external tables package, and while it’s a clever solution, it introduces a fundamental design concern: dbt shouldn’t be responsible for managing sources. Sources are meant to be stable, pre-existing references that dbt reads from — not objects that dbt creates or controls. When dbt starts owning source definitions, the line between infrastructure and transformation blurs, making pipelines harder to reason about, debug, and hand off to other teams. Models should run and transform; sources should simply exist.

Why Hive?

Apache Hive has been around since 2008, originally developed at Facebook to bring SQL-like querying to Hadoop’s distributed file system. At its core, Hive introduced a revolutionary idea: separate the metadata (where the data lives, how it’s structured, what format it’s in) from the data itself. This metadata layer — the Hive Metastore — became the backbone of the Hadoop ecosystem and eventually the de facto standard for Data Lake table management, adopted by virtually every major distributed query engine, including Spark, Presto, and later Trino.

Over time, more modern table formats emerged to address Hive’s limitations. Apache Iceberg, for example, was designed to handle the pain points that Hive struggled with at scale: full ACID transactions, reliable schema evolution, time travel, and efficient handling of billions of files. Iceberg is undeniably powerful — and if you’re building a large-scale, write-heavy lakehouse with complex update and delete patterns, it’s likely the better choice.

But here’s the thing: Hive’s simplicity is a feature, not a bug. Iceberg’s richness comes with real operational overhead — more configuration, more moving parts, and a steeper learning curve. When your goal is straightforward S3 file ingestion without complex upserts or time travel requirements, Iceberg can feel like bringing a sledgehammer to crack a nut. Hive’s loss in the feature war is our gain in the simplicity game.

This is where Hive shines. By leveraging Hive Metastore alongside AWS Glue Catalog as the metadata layer, we can define, manage, and transform S3 data in a consistent, code-first way — entirely within dbt. No console gymnastics, no permission rabbit holes. Sources remain untouched and independently managed, while dbt focuses on what it does best: running transformations.

That said, Hive isn’t a silver bullet:

• It adds an extra layer to your stack, which means more components to monitor and maintain

• Query performance on very large, unpartitioned datasets can lag behind native warehouse solutions

• It works best when you’re already operating within the AWS ecosystem — outside of it, the setup overhead may not be worth it

• Schema evolution (adding/changing columns) requires careful handling to avoid breaking downstream models

If you’re already working in AWS and want to manage your ingestion logic cleanly in dbt, Hive is a natural, battle-tested fit. Let’s walk through how we set it all up.

Hive-Trino Setup

With the “why” out of the way, let’s get our hands dirty. The setup boils down to two things: telling Trino how to talk to Hive via AWS Glue, and giving the right IAM permissions to make it all work.

1. Adding the Hive Catalog to Trino

Trino uses catalog configuration files to connect to different data sources. To wire up Hive with AWS Glue as the metastore, add the following catalog:

s3_source_files_hive: |
    connector.name=hive
    hive.metastore=glue
    hive.metastore.glue.aws-access-key=${aws_access_key_hive}
    hive.metastore.glue.aws-secret-key=${aws_secret_key_hive}
    hive.metastore.glue.region=us-east-1
    fs.native-s3.enabled=true
    s3.aws-access-key=${aws_access_key_hive}
    s3.aws-secret-key=${aws_secret_key_hive}
    s3.region=us-east-1
    s3.path-style-access=true

A few things worth noting here:

• hive.metastore=glue is the key line — it tells Trino to use AWS Glue Catalog instead of a self-hosted Hive Metastore, which saves you from spinning up and maintaining yet another service

• fs.native-s3.enabled=true enables Trino’s native S3 filesystem, which is more performant and reliable than the legacy Hadoop-based S3 driver

• Credentials are injected via environment variables, keeping secrets out of your config files — make sure these are properly set in your Trino deployment

2. IAM Policy

The catalog configuration alone isn’t enough — the IAM user referenced by those credentials needs the right permissions to both read/write from S3 and interact with the Glue Catalog. Attach the following policy:

{
    “Version”: “2012-10-17”,
    “Statement”: [
        {
            “Sid”: “S3DataReadWrite”,
            “Effect”: “Allow”,
            “Action”: [
                “s3:GetBucketLocation”,
                “s3:ListBucket”,
                “s3:GetObject”,
                “s3:PutObject”,
                “s3:DeleteObject”
            ],
            “Resource”: [
                “arn:aws:s3:::source-files-hive”,
                “arn:aws:s3:::source-files-hive/*”
            ]
        },
        {
            “Sid”: “GlueMetadataReadWrite”,
            “Effect”: “Allow”,
            “Action”: [
                “glue:GetDatabase”,
                “glue:GetDatabases”,
                “glue:CreateDatabase”,
                “glue:GetTable”,
                “glue:GetTables”,
                “glue:CreateTable”,
                “glue:UpdateTable”,
                “glue:DeleteTable”,
                “glue:GetPartition”,
                “glue:GetPartitions”,
                “glue:CreatePartition”,
                “glue:BatchCreatePartition”,
                “glue:DeletePartition”,
                “glue:BatchDeletePartition”
            ],
            “Resource”: [
                “arn:aws:glue:us-east-1:<account_id>:catalog”,
                “arn:aws:glue:us-east-1:<account_id>:database/s3_source_files_hive”,
                “arn:aws:glue:us-east-1:<account_id>:table/s3_source_files_hive/*”
            ]
        }
    ]
}

The policy is split into two clear responsibilities:

• S3DataReadWrite — grants access to the source-files-hive bucket, which is where your raw source files will be initially dropped. Trino needs read access to query them and write access to manage any intermediate outputs

• GlueMetadataReadWrite — grants Trino the ability to interact with Glue Catalog: reading and creating databases, managing tables, and handling partitions. Notice the resources are intentionally scoped to the s3_source_files_hive database only — following the principle of least privilege, we don’t hand out catalog-wide permissions

💡 Tip: Replace <account_id> with your actual AWS account ID before applying the policy. If you’re managing this via Terraform or CloudFormation, this is a good candidate for a variable substitution.

With the catalog registered and IAM permissions in place, Trino can now discover and query any Hive tables registered in your Glue Catalog. Next, let’s look at how dbt fits into the picture.

dbt Integration

Now that Trino can talk to Glue and read from S3, we need to bridge the last gap: making dbt aware of our Hive tables and using them as sources for transformations. The elegant part of this approach is that we don’t register tables manually — we create them dynamically using a dbt macro, triggered as a pre_hook before the model runs.

1. The Macro

Add the following macro to your dbt Trino project:

{% macro create_external_hive_table(model_name, schema_definition, s3_location) %}
    {% set target_relation = “s3_source_files_hive.s3_source_files_hive.” ~ model_name %}
    {% set sql %}
        DROP TABLE IF EXISTS {{ target_relation }};
        CREATE TABLE {{ target_relation }} (
            {{ schema_definition }}
        )
        WITH (
            external_location = ‘{{ s3_location }}’,
            format = ‘CSV’,
            skip_header_line_count = 1
        )
    {% endset %}
    {% if execute %}
        {% do run_query(sql) %}
        {{ log(”Created Hive Table: “ ~ target_relation, info=True) }}
    {% endif %}
    {{ return(”SELECT 1”) }}
{% endmacro %}

Let’s break down what’s happening here:

• target_relation — builds the fully qualified table name in the format catalog.database.table, pointing directly at our s3_source_files_hive catalog and database

• DROP TABLE IF EXISTS — ensures idempotency. Every time the model runs, the Hive table is recreated fresh. This is intentional: if the S3 files or schema change, the table definition stays in sync

• CREATE TABLE ... WITH (...) — this is a Hive external table definition. The key properties are:

  • external_location — the S3 path where the raw files live. Hive doesn’t move or copy the data, it simply points to it

  • format = ‘CSV’ — tells Hive how to parse the files. This can be swapped for ORC, PARQUET, JSON, etc. depending on your source files

  • skip_header_line_count = 1 — skips the first row of the CSV, assuming it contains column headers

• {% if execute %} — a dbt guard that prevents the macro from running during the parsing/compilation phase, only firing during actual execution

• run_query(sql) — executes the DDL statement against Trino

• return(”SELECT 1”) — macros used in pre_hook must return a valid SQL string. Returning SELECT 1 is a clean no-op that satisfies this requirement without side effects

2. The Model

With the macro in place, here’s how a dbt model using it looks:

{%- set hive_table = “table_name” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                column1 VARCHAR,
                column2 VARCHAR,
                column3 VARCHAR,
                column4 VARCHAR”,
            s3_location=”s3://source-files-hive/table_folder/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

Here’s the flow this model follows when dbt runs it:

1. pre_hook fires — before anything else, create_external_hive_table is called, dropping and recreating the Hive external table in Glue Catalog, pointed at the S3 location

2. The model runs — dbt executes the SELECT statement, reading directly from the freshly registered Hive table, which transparently reads the CSV files from S3

3. current_date as ingested_at — a small but important addition: we stamp every ingested row with the current date, giving us a simple audit trail of when the data was loaded

💡 Tip: Notice that the model name is defined once as a variable (hive_table) and reused in both the config block and the FROM clause. This keeps things DRY — if you rename the table, you change it in one place only.

This pattern keeps the separation of concerns intact: the macro handles infrastructure (table registration), and the model handles transformation. dbt does what dbt is supposed to do — and nothing more.

A Real-World Example: School Network Data Ingestion

To make this concrete, consider a large school network – say, 50+ schools across a district – already operating within AWS. Their data team is responsible for centralizing student, staff, and assessment data from a patchwork of source systems: a Student Information System, an HR platform, and several third-party assessment providers.

None of these systems share a common integration standard. What they do all share: a scheduled CSV export, dropped into S3.

Here’s what that might look like in practice:

s3://source-files-hive/

├── students/

├── staff/

└── assessments/

Each folder receives nightly file drops from the respective source system. The data team defines one dbt model per source, each invoking the same create_external_hive_table macro:

sql{%- set hive_table = “students” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                student_id VARCHAR,
                first_name VARCHAR,
                last_name VARCHAR,
                date_of_birth VARCHAR,
                enrollment_status VARCHAR,
                school_id VARCHAR”,
            s3_location=”s3://source-files-hive/students/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

sql{%- set hive_table = “staff” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                staff_id VARCHAR,
                full_name VARCHAR,
                role VARCHAR,
                department VARCHAR,
                school_id VARCHAR,
                employment_status VARCHAR”,
            s3_location=”s3://source-files-hive/staff/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

sql{%- set hive_table = “assessments” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                assessment_id VARCHAR,
                student_id VARCHAR,
                subject VARCHAR,
                score VARCHAR,
                max_score VARCHAR,
                assessment_date VARCHAR,
                school_id VARCHAR”,
            s3_location=”s3://source-files-hive/assessments/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

The result: three clean, independently managed source tables, registered automatically on each dbt run, with zero manual intervention. The data team can onboard a new source system – say, a library platform or a special education tracking tool – by adding a single model file and an S3 folder. No console access required, no infrastructure tickets, no cross-team dependency on a cloud engineer to wire up permissions.

Downstream, dbt models join students, staff, and assessments to produce analytics-ready datasets: attendance summaries, assessment score trends by school, staff-to-student ratios. The ingestion layer stays untouched – it simply exists, and the transformation layer builds on top of it.

Summary

Data ingestion doesn’t have to be complicated. In this post, we walked through a clean, practical approach to bringing raw S3 files into a queryable Data Lake — without overengineering the pipeline or blurring the lines between infrastructure and transformation.

Here’s what we covered:

• Why Hive over alternatives — Hive’s maturity and simplicity make it a natural fit for straightforward S3 ingestion. Where Iceberg wins on features, Hive wins on pragmatism. When you don’t need ACID transactions or time travel, there’s no reason to pay the complexity tax

• Why not external tables or dbt-redshift packages — external tables introduce permission headaches and lack standardization across warehouses. The dbt external tables package, while clever, violates a core dbt principle: sources should exist independently, not be managed by the transformation layer

• Trino + AWS Glue Catalog — by configuring a Hive catalog in Trino and pointing it at Glue, we get a fully managed, serverless metastore without spinning up any additional infrastructure. A tightly scoped IAM policy keeps things secure and follows the principle of least privilege

• Dynamic table creation with dbt macros — instead of manually registering Hive tables, a single reusable macro handles the DDL as a pre_hook, keeping table definitions in sync with your S3 files on every run

• Clean separation of concerns — the macro owns infrastructure, the model owns transformation. dbt stays in its lane, and your pipeline stays maintainable

The result is a lightweight, reproducible, and version-controlled ingestion pipeline that lives entirely in your dbt project — no console gymnastics, no permission rabbit holes, no overengineering.

If you’re already running Trino on AWS and want a simple on-ramp for S3 file ingestion, this stack is hard to beat. As always, the best pipeline is the one your team can actually understand, maintain, and extend — and this one fits that bill.

Introduction

Like many Charter Management Organizations (CMOs) across the U.S., Uncommon Schools relies on PowerSchool as its Student Information System (SIS) -- the system of record for student enrollment, demographics, attendance, grades, and scheduling. While PowerSchool is critical for day-to-day school operations, it wasn’t designed for fast, large-scale analytics use cases.

Under the hood, PowerSchool stores its data in an Oracle database, which makes extracting large tables -- like Students -- surprisingly slow when using conventional approaches.

In this post, I’ll walk through how we reduced ingestion time for the Students table from nearly 50 minutes to under a minute. We achieved this by migrating from a custom Python-based pipeline to Trino, enabling significantly faster data access and simpler ingestion logic. Along the way, I’ll share a few practical techniques to speed up data extraction -- not just in Oracle, but in any database system that supports procedural logic.

This work builds on ideas we explored in our earlier post, Trino + dbt: Simplifying ELT with Pure SQL, where we showed how combining Trino and dbt can simplify ingestion, improve reliability, and unify ingestion with transformation workflows. Here, we take that foundation further -- focusing on performance, real-world constraints, and how to make ingestion not just simpler, but dramatically faster.

The Original Pipeline: Where We Started

Our initial approach was built around a containerized extraction layer. We created a custom Docker image based on an Oracle Instant Client setup, which allowed us to connect directly to the PowerSchool Oracle database and run extraction queries. This image included Python, SQL*Plus, and a set of internal scripts used to execute parameterized queries and export data.

We orchestrated everything with Airflow, where each table ingestion was defined as a separate task in a DAG. Each task would spin up a container (via ECS/EKS) using our custom operator, ensuring that extraction workloads didn’t run on Airflow workers themselves but instead executed in isolated pods. This design gave us scalability and parallelism—but also introduced operational complexity.

To control what data gets extracted, we relied on SQL templates. Each table had its own query template, dynamically parameterized (e.g., by PowerSchool instance), and executed inside the container. For example, the Students table template included not only direct column selections, but also calls to Oracle PL/SQL functions like ps_customfields.getStudentsCF.

These functions are part of PowerSchool’s internal logic layer and allow access to data that isn’t physically stored in the base tables. While this gave us flexibility and completeness, it also came at a significant performance cost—each function call is effectively an additional computation executed per row.

Over time, several issues with this approach became clear:

• Lack of lineage and validation: Changes to SQL templates were difficult to track and validate against downstream models. A simple column removal could silently break transformations later in the pipeline.

• Operational overhead: Each table required its own Airflow task and container. With ~30 tables, this meant dozens of parallel pods, making logs harder to read and increasing pressure on orchestration infrastructure.

• Performance bottlenecks: The biggest issue—queries were slow. Very slow. Extracting the Students table for a single instance could take up to 50–60 minutes, largely due to heavy use of PL/SQL functions and row-by-row processing.

In short, while the pipeline worked, it wasn’t scalable, transparent, or fast enough for our needs.

The New Approach: Fast Ingestion with Trino

To address the limitations of our original pipeline, we leaned into the approach we described in our previous post, Trino + dbt: Simplifying ELT with Pure SQL. The core idea is simple but powerful: move ingestion logic into SQL, unify it with transformation workflows in dbt, and let Trino handle execution across systems.

This shift solved several problems at once.

First, unpredictable production issues became much easier to manage. By bringing ingestion into dbt, we gained lineage, dependency tracking, and the ability to validate changes before they reach production. Instead of discovering issues at runtime, we can now catch them during development or CI – exactly the gap we highlighted in our earlier work.

Second, we eliminated the overhead of “millions of tasks and pods.” Instead of running one Airflow task per table (and spawning dozens of containers), we now have a single Airflow task that triggers a dbt run. Within that run, dbt executes multiple models concurrently using threads – all inside a single pod. This dramatically simplifies orchestration, improves observability, and reduces load on both Airflow and our container infrastructure.

But the biggest win came from query performance.

In the new approach, the ingestion logic is written as a dbt model that uses Trino’s system.query to push down execution directly to Oracle. Instead of relying heavily on row-by-row PL/SQL function calls, we restructured the query to:

Replace expensive function calls where possible with joins (e.g., leveraging studentcorefields, vwstudentcorefieldsuc, and other tables instead of repeatedly calling ps_customfields.getStudentsCF)

The result is a much more efficient execution plan: fewer function calls, better use of joins, and improved parallelism across databases.

Combined with Trino’s ability to execute queries concurrently and dbt’s threaded execution model, this reduced the ingestion time for the Students table from nearly an hour to under a minute.

Just as importantly, the pipeline is now simpler, more maintainable, and fully transparent. Instead of spreading logic across Docker images, Airflow tasks, and SQL templates, everything lives in version-controlled dbt models—fully integrated with the rest of the data platform.

To tie everything together operationally, we also introduced a lightweight custom Airflow operator that runs dbt + Trino workloads inside a single Kubernetes pod:

Summary

Uncommon Schools’ data team cut ingestion time for PowerSchool student records from nearly an hour to under a minute by replacing a custom Python/Docker pipeline with Trino and dbt. The original setup relied on containerized Oracle extractions, row-by-row PL/SQL function calls, and one Airflow task per table — functional, but slow, hard to maintain, and operationally expensive. The new approach moves ingestion logic into dbt models, uses Trino to push queries directly down to Oracle, and replaces costly function calls with efficient joins. The result is a faster, simpler, and fully version-controlled pipeline where lineage, validation, and transformation all live in one place.

When we first introduced Airflow Schedule Insights, we told the story of Dr. Movie, a cinema chain struggling with a “scheduling black box.” They had DAGs that silently refused to run, resource bottlenecks from over-parallelization, and zero visibility into when their next asset-based trigger would actually fire.

That story helped explain the problem.

But with Airflow 3, we decided to evolve the example environment.

Instead of fictional cinema DAGs, we now ship example educational DAGs that mimic real academic data platform dependency patterns. These DAGs are not production pipelines — they simply replicate common educational orchestration structures and asset relationships so users can test complex scheduling scenarios in a realistic setting.

In other words:
They copy educational DAG dependency patterns — not institutional data.

This makes the demo environment more relatable for universities and training organizations, while remaining lightweight and safe to use as a sandbox.

And alongside that shift, Schedule Insights v0.2.0 is officially out — fully migrated, dependency-light, and optimized for Airflow 3.

pip install airflow-schedule-insights

👉 GitHub: https://github.com/ponderedw/airflow-schedule-insights

What Changed in v0.2.0 (Airflow 3 Migration)

Under the hood, 22 files changed. If you’re a plugin developer, these are the silent Airflow 3 metadata changes you need to understand.

1️⃣ The Great Renaming: Datasets → Assets

Airflow 3 formalized the shift from “datasets” to “assets.”

Old:

task_outlet_dataset_reference

New:

task_outlet_asset_reference

If your plugin queries the metadata DB directly, this rename will break you.

2️⃣ Logic Flip: is_active → is_stale

Airflow 2:

DagModel.is_active == True

Airflow 3:

DagModel.is_stale.is_(False)

The column is gone. The logic is inverted. If you don’t update this filter, your visibility logic silently fails.

3️⃣ Asset Dependencies Are Trees, Not Lists

Airflow 3 introduces conditional asset expression9s with nested AND / OR logic.

Example:

(Asset_A AND (Asset_B OR Asset_C))

This is no longer a flat dependency list.

We implemented a recursive evaluator that parses the new asset_expression JSON structure so Schedule Insights can still predict:

• When a DAG will run

• What it’s waiting on

• Which branch of logic is currently blocking execution

Without this, “next run” predictions are simply wrong.

4️⃣ Better Performance, Fewer Dependencies

We removed pandas and numpy.

Run duration medians are now computed directly in SQL using percentile_cont via SQLAlchemy.

Result:

• Faster installs

• Lower memory usage

• Cleaner dependency graph

Example Educational DAGs (Dependency Patterns Only)

For Airflow 3, the repository now includes an Academic Data Platform example DAG suite.

These DAGs simulate patterns such as:

• Ingestion → normalization → transformation → consolidation

• Dataset/asset-triggered execution

• Hybrid cron + asset scheduling

• Conditional AND/OR asset logic

• Reporting layers dependent on multiple upstream assets

They exist purely to demonstrate:

• Complex asset expressions

• Multi-layered DAG orchestration

• Real-world dependency graphs

No real institutional integrations.
No sensitive data.
Just realistic dependency topology.

This makes Schedule Insights far more meaningful to demonstrate than the original cinema example, especially when working with Airflow 3’s asset trees.

Core Features (Now Tested on Complex Asset Graphs)

🛠 The Next Run Table — No More Silent Failures

Asset-driven DAGs often “don’t run” for subtle reasons:

• Paused DAGs

• Waiting on asset expressions

• Hybrid cron + dataset logic

• Unmet upstream conditions

The Next Run table explains exactly why:

• “This DAG is paused and will not run.”

• “No schedule or dependencies found.”

• “Waiting for upstream Asset/DAG triggers.”

📊 The Schedule Impact Simulator

Want to change:

*/25 → */5

Before committing?

Simulate it.

The UI highlights impacted future runs in purple and shows how the change propagates through downstream asset dependencies.

This is especially useful in complex multi-layer DAG graphs.

🕒 Timezone-Aware Gantt Charts

The plugin detects your local timezone and renders:

• Historical runs

• Predicted runs

• Gantt charts

No more UTC mental math.

🔌 Programmatic Insights via API

We exposed the scheduling engine via FastAPI endpoints:

• get_future_runs_json

• get_next_future_run_json

Your internal tooling can now query predicted Airflow behavior programmatically.

Visualizing Structure with airflow-dag-dependencies

To complement Schedule Insights, you can use our companion plugin:

airflow-dag-dependencies

Schedule Insights tells you when a DAG will run.
airflow-dag-dependencies shows you how everything connects.

Together they provide:

• Structural visibility (dependency graph)

• Temporal visibility (future execution prediction)

Especially powerful with Airflow 3’s conditional asset trees.

Why This Matters at Scale

As more teams contribute DAGs to a shared Airflow environment — as is the case at Uncommon Schools, where multiple teams orchestrate data across a large network of schools — scheduling complexity compounds quickly. Each team adds their own timing requirements, and as run durations drift, overlap and resource contention become increasingly hard to spot from any single vantage point. Schedule Insights gives you a cross-team view — so you can see potential collisions before they become incidents. And when a DAG pile-up does occur, the historical run data lets you work backward and reconstruct what likely triggered the cascade.

Try It Today

Spin it up locally:

git clone https://github.com/ponderedw/airflow-schedule-insights
cd airflow-schedule-insights
docker compose -f local_airflow_3/docker-compose.yml up

Then navigate to:

Browse → Schedule Insights

You’ll see the plugin running against realistic educational dependency patterns — not fictional cinema workflows.

Airflow 3 made orchestration more powerful.
Schedule Insights makes it understandable.

Introduction

In our previous post, we explored how to bridge the gap between complex data structures and end-users by grounding natural-language queries in a Cube-governed semantic layer. By using RAG, we enabled anyone—from teachers to board members—to query data without needing to touch SQL, GraphQL, or REST APIs.

But while answering a question with text is a great start, data is often best understood visually. Even for those who can write code, the goal is often the same: getting to the insight as quickly as possible. We asked ourselves: Can we move beyond simple text responses and visualize this data directly within the chat interface?

The answer is a resounding yes.

In this post, we’re introducing a new capability that turns conversational insights into immediate, actionable BI charts. We’ve implemented this by leveraging Streamlit’s native charting capabilities combined with custom logic for parsing the AI’s response.

Of course, we aren’t alone in this pursuit—the entire industry is racing toward “conversational BI.” This is simply our take on the problem. Our hope is that this specific implementation provides a blueprint or some inspiration for bringing similar capabilities to your own organization. If you have a different approach or wider experience in this space, we’d love for you to share your thoughts in the comments.

You can find the updated code in the same repository here:

👉 GitHub: https://github.com/ponderedw/dbt-to-cube

Let’s dive into how we move from a simple chat response to a fully rendered chart.

One quick note before we dive in: all of the examples and visualizations in this post use synthetic sample data generated as part of the demo environment.

Quickstart

Ready to see it in action? You can get the entire environment running in just a few steps.

1. Clone the repository: git clone https://github.com/ponderedw/dbt-to-cube

2. Configure your environment: Create your .private.env file from the .private.env.template as explained here. This ensures your LLM provider and credentials are ready to go.

3. Launch the stack: Simply run just all. This automated workflow handles the heavy lifting: spinning up the database, generating mock data, running dbt models, and ingesting metrics into both Cube and Superset. (You can read more about the underlying architecture here).

4. Access the UI: Head over to http://localhost:8501/, enter your STREAMLIT_PASSWORD, and you’re ready to chat with your data.

From Questions to Visuals

Start by getting familiar with the metadata the RAG layer has indexed:

“Hey. What measures and dimensions do we have?”

Once the AI confirms the available metrics, it’s time to move beyond text. Let’s ask for a specific visualization:

“Can you visualize the total number of course enrollments by difficulty level using a pie chart?”

The result? A fully rendered BI graph in just 11 seconds.

Let’s try a different metric and a more dynamic chart type:

“Return the number of students with excellent performance across academic years and visualize it using a line chart.”

In seconds, you receive an amazing, interactive line chart—mapping complex trends without you ever touching a visualization library.

More to Explore

The system is flexible enough to handle various chart types and comparison logic:

• Area Charts: “Can you create an area chart showing the number of students with excellent performance compared to the total number of enrolled students across semesters?”
  

• Scatter Charts: “Can you visualize the relationship between course difficulty level and average course GPA using a scatter chart?”
  

How does it work?

To enable these visualizations, we didn’t need to reinvent the wheel. We kept the same agent-based architecture and Streamlit frontend but introduced two key modifications to the workflow:

1. Enhancing the Prompt (The “Brain”)

We updated our LangChain prompt to give the agent “visual awareness.” It now understands how to structure its output so the frontend knows when a chart is requested.

• Where to look: You can find the specific instructions we added in cube-to-rag/app/server/chat.py.

2. Intelligent Response Parsing (The “Bridge”)

Once the AI generates a response, the Streamlit application needs to know whether to display a paragraph of text or render a dynamic chart. We updated the handling logic to parse the AI’s response and trigger the appropriate Streamlit chart component.

• Where to look: The updated logic is located in streamlit/Main.py.

By separating the “intent” (in the prompt) from the “rendering” (in Streamlit), we maintain a clean separation of concerns while delivering a seamless user experience.

Conclusion

This experiment marks a significant step in making data truly accessible. By moving from simple text-based answers to dynamic BI visualizations, we’ve closed the loop between asking a question and gaining a visual insight.

The beauty of this approach lies in its governance. Because the AI is still grounded in the Cube semantic layer, the charts aren’t just “hallucinated” guesses—they are visual representations of verified business logic. Whether it’s a school administrator tracking student performance or a developer looking for a quick metric check, the interface provides a “hands-free” BI experience that stays accurate at scale.

What’s Next?

While this prototype demonstrates the power of combining RAG with Streamlit, there is still much to explore:

• Embedded BI Dashboards: Moving beyond single charts to embed fully functional, pre-built dashboards (like those from Apache Superset) directly into the chat when a high-level overview is requested.

• Advanced Interactivity: Allowing users to drill down into chart elements through further conversation or “chatting with the legend.”

• Complex Multi-Step Reasoning: Improving how the agent handles requests that require joining multiple cubes or complex pre-aggregations.

We believe that the future of BI isn’t just a static dashboard; it’s a conversation that adapts to the user’s needs in real-time. We hope this project helps you kickstart your own journey into conversational analytics.

Join the Conversation

We’d love to hear your thoughts!

• Did this setup work for your data?

• Do you prefer a different library for parsing AI-generated charts?

• What’s the biggest challenge you’ve faced with Text-to-Chart workflows?

Drop a comment below, or better yet, open an issue or PR on the GitHub repository.

Introduction

In one of our previous posts, we showed how dbt, Cube, and Superset can work together to automate and standardize the semantic layer. The response from the community was overwhelmingly positive but it also raised a recurring question: why introduce Cube between dbt and Superset instead of connecting BI tools directly to the database?

The answer is consistency at scale. A single, shared semantic layer ensures that metrics, dimensions, and business logic are defined once and reused everywhere – across BI tools, APIs, and downstream applications – rather than being reimplemented in each consumer.

This naturally led us to a bigger idea. If Cube already exposes a governed semantic layer through SQL, REST, and GraphQL, why force natural-language analytics back into brittle English-to-SQL translations? What if users could converse directly with the semantic layer instead?

In this post, we explore how to turn Cube analytics into a natural-language interface using RAG. We use RAG to store and retrieve Cube metrics and dimensions, allowing each user query to be grounded in the correct semantic context before any query is generated or executed. This keeps conversations aligned with predefined business logic while remaining flexible and intuitive for users.

Let’s walk through how it works.

👉 GitHub: https://github.com/ponderedw/dbt-to-cube

Quickstart

1. Clone the repository:
   👉 https://github.com/ponderedw/dbt-to-cube

2. Copy .private.env.template to .private.env and configure the required secrets for your LLM provider.

3. just all

Once everything is up and running:

• Open:

http://localhost:8000/

• and navigate to the cube_schemas collection.
  You’ll see a set of vectors representing Cube metadata – dimensions, measures, and cubes.
  These embeddings are generated using the cube-to-rag library (https://pypi.org/project/cube-to-rag/), which we use to extract and embed Cube schema metadata into the vector store.
  

• Next, open

http://localhost:8501/

• If you’ve set STREAMLIT_PASSWORD in .private.env, use it to log in.
  

You’re all set, start asking questions and explore your Cube analytics through natural language.

Talking to Your Data

Let’s start simple:
“What metrics and dimensions do we have?”

Great. Now let’s explore the data itself:
“Show all available course names.”

Nice. Let’s narrow it down:
“What metrics are available for Linear Algebra?”

Now for a concrete business question:
“How many enrollments do we have across all courses?”

And finally, let’s push it a bit further with a more complex query involving filters and breakdowns:
“What is the student engagement score for the Data Structures and Algorithms course, broken down by semester?”

And just like that, everything works. Metrics stay consistent, dimensions are respected, and the conversation maps cleanly to the semantic layer

Conclusion

This experiment shows one possible direction for moving from static semantic layers to truly conversational analytics. By grounding natural-language queries in Cube’s governed semantic layer using RAG, we avoid many of the pitfalls of naive text-to-SQL approaches: inconsistent metrics, broken joins, and logic that drifts away from business definitions. Instead, users can explore data conversationally while staying aligned with a single source of truth.

That said, this is not the solution – it’s our attempt at solving a real and growing problem. There are open questions around performance, UX, evaluation, security, and how this approach compares to alternatives such as agent-driven query planning, stricter semantic parsers, or hybrid BI experiences. We’re certain there are improvements to be made, edge cases we haven’t covered, and ideas we haven’t considered.

We’d love to hear what others think about this: What works? What doesn’t? What would you do differently? If you’re experimenting with similar approaches – or taking a completely different path – your feedback, suggestions, and alternatives are more than welcome.

Introduction

Most teams have been there. You start storing company documentation as Word files or PDFs because it’s quick, familiar, and temporary. A proof of concept, a contract, a report – throw it into a folder and move on. But over time, that “temporary” storage often becomes a critical production knowledge base.

At that point, problems emerge: documents are hard to search, relationships are implicit, extracting structured data is manual, and automation becomes painful.

A common approach today is “just put everything into RAG.” While retrieval helps with search, it doesn’t solve the core issue: the information itself is still unstructured. If the content were normalized into a machine-friendly format (like JSON with well-defined schemas), working with it would be faster, more reliable, and would open new workflows – analytics, validation, integrations, and automation.

That gap is what inspired this experiment. I wanted a way to turn business documents into structured, queryable data using LLMs. The result is StructuDoc, an open-source proof of concept that combines document parsing, image understanding, schema discovery, and prompt tracking into a single pipeline. It can move PDFs and DOCX files into structured datasets, identify common schemas, and store everything in S3.

This post introduces StructuDoc, explains how it works, and shares the reasoning behind its design. It’s offered in a spirit of sharing rather than as a commercial product – we’d love your feedback, alternative approaches you’ve tried, or contributions to the project.

• GitHub: https://github.com/ponderedw/StructuDoc

• Docker Hub: https://hub.docker.com/repository/docker/pondered/structudoc/general

Quickstart with StructuDoc

StructuDoc makes it easy to turn PDFs and DOCX files into structured JSON, discover common schemas, and even analyze images using AI. Here’s how to get started in minutes:

1. Clone the repository

git clone https://github.com/ponderedw/StructuDoc.git
cd StructuDoc

2. Set up environment variables

Copy the example file:

cp .env.example .env

Edit .env to configure your local storage (MinIO) or S3, and optionally your LLM provider keys:

ENV=local
SOURCE_BUCKET=minio/source-bucket
MINIO_HOST=http://localhost:9000
MINIO_SECURE=false
AWS_ACCESS_KEY_ID=admin
AWS_SECRET_ACCESS_KEY=password
LLM_MODEL=Bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0

3. Start the application (this file can be executed independently of the repository)

docker compose up -f docker-compose-prod.yml

• Streamlit UI:

 http://localhost:8501

• FastAPI backend:

 http://localhost:8080

• MinIO console (login: admin/password):

 http://localhost:9001

4. Upload and process documents

1. Go to Upload Source Files in the UI.

2. Upload DOCX or PDF files.

3. StructuDoc will automatically:

   • Convert PDFs to Markdown

   • Extract images

   • Store all files in structured folders

5. Parse and analyze with AI

• Navigate to Parse Files With LLM:

• Analyze images with AI prompts

• And remember to save those descriptions to your s3 bucket:

• Discover common schemas across multiple documents

• Parse your documents to JSON

Now you have structured, queryable data from your documents – ready for testing, analysis, or building pipelines!

Conclusion

StructuDoc is an open-source experiment for turning PDFs and DOCX files into structured, queryable data. Instead of relying solely on RAG for search, it lets you extract JSON, discover common schemas, and analyze images with AI. This isn’t a fully polished product – it’s a proof of concept born from real challenges we’ve faced, shared in the hope it might help others or spark new ideas. We’d love to hear your feedback, learn about alternative approaches you’ve tried, or see contributions that improve StructuDoc.

GitHub: https://github.com/ponderedw/StructuDoc
Docker Hub: https://hub.docker.com/repository/docker/pondered/structudoc/general

Introduction

dbt has become the backbone of modern analytics engineering, and one of its best features is CI: the ability to compare a developer branch with main, rebuild only the changed models in a temporary schema, and use production data from unchanged models. It’s an amazing way to catch issues before they hit production.

But what about incremental models and snapshots? Metrics like Grade Point Average whose day-by-day variations are captured in a dbt snapshot are then tracked and rendered in various contexts over time. This per-student metric is calculated in one environment (production) each day to ensure a clear source of truth. But for CI…that’s where things get tricky.

Both incremental models and snapshots have different logic for initial runs vs. upsert runs. In CI, everything is always built from scratch – so your CI may pass, only for the query to fail the moment it meets existing data or objects in production.

To solve this for Uncommon Schools, we built a tiny but powerful solution – and now we’re sharing it with everyone:
📦 dbt-incremental-ci (https://pypi.org/project/dbt-incremental-ci/)

Quickstart

Get up and running in under 5 minutes.

dbt-incremental-ci copies your production incremental models and snapshots into your CI schema so you can test against real production data, not empty tables.

Installation

pip install dbt-incremental-ci

Option 1: Using a Local Manifest File

Use this approach if your production manifest.json is stored in S3, GCS, or available locally.

Step 1: Download your production manifest

# From S3
aws s3 cp s3://my-dbt-artifacts/prod/manifest.json ./prod_manifest.json

# From GCS

gsutil cp gs://my-dbt-artifacts/prod/manifest.json ./prod_manifest.json

Step 2: Copy production tables into your CI schema

dbt-incremental-ci \
  --prod-manifest-path ./prod_manifest.json \
  --dbt-project-dir . \
  --database-uri “postgresql://user:pass@host:5432/db” \
  --ci-schema “ci_pr_123” \
  --threads 4

Step 3: Run dbt with the copied production data

dbt build --select state:modified+ --defer --state ./prod_manifest.json

Step 4: Run your tests

dbt test --select state:modified+ --defer --state ./prod_manifest.json

You’re now running incremental models and snapshots against actual production state—not empty CI tables.

Option 2: Using dbt Cloud

If you’re on dbt Cloud, you don’t need to download the manifest—the tool fetches it automatically from the API.

Step 1: Set your credentials

export DBT_CLOUD_API_TOKEN=”your_token”
export DBT_CLOUD_ACCOUNT_ID=”12345”

Step 2: Copy production tables into your CI schema

dbt-incremental-ci \
  --dbt-cloud-job-id “67890” \
  --dbt-project-dir . \
  --database-uri “$DATABASE_URI” \
  --ci-schema “ci_pr_123” \
  --threads 4

The tool will automatically:

• Connect to the dbt Cloud API

• Fetch the latest successful production run

• Download the manifest

• Copy the required tables

Step 3: Run dbt

dbt build --select state:modified+ --defer

Step 4: Run your tests

dbt test --select state:modified+ --defer

What Just Happened?

The tool:

1. Identified changed models using state:modified+

2. Filtered to incremental models and snapshots that already exist in production

3. Copied those production tables (with data!) into your CI schema

4. Preserved custom schemas (e.g., prod_marts → ci_pr_123_marts)

This ensures your CI pipeline behaves like a real incremental run—catching bugs that only appear when existing data is present.

Preview Before Running (Dry Run)

Want to see what will be copied without actually doing anything?

dbt-incremental-ci \
  --prod-manifest-path ./prod_manifest.json \
  --dbt-project-dir . \
  --database-uri “postgresql://user:pass@host:5432/db” \
  --ci-schema “ci_pr_123” \
  --dry-run \
  --verbose

This prints exactly which tables would be copied and the SQL that would run.

Final Thoughts

Incremental models and snapshots are some of the most powerful parts of dbt – but they’re also uniquely tricky to validate in CI. Full rebuilds simply don’t reflect how these models behave in production, which means some of the most painful bugs can slip past your pipeline undetected.

dbt-incremental-ci is our attempt to fix that gap. It’s a lightweight, practical solution that brings real production state into your CI runs so you can test incremental logic the way it actually runs in the real world.

But this isn’t the final word – far from it.
This is our first take at solving the problem, and we know the community will have ideas we haven’t thought of yet. If you have a better approach, want to improve this one, or see opportunities to make it smarter, faster, or more general – we’d love your help.

👉 PyPI: https://pypi.org/project/dbt-incremental-ci/
👉 GitHub: https://github.com/ponderedw/dbt-incremental-ci – issues, discussions, and PRs are all very welcome.

Introduction

A couple of weeks ago, we shared several blog posts about different ways we’re making data more accessible and interactive: our Superset MCP (link), loading dbt metadata into GraphDB (link), and an amazing AI-powered UI plugin for Airflow that lets you interact with your workflows directly in the UI (link).

Today, I want to bring all three together and introduce superset-chat - a library that helps you manage both your dbt and dashboard metadata right in the Superset UI, powered by AI, Superset MCP, and our Falkor DB plugin. You can check it out on GitHub: superset-mcp-plugins.

If you’ve ever felt a disconnect between BI and transformation logic, this library might resonate. It can answer questions like, “Which dashboards depend on source X?” - even when those sources aren’t referenced directly. While still a POC, superset-chat has already helped us organize educational metadata, and we’re excited to share it with the community. Maybe you’ll help improve it or come up with something entirely new!

Quickstart

You can try superset-chat in two ways:

1. Use your existing Superset

If you already have a Superset deployment, you can integrate the plugin directly:

• Install the plugin:

pip install superset-chat

• Update your superset_config.py to register the plugin:

FLASK_APP_MUTATOR = lambda app: init_custom_views(app)
def init_custom_views(app):

    from superset_chat.ai_superset_assistant import AISupersetAssistantView

    appbuilder = app.appbuilder
    appbuilder.add_view(
        AISupersetAssistantView,
        “AI Superset Assistant”,
        icon=”fa-robot”,
        category=”Custom Tools”
    )

• Make sure you have the following environment variables set for your deployment:

LLM_MODEL_ID=bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0  # Must start with ‘bedrock’, ‘anthropic’, or ‘openai’
# ANTHROPIC_API_KEY=...              # Required if using Anthropic
AWS_ACCESS_KEY_ID=’...’             # Required if using Bedrock
AWS_SECRET_ACCESS_KEY=’...’         # Required if using Bedrock
SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://postgres:postgres@postgres:5432/postgres
SUPERSET_API_URL=http://localhost:8088
SUPERSET_USERNAME=superset_admin
SUPERSET_PASSWORD=superset
GRAPH_DB=falkordb                     # Optional: enable GraphDB connection
GRAPH_HOST=...                        # Optional: if using a remote GraphDB
GRAPH_USER=...
GRAPH_PASSWORD=...

Restart Superset, log in, and open AI Superset Assistant under Custom Tools.

2. Run the local deploy

If you don’t have a Superset deployment, you can use our preconfigured local setup:

• Clone the repository:

git clone https://github.com/ponderedw/superset-mcp-plugins.git
cd superset-mcp-plugins

• Copy the environment template and configure credentials:

cp template.env .env

• The minimal .env for local deploy (we use FalkorDB in this tutorial) can be:

LLM_MODEL_ID=bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0
# ANTHROPIC_API_KEY=...
AWS_ACCESS_KEY_ID=’...’
AWS_SECRET_ACCESS_KEY=’...’
SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://postgres:postgres@postgres:5432/postgres
SUPERSET_API_URL=http://localhost:8088
SUPERSET_USERNAME=superset_admin
SUPERSET_PASSWORD=superset
GRAPH_DB=falkordb

• Run Superset with superset-chat:

just run_superset

• Navigate to your Superset instance (usually http://localhost:8088) and open AI Superset Assistant under Custom Tools.

This local deploy is a fully working Superset + superset-chat environment, so you can start experimenting even if you don’t have your own deployment.

Using superset-chat: Examples

If you’re using our project, we’ve included a dbt project that was automatically loaded into FalkorDB, along with Superset charts and dashboards that were imported into Superset. You can explore all of them in the UI as well as in dbt docs.

Now, let’s try out the plugin with a few example queries:

1. Superset-related question:

“Quick question – what’s our current count of Superset dashboards and charts?”

As expected, superset-chat lists all the Superset metadata correctly – dashboards and charts – just like in the UI. Cool!

2. dbt-related question:
   
   

“Hey! Do you know how many dbt models we have in total?”

We can check the answer in dbt docs, and it’s completely correct. Awesome!

3. Combined dbt + Superset question:

“Could you help me find all the dbt sources (not models) used in the Student Academic Standing chart?”

Amazing! superset-chat traced the chart all the way back to the underlying database tables. This shows how we can connect Superset dashboards directly to dbt metadata seamlessly.

Connecting BI and Transformation Metadata with AI

Connecting BI and Transformation Metadata with AI
With superset-chat, you can finally see your BI layer and your transformation logic in one place, without manually tracing dependencies or opening multiple tools. By unifying dbt lineage and Superset metadata, the plugin allows you to:

• Ask natural-language questions spanning both dashboards and data models.

• Trace dashboards and charts back to the underlying source tables.

• Verify dbt lineage directly from Superset without leaving the UI.

• Quickly get insights that previously required multiple queries or digging through documentation.

Even as a POC, superset-chat already shows how AI can bridge the gap between data transformation and business intelligence. It’s also beginning to influence some of our internal workflows - for example, in our work with Uncommon Schools we’re exploring how the assistant can accelerate outage protocols, particularly when on-call responders aren’t familiar with the specific code paths involved. This kind of scenario highlights the value of unifying metadata and making it accessible through natural language.

Whether you’re exploring dashboards, analyzing dbt models, or troubleshooting lineage issues, this tool turns what was once a fragmented process into a single, interactive experience. We’re excited to share this with the community and invite you to try it, suggest improvements, or build on it for your own workflows. It’s a step toward truly connected, AI-powered data exploration.

Introduction

Rebuilding the same semantic layer over and over across different BI tools has always felt… unnecessarily painful. Most dashboards still reach directly into databases, each with their own idea of what a “metric” is, so you end up maintaining the same logic in three or four places. Even tools built to solve this – like Cube.js – don’t really solve the end-to-end problem of getting consistent metrics into your BI layer.

Out of curiosity (and a bit of frustration), I started tinkering with a small proof of concept to see whether dbt, Cube.js, and Superset could be wired together into a single automated flow. The result is dbt-to-cube:

https://github.com/ponderedw/dbt-to-cube

It’s very much a work in progress, but it already connects dbt → Cube.js → Superset and keeps metrics in sync without any manual steps. Sharing it here in case others are experimenting with similar setups – I’d genuinely love feedback, ideas, or contributions from anyone who gives it a try.

Quickstart

See dbt-cube-sync in action! This walkthrough demonstrates a full semantic layer pipeline—from dbt transformations to BI dashboards—without any manual metric duplication.

Already Have dbt, Cube.js, and Superset?

If your stack is already deployed, you don’t need Docker. Just install dbt-cube-sync and point it to your existing deployment:

pip install dbt-cube-sync

# Convert dbt models to Cube.js schemas

dbt-cube-sync dbt-to-cube \

  --manifest /path/to/manifest.json \

  --catalog /path/to/catalog.json \

  --output /path/to/cube/output

# Sync Cube.js schemas to Superset

dbt-cube-sync cube-to-bi superset \

  --cube-files /path/to/cube/output \

  --url http://your-superset:8088 \

  --username admin \

  --password admin \

  --cube-connection-name Cube

This workflow works with Superset, Tableau, or PowerBI, giving you full control over your production-ready semantic layer.

Starting from Scratch? Spin Up Everything Locally

If you want a full demo stack including sample data, all services run locally in Docker: PostgreSQL, dbt, Cube.js, Superset, and dbt-cube-sync.

Prerequisites

• Docker & Docker Compose

• Git

• ~10 minutes

Step 1: Clone & Start the Stack

git clone https://github.com/ponderedw/dbt-to-cube
cd dbt-to-cube
docker-compose up --build

This will:

1. Initialize PostgreSQL with sample data

2. Run dbt transformations on 46 pre-built models

3. Start Cube.js API serving your semantic layer

4. Configure Superset with admin access and database connection

5. Automatically generate Cube.js schemas from dbt and sync them to Superset

Step 2: Explore Your Data Stack

Once the logs show “Pipeline completed successfully!”, you can access:

• Superset: http://localhost:8088 — login: admin/admin

  • 46 datasets auto-created from dbt models

  • Column metadata and metrics like Average GPA and Pass Rate

• dbt Docs: http://localhost:8080 — full DAG and column-level documentation

• Cube.js API: http://localhost:4000 — REST and GraphQL endpoints

Step 3: Watch the Pipeline in Action

docker-compose logs -f dbt-cube-sync

The pipeline runs in three automated phases:

1. dbt Parsing – reads manifest.json & catalog.json, finds models, and maps dependencies

2. Cube.js Schema Generation – converts dbt models into Cube.js dimensions and measures

3. BI Tool Sync – pushes datasets and metrics to your BI tool with proper metadata

Everything happens automatically for all models—no manual intervention required.

✅ What You’ve Built

In minutes, you now have a unified semantic layer pipeline:

1. Single source of truth: Metrics defined once in dbt

2. Automatic propagation: dbt → Cube.js → BI tools

3. Cross-tool flexibility: Works with Superset, Tableau, PowerBI

4. Developer-friendly: Version-controlled, testable, and documented

Replace sample data with your warehouse and dbt models, and you have a production-ready semantic layer – no metric inconsistencies across tools.

The Takeaway

This little experiment convinced me that an automated semantic layer doesn’t have to be complicated. Define metrics once in dbt, let a small tool translate them to Cube.js, and push them into your BI tool of choice – no more re-implementing the same logic across dashboards.

If you end up trying out dbt-to-cube, I’d be really interested to hear what works, what breaks, or what you’d want it to support next. It’s still a POC, but with feedback from others running into the same problems, it could grow into something genuinely useful for the community.

Always happy to chat if you’re experimenting with semantic layers or have ideas on making this whole workflow smoother!

Introduction

At Ponder, we’ve developed several plugins for Airflow 2.* and recently began migrating them to Airflow 3. Among our most valuable tools are Airflow Schedule Insights and Airflow Chat.

During the migration process, we discovered that Airflow 3.1.1 removed the DAG Dependencies tab that was present in Airflow 2.

Rather than wait for this feature to return, we saw an opportunity: we could build a plugin to restore this functionality, help the community migrate to Airflow 3 more smoothly, demonstrate how to create plugins for Airflow 3, and even add improvements we found missing in Airflow 2.

The result is airflow-dag-dependencies. In this post, we’ll walk you through how we built it and how it helps us manage our many educational DAGs.

Quick Start

To use our plugin, simply install airflow-dag-dependencies in your Airflow environment. If you don’t have an environment set up, clone our GitHub repository and run docker compose up.

After a few minutes, once Airflow is running, navigate to http://localhost:8080/ and log in with username “airflow” and password “airflow”. You’ll find a dozen educational DAGs — load_class_attendance, load_course_evaluations, and others.

Some are triggered by datasets, others by DAGs, and some run on cron schedules. The challenge? Understanding how they all connect.

This is where our plugin shines. Navigate to the Browse menu and click on DAG Dependencies.

You’ll immediately see all the relationships visualized clearly. Notice how transform_program_effectiveness has two dependencies: a cron schedule and a dataset trigger.

You can also spot that load_curriculum_catalog and load_student_enrollment_data have no dependencies or schedules—they won’t run automatically.

And you’ll see that transform_academic_performance is triggered by load_course_evaluations.

Enhanced Filtering

But we didn’t stop at recreating Airflow 2’s functionality—we added the filtering capabilities we always wished it had. Want to see only the direct downstream dependencies of transform_student_feedback_analysis and all its upstream dependencies? Simply enter +transform_student_feedback_analysis+1 in the search bar. The graph instantly filters to show only the relevant connections.

The search syntax is intuitive: use + before a DAG name to include upstream dependencies, after to include downstream, or both for the full picture. Add a number (like +dag_name+2) to control how many levels deep to traverse. You can also search multiple DAGs at once using commas, or use @exact_name for precise matching.

That’s the power of airflow-dag-dependencies!

Building Your First Airflow 3 Plugin: A Technical Deep Dive

Now that we’ve seen the plugin in action, here is how to build it from scratch. This walkthrough highlights the key differences between Airflow 2 and Airflow 3 plugin development and shows how to create your own plugin.

Plugin Structure: Airflow 3 vs Airflow 2

One of the most significant changes in Airflow 3 is how plugins are structured and registered.

Airflow 2 Plugin Structure

from airflow.plugins_manager import AirflowPlugin
from flask import Blueprint

from flask_admin import BaseView

class MyAirflow2Plugin(AirflowPlugin):

    name = “my_plugin”

    flask_blueprints = [my_blueprint]  # Flask blueprints

    admin_views = [MyAdminView]        # Flask-Admin views

    menu_links = [my_menu_link]        # Menu items

Airflow 3 Plugin Structure

from airflow.plugins_manager import AirflowPlugin
class MyAirflow3Plugin(AirflowPlugin):

    name = “my_plugin”

    fastapi_apps = [my_fastapi_app]     # FastAPI applications

    external_views = [my_external_view] # External view definitions

Key Changes in Airflow 3

• FastAPI replaces Flask for web applications, improving performance, async support, and automatic API documentation.
• External views replace Flask-Admin views, giving more flexible UI integration.
• Menu items are now part of external views rather than separate menu link definitions.

Anatomy of the DAG Dependencies Plugin

Package Structure

airflow_dag_dependencies/

├── __init__.py

└── plugins/

├── __init__.py

└── dags_dependencies_plugin.py

FastAPI Application

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI()

@app.get(”/”, response_class=HTMLResponse)

async def dag_dependencies_graph():

    # Return HTML with embedded D3.js visualization

    return html_content

@app.get(”/api/dag-dependencies”)

async def get_dag_dependencies():

    # Return JSON data for DAG relationships

    return {”nodes”: nodes, “links”: links}

Plugin Registration

FastAPI app configuration

fastapi_app_with_metadata = {
    “app”: app,
    “url_prefix”: “/dags_dependencies”,
    “name”: “DAG Dependencies App”
}

External view for menu integration

dag_dependencies_external_view = {
    “name”: “DAG Dependencies”,
    “href”: “/dags_dependencies/”,
    “destination”: “nav”,
    “category”: “browse”,
    “url_route”: “dag_dependencies”
}

Plugin class

class DagDependenciesPlugin(AirflowPlugin):
    name = “dags_dependencies_plugin”
    fastapi_apps = [fastapi_app_with_metadata]
    external_views = [dag_dependencies_external_view]

Data Access: Using DagBag

The plugin accesses Airflow’s DAG metadata through DagBag.

from airflow.models import DagBag
async def get_dag_dependencies():

    dagbag = DagBag(include_examples=False)

    nodes = []

    for dag_id, dag in dagbag.dags.items():

        nodes.append({

            “id”: dag_id,

            “type”: determine_dag_type(dag),

            “schedule”: get_schedule_info(dag)

        })

    links = analyze_dag_relationships(dagbag)

    return {”nodes”: nodes, “links”: links}

Frontend Options: Embedded or External

Airflow 3 supports multiple frontend approaches:

• Embedded HTML/JS (current plugin)
• Standalone React components (future tutorial)
• Serving static assets for more advanced UIs

Packaging for Distribution

Example pyproject.toml configuration:

[tool.poetry]
name = “airflow-dag-dependencies”

version = “0.1.1a1”

description = “An Apache Airflow plugin that provides interactive visualization of DAG dependencies”

authors = [”Ponder”]

[tool.poetry.dependencies]

python = “>=3.12, <3.14”

apache-airflow = “^3.1.0”

[tool.poetry.plugins.”airflow.plugins”]

dag-dependencies-plugin = “airflow_dag_dependencies.plugins.dags_dependencies_plugin:DagDependenciesPlugin”

What’s Next?

The next tutorial will cover:

• Building React components that integrate with Airflow’s UI
• Handling authentication and permissions

Airflow 3’s FastAPI foundation opens the door to more capable, modern, and faster plugin development. More powerful integrations are now possible than ever.

Conclusion

Airflow 3 marks a real shift in how developers extend the platform. FastAPI, external views, and the new plugin model make it possible to build tools that are faster, more flexible, and much easier to maintain. Rebuilding the DAG Dependencies feature gave us a chance to explore these new capabilities in depth and turn a missing Airflow 2 feature into something more capable than before.

The airflow-dag-dependencies plugin restores a critical part of the Airflow UI, adds filtering and search features that help teams understand complex pipelines, and acts as a practical example of how to build real plugins in Airflow 3. If you are migrating from Airflow 2 or planning to develop your own extensions, this approach will give you a strong foundation.

There is still more room to grow. React-based UI components, richer interactions with Airflow’s new frontend APIs, and improved permission handling are all on the way. Airflow 3 opened the door to a modern plugin ecosystem, and this is only the start.

Introduction

At Ponder, we first introduced Airflow MCP and its plugin to make working with DAGs and pipelines smoother. But along the way, we realized something: managing Superset dashboards and their metadata can be just as challenging—if not more so.

Take education as an example. Schools and educational institutions manage dozens—sometimes hundreds—of dashboards tracking everything from student performance and attendance to resource allocation and budget utilization. Multiple BI developers work constantly on these dashboards, making frequent updates and refinements so stakeholders can access the most current data to improve education and help pupils succeed. The result? A powerful ecosystem of insights—but also an overwhelming maze of metadata.

Questions like “Who changed the attendance dashboard last week?”, “Which table powers our graduation rate metrics?”, or “What’s the data source behind this enrollment forecast?” can quickly become a time sink. And it’s not just schools—any organization with a complex BI environment faces the same challenge.

That’s why we built Superset MCP—so you can explore and manage Superset metadata through natural language, directly from your Claude code. No need to wrestle with the UI, no need to dig through layers of metadata. Just type your question, and get clear answers in plain language.

You can check out the project on GitHub, pull the image from Docker Hub, or install it from PyPI.

Quickstart

1. Run Superset Locally

If you don’t already have a Superset deployment but still want to try out Superset MCP, simply clone our GitHub repository and start it with Docker Compose:

git clone https://github.com/ponderedw/superset-mcp
cd superset-mcp
docker compose up

Once it’s up, open your browser and visit:

http://localhost:8088/

Login credentials for the example environment:

• Username: superset_admin

• Password: superset
  
  

2. Using Superset MCP in Claude Desktop

1. Open Claude Desktop.

2. Navigate to: Settings → Developer → Edit Config.

3. Add the following entry to your MCP servers:

{
  “mcpServers”: {
    “superset”: {
      “args”: [
        “run”,
        “-i”,
        “--rm”,
        “-e”,
        “SUPERSET_API_URL”,
        “-e”,
        “SUPERSET_USERNAME”,
        “-e”,
        “SUPERSET_PASSWORD”,
        “pondered/superset-mcp:latest”
      ],
      “command”: “docker”,
      “env”: {
        “SUPERSET_API_URL”: “http://host.docker.internal:8088”,
        “SUPERSET_USERNAME”: “superset_admin”,
        “SUPERSET_PASSWORD”: “superset”
      }
    }
  }
}

Note: The Superset URL and credentials above match our local example environment.

4. Restart Claude Code.

3. Using Superset MCP with LangChain

To integrate with LangChain:

1. Install our PyPI package:

pip install superset-mcp-server

📦 PyPI link

2. Add the following MCP server configuration to your tools:

from langchain_mcp_adapters.client import MultiServerMCPClient
import os

mcps = {
    “SupersetMCP”: {
        “command”: “python”,
        “args”: [”-m”, “superset_mcp_server.mcp_server”],
        “transport”: “stdio”,
        “env”: {
            k: v for k, v in {
                “SUPERSET_API_URL”: os.getenv(”SUPERSET_API_URL”),
                “SUPERSET_USERNAME”: os.getenv(”SUPERSET_USERNAME”),
                “SUPERSET_PASSWORD”: os.getenv(”SUPERSET_PASSWORD”),
            }.items() if v is not None
        }
    }
}

client = MultiServerMCPClient(mcps)
mcp_tools = await client.get_tools()

Now you’re ready to simply ask questions about your Superset instance in natural language—and let MCP handle the rest!

Talking to Superset MCP

Let’s connect to our example Superset deployment and build a simple dashboard using the data automatically ingested into our Postgres database.

First, we’ll add a database connection — specify “postgres” as both the host and the database name, and use “superset” for the login and password.

Once connected, we can create a few simple charts from the database tables and organize them into an E-commerce Performance Dashboard.

Finally, don’t forget to publish your dashboard so it’s ready to use!

You can explore all the available tools in Claude Desktop — and more will be coming soon.

Now, let’s start asking some interesting questions:

• How many dashboards and charts do we have?

  

  

  Great! We get the exact number of dashboards and charts in our environment.

• Who are the most active users?

  

  

  
  Perfect — not only do we see the top active user, but we also get an Activity Timeline, which makes it much easier to understand what’s happening in our cluster.

• What is the source table for Customer-Product Network?

  

  Excellent — we now have the precise database object name.

• When was the Revenue by Country chart last changed, and by whom?

  

  Spot on — we’ve got the correct answer here as well.

Pretty cool, right? Keep exploring and help us uncover even more capabilities of Superset MCP!

Conclusion — Managing Metadata the Smart Way

Superset MCP turns the often painful task of navigating dashboard metadata into a fast, intuitive conversation. Whether you’re an educational administrator tracking changes to student performance dashboards, a data analyst identifying which tables feed into critical reports, or a team lead trying to understand who’s been most active in your BI environment—you can now skip the clicks, menus, and manual searches. Just ask in plain language, and get precise, actionable answers.

In education settings where BI teams constantly update dashboards to provide stakeholders with real-time insights for improving outcomes, Superset MCP becomes invaluable. But the benefits extend far beyond schools—any business with complex dashboard environments, whether in healthcare, finance, retail, or manufacturing, can leverage this natural language interface to cut through the metadata complexity.

Because it works entirely through natural language, you can easily delegate these tasks to analysts, project managers, principals, department heads, or any non-technical users—empowering more people in your team to get the information they need without relying on developers or admins.

We’re excited to see how the community will use Superset MCP—and we’d love your feedback. Try it out, share your ideas, and help us shape the next set of capabilities.

🔗 Resources:

• GitHub Repository

• Docker Hub Image

• PyPI Package

  Thanks for reading Data & AI Engineering @ Ponder! Subscribe for free to receive new posts and support our work.

Introduction

In education data teams, our analytics models often grow organically alongside organizational needs. As school networks expand their data infrastructure—adding new student information systems, assessment platforms, and operational tools—their dbt projects can become increasingly complex. What starts as a clean set of staging models quickly evolves into an intricate web of dependencies that becomes difficult to navigate and maintain.

In previous blog posts—dbt-to-English and English-to-SQL—we demonstrated how LLMs can assist with understanding and generating dbt metadata. However, these tools fall short when it comes to answering complex questions about dbt’s project-wide structure and dependencies. For instance, what is the most widely reused source across models (recursively)? Or, how many seed datasets are used by models materialized as tables?

We’ve seen this challenge firsthand. At one school network, their analytics catalog grew from a handful of models serving basic enrollment reporting to hundreds of interconnected models supporting everything from attendance analytics to intervention tracking to college persistence dashboards. This organic growth led to inadvertent circular dependencies that made it increasingly difficult to refactor models or optimize their orchestration. The team needed a way to visualize and understand these dependencies before they could untangle them.

That’s when we had a realization: dbt’s structure is naturally graph-shaped, making it an excellent fit for graph databases. This inspired us to launch a new project—dbt-kg—which shows how to upload dbt metadata (from manifest.json and catalog.json) into a graph database and leverage it in an AI-powered chat. While our demo uses FalkorDB, the project is also compatible with Neo4j—just update the configuration settings accordingly.

Quickstart

To get started, clone our GitHub repository:
https://github.com/ponderedw/dbt-kg

Next, copy the environment template and configure it:

cp template.env .env

In your .env file, set the LLM_MODEL_ID according to the provider you want to use:

LLM_MODEL_ID=’bedrock:...’
LLM_MODEL_ID=’anthropic:...’
LLM_MODEL_ID=’openai:...’

The prefix (bedrock, anthropic, or openai) determines which LLM provider will be used.
Be sure to include the appropriate credentials as well:

• AWS_SECRET_ACCESS_KEY for Bedrock

• ANTHROPIC_API_KEY for Anthropic

• OPENAI_API_KEY for OpenAI

Then, choose your graph database by setting the GRAPH_DB variable. This can be either falkordb or neo4j. For this demo, we’ll be using FalkorDB.

If your database requires authentication, provide values for GRAPH_USER and GRAPH_PASSWORD.

⚠️ Our default FalkorDB setup does not use authentication, so these can remain commented out. If you’re using our default Neo4j setup, uncomment those variables and set the default credentials accordingly.

Now, run the setup:

just all

Once everything is set up, open your browser and go to:

 http://localhost:8501/

Navigate to the “Load DBT Manifest” page from the sidebar. You’ll be prompted to upload your manifest.json and catalog.json files. If you have your own, upload them; otherwise, use the sample files located in DbtExampleProject/target within the repo.

Upon successful upload, you’ll see a confirmation message.

Finally, open your graph database UI at:

 http://localhost:3000/

Run the following query to view your dbt Knowledge Graph:

MATCH (a)-[b]-(c)
RETURN a, b, c

Congratulations! Your dbt Knowledge Graph is now up and running. 🎉

Interacting with Your dbt Graph via Chat

Let’s return to

 http://localhost:8501/

—this time, we’ll stay on the Main page and begin chatting with our dbt Knowledge Graph.

We’ll start by asking a high-level question about source table usage:

Prompt:
“Can you find the source tables that are used by the most models, including indirect dependencies through the entire dependency chain?”

The system correctly identifies both source tables from our example project and ranks them by usage in descending order. Perfect!

Next, let’s try something more challenging:

Prompt:
“Please identify the model with the highest number of dependent models, including transitive dependencies. This model must be materialized as a table. Return all the information you have about this table.”

This is tricky—because in our project, there are no models materialized as table.

Impressively, the LLM handles this gracefully by identifying the closest match: a view instead, and clearly stating the limitation. That’s a solid response.

Let’s follow up with another prompt to dig deeper:

Prompt:
“Can you also list indirect dependent models for stg_orders?”

Now we’re talking. The model returns the full dependency chain. Mission accomplished.

From One Monolithic Job to Smart Job Grouping

In many real-world workflows, especially when integrating with orchestrators like Airflow, we don’t want to run the entire dbt graph as a single job. Instead, we aim to break it into smaller, manageable jobs (DAGs) that can trigger downstream non-dbt processes as early as possible. But there’s a catch: circular dependencies often make this kind of separation complex and error-prone.

Let’s ask the LLM to help with that:

Prompt:
“Can you analyze our dbt dependency graph and create a job separation strategy? I need to group models, seeds, and snapshots into separate jobs that eliminate circular dependencies while defining clear inter-job dependencies.”

And it delivers! The model intelligently separates our graph into four distinct jobs and describes the cross-job dependencies clearly.

Summary

By leveraging the graph-shaped nature of dbt projects, the dbt-kg initiative demonstrates how graph databases like FalkorDB and Neo4j can supercharge the way we interact with dbt metadata. With an intuitive setup, seamless integration of LLMs, and a simple web-based interface, users can upload their dbt artifacts and start asking complex, project-wide questions in natural language.

For education data teams in particular, this approach addresses a common challenge: as schools and networks scale their analytics capabilities, their dbt projects naturally grow more complex. The AI-powered chat not only handles nuanced prompts about source usage, dependencies, and materializations but also assists in advanced tasks like intelligent job grouping to optimize orchestration workflows—exactly the kind of refactoring that the school network mentioned earlier needed to break apart their circular dependencies and create a more maintainable structure.

Whether you’re supporting a single school, a district, or a multi-state network, transforming static metadata into a dynamic, queryable knowledge graph makes data engineering more intelligent, conversational, and manageable. This is especially valuable in education, where data teams often wear multiple hats and need efficient ways to understand and maintain increasingly sophisticated analytics infrastructure.

Introduction

After recently launching our new library, airflow-chat—a plugin that enables seamless LLM chat interactions within your Airflow instance via our Airflow MCP—we quickly turned our attention to an important question: how do we effectively observe and log these conversations?

Imagine a scenario where a user triggers an unintended action due to safe mode being disabled, leading to a production error. How do you trace the issue? Who initiated the command? What was the exact prompt that caused the failure? Or more generally—how do you evaluate LLM interactions over time?

To address these questions, we turned to Langfuse, an elegant open-source tool purpose-built for developing, monitoring, evaluating, and debugging LLM applications. In this blog post, we’ll walk you through integrating Langfuse into your LangChain workflow—and specifically, how we’ve already done this within the airflow-chat plugin. You can follow along using our repo: https://github.com/ponderedw/airflow-mcp-plugins.

Quickstart

Let’s walk through a quick setup to see how Langfuse integrates with the airflow-chat plugin.

1. Clone the Repository and Set Up Locally

Start by cloning our GitHub repository:

git clone https://github.com/ponderedw/airflow-mcp-plugins

Then follow the setup steps outlined in our airflow-chat blog post to get the plugin running in your local environment.

Note: Running both Airflow and Langfuse locally can be resource-intensive. If your machine starts lagging, consider connecting to pre-deployed instances of Airflow and Langfuse for a smoother experience.

To launch both locally, run:

just airflow_langfuse

Once the containers are up, head to

 http://localhost:3000/

and sign up for a new Langfuse account.

2. Create an Organization, Project, and API Key

After signing up:

• Create a new Organization

  

• Create a Project under that organization

  

• Generate an API key

  

Copy the credentials and add them to your Airflow environment variables. If you’re using our GitHub setup, you can simply paste them into the airflow-plugin.env file:

LANGFUSE_SECRET_KEY=sk-lf-053f1073-789b-4705-8daa-23b825829d3d
LANGFUSE_PUBLIC_KEY=pk-lf-25851900-8b27-45f7-b7a0-617321e988cf
LANGFUSE_HOST=http://host.docker.internal:3000

3. Restart and Explore

Restart your environment:

just airflow_langfuse

Now, navigate to the Airflow UI, interact with the airflow-chat plugin, and ask a few test questions.

Then head over to the Langfuse dashboard. Open the "Tracing → Traces" tab:

Voilà—your entire chat session is logged, complete with trace-level details.

Just like that, observability is in place!

Going Further with Langfuse

Langfuse offers much more than just trace logging—it provides powerful tools to evaluate and improve your LLM application.

Users
Navigate to the Users tab to see a list of application users. You’ll notice a user named "airflow"—this represents interactions made through the airflow-chat plugin. Click on it to view all associated sessions. This is a great way to monitor how your team is using the plugin over time.

Cost Tracking
Langfuse makes it easy to track costs. In the Traces tab, you’ll see session-level costs, while the Home tab gives you an overview of model-level expenses. For deeper insights, head over to the Dashboards tab and explore the built-in Langfuse Cost Dashboard, or create a custom dashboard with your own cost graphs.

If you suspect that token prices are incorrect, go to Settings → Models, and either “Add model definition” or “Clone” an existing one to customize pricing.

Evaluations
Langfuse supports two types of evaluations: Human Annotation and LLM-as-a-Judge.

• Human Annotation
  Go to the Scores tab and create a new score. Next, create an Annotation Queue, and then add traces to the queue via the Traces tab. You can now manually evaluate each session. While this method offers fine-grained control, it can be time-consuming.

  

  

  

• LLM-as-a-Judge
  To automate evaluations, go to Settings → LLM Connection and add a model connection.
  Then create a custom evaluator by navigating to Create Custom Evaluator → Create New Evaluator, and set up a Default Evaluation Model.

  

  Choose a metric—such as Relevance—and specify whether you want to evaluate existing traces or only new ones.

  

  

  For inputs and outputs, map query to input and generation to output.

  

  Now, go back to Airflow and ask a question. Check the Evaluation Logs and you’ll see a score and comment like:

“The generation directly and comprehensively answers the user's query by listing all the DAGs, providing their names, and offering additional context about their purpose and scheduling. The response is focused, informative, and adds significant value to the user's understanding of the available DAGs in the Airflow environment.”

This demonstrates how Langfuse helps you evaluate, measure, and continuously improve your LLM-powered apps.

Setting Up Langfuse for Your LangChain Application

Integrating Langfuse into your LangChain app is straightforward. The configuration is handled in the airflow_chat/plugins/app/server/llm.py file, specifically within the get_user_chat_config function:

def get_user_chat_config(session_id: str, username: str = None) -> dict:
    chat_config = {
        'configurable': {'thread_id': session_id},
        'recursion_limit': 100
    }

    if os.environ.get('LANGFUSE_HOST'):
        langfuse_handler = CallbackHandler(
            user_id=username if username else session_id,
            session_id=str(session_id),
            public_key=os.environ.get('LANGFUSE_PUBLIC_KEY'),
            secret_key=os.environ.get('LANGFUSE_SECRET_KEY'),
            host=os.environ.get('LANGFUSE_HOST')
        )
        chat_config['callbacks'] = [langfuse_handler]

    return chat_config

This function builds the chat configuration and conditionally attaches the Langfuse callback handler if the environment variables are present.

The resulting configuration (chat_config) is then passed to the astream_events method like this:

async for event in self._agent.astream_events(
    {"messages": [HumanMessage(content=message)]},
    config=chat_session,
    version='v2',
)

The _agent object itself is initialized using the create_react_agent utility from LangGraph:

from langgraph.prebuilt import create_react_agent

self._agent = create_react_agent(
    self._llm,
    tools,
    checkpointer=checkpointer,
    prompt=PROMPT_MESSAGE
)

All of this logic can be found in the airflow_chat/plugins/app/server/llm.py file. This setup enables seamless event streaming and tracking via Langfuse within your LangChain-based application.

Summary

By combining the airflow-chat plugin with Langfuse, we’ve created a robust observability stack for LLM-powered workflows within Airflow. From real-time trace logging and cost tracking to advanced evaluation methods like LLM-as-a-judge, Langfuse brings clarity and control to your AI interactions. Whether you're debugging unexpected prompts, monitoring usage, or analyzing performance, Langfuse helps you iterate with confidence. For a deeper dive into all the features Langfuse offers, visit https://langfuse.com.

Introduction

Not long ago, we introduced our MCP for Airflow—designed to make operating DAGs and workflows as intuitive as chatting with a colleague. But there was a catch: it depended on Claude Desktop, which some organizations are unable or unwilling to adopt due to concerns around sharing data with Anthropic—or because of other operational, compliance, or policy-related reasons.

That got us thinking: what if we brought the same conversational interface directly into the Airflow UI?

That’s how the airflow-chat plugin came to life.

Under the hood, the plugin still leverages our existing MCP. You can run it fully embedded inside the Airflow webserver, or flexibly configure it in one of two other ways:

• As a plugin running inside the Airflow webserver (simplest setup)

• Using the plugin alongside a separate MCP service

• Or in a fully decoupled three-process architecture: Airflow webserver plugin → FastAPI server → MCP service

You also have full control over the LLM backend. MCP supports multiple providers, so you can choose to run it with Amazon Bedrock, Anthropic, or OpenAI—whichever best suits your infrastructure and policies.

This makes it easier than ever to integrate natural language interfaces into your pipelines—entirely within your stack and control.

Explore the project here:

• PyPI: airflow-chat

• GitHub: airflow-mcp-plugins

Quickstart

If you don’t have your own Airflow cluster, feel free to use the one provided in our GitHub repository.

Let’s start with the simplest setup. To integrate our plugin, add the following to your Airflow environment:

airflow-chat==0.1.0a8

Note: Version 0.1.0a8 is the latest at the time of writing. Check our PyPI repository for updates before installing.

🛠 Required Environment Variables

Add the following environment variables to your cluster configuration:

LLM_MODEL_ID=anthropic:claude-3-5-haiku-20241022
AIRFLOW_ASSISTENT_AI_CONN=airflow:airflow@http://localhost:8080
ANTHROPIC_API_KEY=your_anthropic_key

AIRFLOW_ASSISTENT_AI_CONN Explained

This connection string is used by the AI Assistant plugin to communicate with your Airflow instance. It follows this structure:

AIRFLOW_ASSISTENT_AI_CONN=<username>:<password>@http://localhost:8080

• Replace <username> and <password> with any valid Airflow credentials you want the assistant to use.

• Do not change localhost:8080, as this address is required for internal routing within the containerized environment where the plugin operates.

🔁 Model Flexibility:
You can switch LLM_MODEL_ID to use models from Bedrock or OpenAI. Simply change the prefix accordingly:

• For Bedrock: bedrock:...

• For OpenAI: openai:...

Be sure to set the correct corresponding API credentials:

• Anthropic: ANTHROPIC_API_KEY

• Bedrock: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

• OpenAI: OPENAI_API_KEY

⚙️ Additional Configuration

You can also configure optional Airflow MCP environment variables such as:

• AIRFLOW_INSIGHTS_MODE

• POST_MODE

We explain these options in detail in this blog post.

🧪 Running with Our Default Environment

If you’re using our pre-configured environment:

cp airflow-plugin-test.env airflow-plugin.env

Then open airflow-plugin.env, fill in all required values, and run:

just airflow

🧭 Accessing the Assistant

Once your cluster is running, open

 http://localhost:8080

, then navigate to:

Tools -> AI Chat Assistant

You’ll now see the Airflow Chat Assistant in action! Try asking:

"What Airflow DAGs do we have, and which ones are paused?"

🎉 Everything should work beautifully out of the box!

🔐 Access Control

Access to the AI plugin is restricted to:

• Admins, or

• Any user assigned the AIRFLOW_AI role.

Other Deployment Strategies

Let’s take a look at how our simple deployment works—and then walk through how to modify it depending on your needs.

🧩 The Basic Setup

Our lightweight Airflow Chat plugin runs both the Langchain modules and Airflow MCP locally inside the Airflow Webserver. This setup works out-of-the-box with minimal configuration.

But what if you don’t want to manage environment variables inside the Airflow instance under the same user? No problem—you can split the deployment and run MCP externally.

🚀 Option 1: Externalize the MCP Server

To run MCP as a separate component:

1. Remove the embedded MCP server from your Airflow Webserver setup.

Add these environment variables to your Airflow instance (you can use airflow-plugin.env in our repo):

TRANSPORT_TYPE=sse
mcp_host=host.docker.internal:8001
MCP_TOKEN=test_token

📌 Note: mcp_host must be updated if you're deploying the MCP server on your own infrastructure.

Remove the Airflow connection variable: AIRFLOW_ASSISTENT_AI_CONN

2. If you're using our repository, just run:

cp airflow-mcp-test.env airflow-mcp.env

Then, customize the following variables as needed (no changes required if using the default setup in our repo):

LLM_MODEL_ID=antropic:claude-3-5-haiku-20241022
AIRFLOW_ASSISTENT_AI_CONN=airflow:airflow@http://airflow-webserver:8080
TOKEN=test_token
TRANSPORT_TYPE=sse

✅ Ensure that TOKEN and MCP_TOKEN have the same value to establish a secure connection between Airflow and the remote MCP server.

To deploy:

• Run just prod if you're using our repo, or

• Use the Docker image: hipposysai/airflow-mcp:latest

• Refer to docker-compose.prod.yml for an example configuration.

🖥️ Option 2: Use the Plugin as a UI Only (Separate Langchain Logic)

If you prefer to fully decouple Langchain logic from the Airflow Webserver (and just use the plugin as a UI layer), update your airflow-plugin.env as follows:

INTERNAL_AI_ASSISTANT_SERVER=false
FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'
BACKEND_URL=http://chat_fastapi:8080

Then, for the FastAPI/Langchain service (using the hipposysai/airflow-mcp-fastapi image), define:

LLM_MODEL_ID=antropic:claude-3-5-haiku-20241022
SECRET_KEY='ThisIsATempSecretForLocalEnvs.ReplaceInProd.'
FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'
TRANSPORT_TYPE=sse
MCP_TOKEN=test_token
mcp_host=mcp_sse_server:8000
POSTGRES_USER='airflow'
POSTGRES_PASSWORD='airflow'
POSTGRES_DB='postgres'

To apply this config:

cp langchain-fastapi-test.env langchain-fastapi.env
just prod

⚖️ Option 3: Separate Backend but Share MCP

Want to offload backend logic from your Airflow Webserver without introducing a separate MCP service? You can:

1. Remove TRANSPORT_TYPE=sse from langchain-fastapi.env (or whichever file defines your backend).

2. Copy all environment variables from your MCP service (airflow-mcp.env) to your backend environment.

This option gives you better performance isolation without significantly increasing deployment complexity.

Summary

The airflow-chat plugin brings a conversational AI interface directly into the Airflow UI, enabling users to inspect DAGs, query task states, and operate pipelines using natural language. Built on top of the existing MCP architecture, it offers flexible deployment options—from running entirely inside the Airflow Webserver to more decoupled setups involving external MCP and FastAPI services. The plugin supports multiple LLM backends (Anthropic, Bedrock, OpenAI), giving teams full control over model providers and data handling. With minimal configuration, customizable environment files, and built-in access controls, airflow-chat makes AI-powered orchestration both accessible and secure—right where data pipelines live.

Introduction

A few weeks ago, we introduced Airflow MCP from Hipposys. Since then, we've fine-tuned its capabilities to better align with real-world user needs. In this post, we’ll walk you through three major updates: Safe Mode for secure operation, enhanced controls to manage DAG state, and a new way to predict upcoming DAG runs using our Airflow Schedule Insights plugin.

Run with Confidence: Safe Mode

We realized that many users—especially those working with production Airflow instances—need to be certain that MCP won’t inadvertently alter DAG states. Even if your Airflow user has permission to run, pause, or delete DAGs, you may want to use MCP strictly for read-only access. That’s why Safe Mode is now the default behavior.

In Safe Mode, only informational (GET) functions are available, ensuring you can inspect your Airflow environment without risk. To enable write capabilities (e.g. triggering or modifying DAGs), you can explicitly opt in by setting the POST_MODE environment variable.

To do this, add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "airflow_mcp": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "airflow_api_url",
        "-e",
        "airflow_username",
        "-e",
        "airflow_password",
        "-e",
        "POST_MODE",
        "hipposysai/airflow-mcp:latest"
      ],
      "command": "docker",
      "env": {
        "airflow_api_url": "http://host.docker.internal:8088/api/v1",
        "airflow_username": "airflow",
        "airflow_password": "airflow",
        "POST_MODE": "true"
      }
    }
  }
}

Once enabled, you’ll have access to all functions—including state-changing operations.

Until then, rest assured: by default, MCP operates in a safe, read-only mode, so you can explore and monitor without risking production disruptions or unscheduled runs.

Pause DAGs Before the Storm

We've all been there: an unexpected (or even expected) outage hits, and you need to urgently pause all Airflow DAGs. So you go through them one by one… but wait—some DAGs were already paused. Should you manually note them down just to restore their state afterward? That’s tedious—and error-prone.

Not anymore.

Thanks to the new change_dags_pause_status method, this process is now streamlined and reliable.

Note: The Airflow cluster I'm working with is deployed from our GitHub repo: hipposys-ltd/airflow-mcp.

Right now, here’s our current state:

Let’s use MCP to prepare for the outage.
Start with the following prompt:

"Hi, please pause all Airflow DAGs, but before doing so, list the DAGs that are currently paused. We’ll use that list to remember which ones need to be unpaused tomorrow."

As you can see, all DAGs have been successfully paused.

Now comes the tricky part: restoring normal operations without unpausing DAGs that were paused before the outage.

Run:

"Great. Now please unpause the DAGs, but make sure to leave the ones that were already paused as they are."

And just like that—done.
It only took two prompts to handle a full-cluster pause and restore cycle, cleanly and safely.

When Will It Run?

Let’s start with something simple: finding out when load_ticket_sales will run next.

To do that, we’ll need to update claude_desktop_config.json once again by enabling Airflow Insights Mode:

{
  "mcpServers": {
    "airflow_mcp": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "airflow_api_url",
        "-e",
        "airflow_username",
        "-e",
        "airflow_password",
        "-e",
        "POST_MODE",
        "-e",
        "AIRFLOW_INSIGHTS_MODE",
        "hipposysai/airflow-mcp:latest"
      ],
      "command": "docker",
      "env": {
        "airflow_api_url": "http://host.docker.internal:8088/api/v1",
        "airflow_username": "airflow",
        "airflow_password": "airflow",
        "POST_MODE": "true",
        "AIRFLOW_INSIGHTS_MODE": "true"
      }
    }
  }
}

⚠️ Note: Make sure the Airflow Schedule Insights plugin is installed in your Airflow instance for this to work.

Now, restart Claude Desktop and ask:

“When is the next run for load_ticket_sales (in EST)?”

After a short wait, you’ll get a detailed answer including the start time, end time, and expected duration—and it’s accurate. But that’s easy—load_ticket_sales is a scheduled DAG.

Let’s go a step further and try something more dynamic.

Event-Driven DAG Prediction

Now let’s look at an event-driven DAG with no fixed schedule: transform_customer_sentiment.

This DAG is triggered when:

• either transform_sales_aggregator and load_customer_feedback complete,

• or load_ticket_sales finishes on its own.
  

  

According to Claude, the next trigger will come from load_ticket_sales. Let’s wait and see…

And yes—it was triggered just one second later than predicted and finished two seconds later than expected. Even better, Claude correctly identified that load_ticket_sales was the trigger.

Querying Multiple DAGs at Once

Now let’s try asking about several DAGs in one go:

“When are the next runs for load_movie_inventory, load_box_office_reports, and transform_loyalty_program_analysis (in UTC)?”

Claude replies—none of them will run, and that’s correct!

With just a few configuration changes, you can now monitor and predict future DAG runs directly from Claude Desktop—scheduled or event-driven. No guesswork, no surprises.

Summary: Next-Level DAG Observability and Control

With the latest updates to Airflow MCP, Claude Desktop evolves from a smart interface into a powerful control and observability tool for Airflow users. You can now operate in Safe Mode for peace of mind in production, pause and restore DAG states safely during outages, and even predict when DAGs—scheduled or event-driven—will run next, thanks to the new Airflow Schedule Insights integration. Whether you're running a complex pipeline or preparing for maintenance, these upgrades ensure you're always one step ahead—with just a couple of prompts.