If you’ve ever onboarded a new data engineer onto an Airflow project, you know the drill. Half a day disappears into Python version mismatches, missing system dependencies, conflicting virtual environments, and the classic “it works on my machine” shrug. Local Airflow development is painful — and it doesn’t have to be.

This post walks through how we use Coder to give every developer a fully isolated, reproducible Airflow environment that spins up in minutes. We’ll use the airflow-mcp-plugins repo as our example — a project that adds a conversational AI interface directly into the Airflow UI – https://newsletter.ponder.co/p/airflow-chat-conversational-ai-built .

The Problem with Local Airflow

Just yesterday I ran into someone who told me they’d been a Dagster advocate for years — not because of the technology itself, but because local development just felt easier there. Airflow had a reputation for being painful to run locally, and that reputation stuck even as Airflow improved significantly over time. I love Airflow, and it bothered me that something as fixable as local dev experience was pushing people toward other tools. This post is partly my answer to that.

Running Airflow locally sounds simple until it isn’t. The typical setup involves Docker Compose with multiple services — webserver, scheduler, worker, Redis, Postgres — all needing to talk to each other, share volumes, and pick up your DAG files. On top of that:

• Everyone needs the same Python version and the same packages

• Port conflicts are common if you’re running multiple projects

• When you stop and restart, state can get corrupted

• New team members spend hours just getting to a working baseline

The usual answer is “just use Docker Compose directly.” But that still leaves every developer managing their own environment, their own .env files, their own credentials, and their own debugging when something breaks.

And then there’s the invisible problem: standardization drift. If you figure out a better way to set up the local environment — a smarter Docker Compose config, a new environment variable that everyone needs, a helper script that saves thirty minutes — how do you roll that out? You send an email. Which probably won’t be read. Or will be read, bookmarked, and forgotten. Six months later you discover half the team is still running the old setup because there was never a centralized, enforced way to define what “local development” actually means.

This challenge becomes even more apparent when supporting a broader internal development team. At Uncommon Schools, we work with engineers and developers across the data stack — from Airflow pipelines to dbt transformations and analytics tooling — with different roles, workflows, and levels of infrastructure familiarity. Before introducing Coder-driven workspaces, keeping everyone’s local setup aligned with the latest infrastructure changes was a recurring challenge.

The goal was to move away from onboarding docs, Slack messages, and tribal knowledge toward a development environment that is itself the source of truth: a reproducible workspace definition that every developer can spin up and keep in sync.

It gets worse when your stack grows beyond a simple Docker Compose. What if some components need to run directly on the laptop — a local Python process, a dbt command, a custom script that talks to both your local Airflow and a cloud service? Now you’re coordinating Docker networks, local processes, environment variables, and credentials across multiple tools, and the “just run this command” onboarding doc is already three pages long and six months out of date.

The Solution: Coder

Coder is an open-source platform for self-hosted remote development environments. Think of it as a control plane for developer workspaces — similar in spirit to Dev Containers, but more flexible. Where Dev Containers are tied to a specific editor and a specific container, Coder lets you define full environments that can run not just containers but also VMs or even bare-metal OS instances. You get a browser-based IDE, a terminal, and proxied access to any app running inside the workspace, all through a single URL.

You define a workspace as code using Terraform, and Coder provisions it on demand — locally or in the cloud. The key insight is that a “workspace” is just a compute resource (a Docker container, a VM, a Kubernetes pod) with a Coder agent running inside it. The agent connects back to the Coder server, which proxies your IDE, your apps, and your terminal outward. Every developer gets their own isolated environment, defined by the same Terraform template, with their own credentials injected at workspace creation time.

Coder supports a wide range of workspace types — AWS EC2 instances, Google Cloud VMs, Kubernetes pods, and more. For this tutorial we’ll use Docker, which is the simplest setup for local development and needs nothing beyond Docker Desktop on your Mac.

For Airflow development this is a perfect fit. Each workspace gets its own Docker daemon (Docker-in-Docker), its own Postgres, its own Redis, and its own Airflow stack — all isolated inside a single container that Coder manages. Want to change the default environment for the whole team? Update the Terraform template. Everyone’s next workspace picks it up automatically — no email required.

Running Coder Locally on Mac

First, install Coder:

curl -L https://coder.com/install.sh | sh

Then start the server. You need to pass both an access URL and a wildcard access URL — the wildcard is what allows Coder to proxy your Airflow UI through a subdomain:

coder server --access-url http://localhost:3000 --wildcard-access-url “*.localhost:3000”

If you’re using our repository, we’ve wrapped this in a just command:

just start_coder

Once it starts, open http://localhost:3000 in your browser. You’ll be prompted to create an admin account on the first run.

Creating Your First Workspace

After logging in:

Go to Templates and create a new template — paste in the coder_template.tf from the repository, then click on Build and Publish

Click Create Workspace

Fill in the parameters — your Git name, your Anthropic/OpenAI/AWS API key depending on which LLM backend you want to use

Hit Create and watch the build logs

The first time takes a few minutes because Docker images need to be pulled. On subsequent starts it’s fast — the Docker image cache is persisted in a named volume so nothing gets re-downloaded.

Once the workspace is running you’ll see two buttons: code-server (VS Code in the browser) and Airflow.

Click Airflow and you’ll land directly on the Airflow UI at http://airflow--yourusername--coder.localhost:3000/. Log in with airflow / airflow.

How Terraform Works in Coder

Every Coder workspace is defined by a Terraform template. When you create a workspace, Coder runs terraform apply with your parameters, which provisions the Docker container, volumes, and agent. When you stop the workspace, it runs terraform destroy on the container (but keeps the volumes). When you restart, it re-applies — same volumes, same data, new container.

The template we use here does a few things:

• Declares input parameters (coder_parameter) that become form fields in the UI — your API keys, your Git name, your LLM model choice

• Creates Docker volumes for /home/coder (your code and env files) and /var/lib/docker (the inner Docker image cache)

• Spins up a privileged workspace container with its own Docker daemon inside (DinD)

• Runs a startup script that installs Docker Engine, clones the repo, writes airflow-plugin.env from your parameters, and launches just airflow

• Registers a coder_app that tells Coder where to proxy the Airflow UI

For private repositories, you add a gitlab_token or github_token parameter and embed it in the clone URL:

git clone “https://oauth2:${TOKEN}@gitlab.com/yourorg/yourrepo.git”

The token gets injected at workspace creation time and never lives anywhere outside the container.

Walking Through the Template

Let’s look at what the template actually does, piece by piece.

Parameters are declared at the top as coder_parameter blocks. Each one becomes a form field when a developer creates a workspace. Here we collect the Git name, the LLM model ID, and whichever API keys are needed depending on whether you’re using Anthropic, OpenAI, or Amazon Bedrock. Parameters can have defaults, descriptions, and validation — they’re the clean interface between “what Coder knows” and “what the developer provides.”

Volumes come next. We create two named Docker volumes per workspace. home_volume mounts at /home/coder and holds everything the developer touches — the cloned repo, the generated .env file, any local changes. docker_data_volume mounts at /var/lib/docker inside the workspace container, which is where the inner Docker daemon stores its image cache, container state, and named volumes (including Postgres data). Both volumes survive workspace stops and restarts, so your data is never lost between sessions.

The workspace container is a codercom/enterprise-base:ubuntu image running with privileged = true. The privileged flag is what makes Docker-in-Docker possible — it allows the container to run its own kernel namespaces and mount its own filesystems. The entrypoint creates the repo directory, fixes ownership, and then hands off to the Coder agent init script. Notice the replace() call that rewrites localhost to host.docker.internal — this ensures the agent can phone home to the Coder server running on your Mac, not try to reach itself.

The startup script runs once the agent is up. It installs Docker Engine from the official Docker apt repository (not the system package, which is often outdated), adds the coder user to the docker group, starts the daemon, clones the repo, and then writes airflow-plugin.env by copying the test env file and injecting each parameter value using a delete-then-append pattern:

sed -i ‘/^ANTHROPIC_API_KEY=/d’ airflow-plugin.env
echo “ANTHROPIC_API_KEY=${data.coder_parameter.anthropic_api_key.value}” >> airflow-plugin.env

This is more reliable than sed replace because it works regardless of whether the key already exists or is blank in the source file. We also add a blank line after the copy step — without it, if the source file has no trailing newline, your first appended key gets concatenated onto the last existing line.

Finally, the script runs exec sg docker “cd $HOME/$REPO_DIR && just airflow”. The sg docker part activates the docker group membership immediately without needing a new login session — a common gotcha with Docker-in-Docker setups where usermod -aG docker doesn’t take effect until the next login.

The coder_app resource registers the Airflow UI with Coder. With subdomain = true, Coder proxies it through http://airflow--yourusername--coder.localhost:3000/ — the wildcard DNS entry you configured when starting the server. The URL points to localhost:8088 because the repo’s docker-compose maps the webserver’s internal port 8080 to 8088 on the workspace container’s network.

One thing worth noting: this setup runs the full Docker daemon inside every workspace container, and each workspace pulls and caches its own images. For a small team of engineers it’s perfectly fine. For a setup with hundreds of developers, you’d want to think about shared image registries, resource limits, and a proper Coder deployment on a server rather than everyone’s laptop. We’ll cover exactly that in a future post — different workspace architectures, how to right-size environments for different roles, and how to deploy Coder in production so your whole team can use it without running anything locally at all.

Workspace Architecture

Here’s what’s actually running when your workspace is up:

Mac (host)
└── Docker Desktop
    └── coder-yourusername-myworkspace (workspace container, privileged)
        └── Inner Docker daemon (/var/lib/docker on persistent volume)
            ├── airflow-webserver  → exposed on workspace's localhost:8088
            ├── airflow-scheduler
            ├── airflow-worker
            ├── postgres           → state persists across restarts
            └── redis:7

The workspace container runs with privileged = true, which lets it run its own Docker daemon. All the Airflow containers are siblings inside that inner daemon — they share a bridge network and can reach each other by service name. From the outside, Coder’s agent proxies localhost:8088 on the workspace container through the wildcard subdomain to your browser.

Because /var/lib/docker is a named Docker volume, the inner daemon’s image cache survives workspace restarts. The first cold start pulls ~1GB of Airflow images. Every restart after that is fast.

Because /home/coder is also a named volume, your cloned repo, your airflow-plugin.env, and any local changes persist between restarts too.

What’s Coming Next

This is the first post in a series on managing data engineering development environments with Coder. Coming up:

• dbt workspaces — same pattern, different stack, with dbt docs served as a Coder app

• AI-assisted development with LiteLLM — routing Claude Code through a self-hosted LiteLLM proxy so your API keys stay centralized and usage is tracked per developer

• Multiple workspace types — how to run lightweight workspaces for analysts alongside heavy workspaces for engineers, all from the same Coder deployment

• Production deployment — moving from coder server on your laptop to a proper deployment behind a reverse proxy, with wildcard DNS, persistent Postgres, and workspace autostop policies to keep costs down

The full template is in the repository. If you run into issues, the most common ones are covered in the troubleshooting section of the README.

Rebuilding the same semantic layer over and over across different BI tools has always felt unnecessarily painful. Most dashboards still reach directly into databases, each with their own idea of what a “metric” is, so you end up maintaining the same logic in three or four places.

Late last year, out of frustration, we built dbt-to-cube to see whether dbt, Cube.js, and Superset could be wired together into a single automated flow. It worked: define metrics once in dbt, let a small tool translate them to Cube.js, and push them into your BI tool of choice—giving you a unified semantic layer across Superset, Tableau, or PowerBI without manual duplication.

But as we started relying heavily on this code-first workflow at Ponder Labs, a glaring logical gap emerged.

It’s great that downstream BI tools can query these unified metrics. But what about using those metrics inside dbt itself? If I wanted to build an aggregated reporting mart or feed a downstream process within my dbt pipeline, I still had to manually rewrite the SUM() or AVG() aggregations in my SQL, breaking the “define it once” rule. I wanted the metrics defined for Cube to be queryable natively in my dbt DAG.

That’s exactly why I built the next piece of the puzzle: dbt_simple_metrics.

The Solution: Metrics as Code Inside dbt

dbt_simple_metrics is a lightweight dbt package that lets you generate aggregated SQL from metric definitions sitting directly inside your model YAML files. Instead of navigating a heavy, standalone semantic layer setup, your columns and metrics live together, and you reuse them across your marts with a single macro call.

In our simplified format, we keep things incredibly close to the data. You define the dimensions under columns, and your measures under config.meta.metrics. It looks like this:

version: 2
models:
  - name: course_performance_summary
    columns:
      - name: course_name
      - name: academic_year
      - name: total_enrollments
    config:
      meta:
        metrics:
          total_course_enrollments:
            type: sum
            label: “Total Course Enrollments”
            sql: ${total_enrollments}

Now, instead of manually writing aggregations in your downstream models, you simply call the macro. The package validates the dimensions and metrics at compile time and generates the proper GROUP BY SQL:

{{ config(materialized=’table’) }}
{{ dbt_simple_metrics.metrics(
    ‘course_performance_summary’,
    [’course_name’, ‘academic_year’],
    [’total_course_enrollments’]
) }}

If you misspell a dimension or an unknown metric is requested, you get a clear compile-time error rather than a runtime failure.

The Elephant in the Room: MetricFlow

If you are immersed in the dbt ecosystem, you are likely wondering: Doesn’t dbt already have a Semantic Layer with MetricFlow?

It does. In fact, if you look at the semantic_models.yml file in the dbt_simple_metrics repository, you’ll see we actually included the MetricFlow definitions. We added them specifically so you can still see your metrics natively in dbt docs and the dbt Cloud catalog.

However, that file is an example of the legacy MetricFlow format. And honestly, it highlights exactly why we built our package. The old spec forces you to define a top-level semantic_models block entirely separate from your actual data models, with verbose, disconnected metrics configurations that look like this:

# The Legacy MetricFlow Format
metrics:
  - name: total_semester_headcount
    type: simple
    type_params:
      measure: total_unique_students
  - name: deans_list_rate
    type: ratio
    type_params:
      numerator: total_deans_list
      denominator: total_semester_headcount

It works for populating a catalog, but it feels detached from the code-first, model-centric workflow developers actually want.

Ahead of the Curve: Preparing for MetricFlow v2.0

Here is the good news: dbt Labs recognizes this friction.

With the upcoming release of dbt 1.12, they are stabilizing MetricFlow spec version 2.0. If you look at the differences, dbt is completely overhauling their approach to look... exactly like what we are already doing. They are abandoning the disconnected top-level wrappers and moving to embed semantic annotations (like semantic_model and simple metrics) directly under the models: block.

By adopting dbt_simple_metrics today, you aren’t fighting against the dbt ecosystem; you are skating to where the puck is going. You get the clean, embedded syntax that dbt is moving toward, with the immediate benefit of native dbt querying and automated syncing to Cube.js today.

And naturally, once dbt 1.12 becomes stable, we will be updating the package to ensure full, seamless compatibility with the final v2.0 spec.

Quickstart & Installation

To start using it in your dbt project and close the loop on your semantic layer, add the git repository and the specific revision to your packages.yml:

packages:
  - git: “https://github.com/ponderedw/dbt_simple_metrics.git”
    revision: v1.0.3

Run dbt deps, and you’re ready to start calling the macro in your marts.

At Ponder Labs, we are constantly pushing the boundaries of what code-first orchestration can do, and eliminating redundant metric definitions has been a massive win for our data engineering workflows.

Give the package a try, and let me know what you think on GitHub—PRs, feedback, and feature requests are always welcome!

Introduction

Data ingestion is rarely glamorous — but getting it right is everything. In a previous post, we explored batch data ingestion using Trino and dbt with a relational database as the source. But real-world data pipelines are messier: files land in S3 buckets from various sources, in various formats, and you need a reliable, manageable way to bring them into your Data Lake.

The natural question is: why not just create an external schema and table directly? You absolutely can — and in many cases, you should. But there are scenarios where that approach introduces more friction than it solves:

• Managing permissions between database users, IAM roles, and S3 buckets can become a tangled, error-prone mess

• Each warehouse (Redshift, BigQuery, Snowflake) handles external tables differently, making your pipeline harder to standardize

• Ingestion logic scattered across SQL scripts and cloud consoles is difficult to version, test, and maintain

• Onboarding new data sources requires touching infrastructure, not just code

We’re also familiar with the dbt-redshift external tables package, and while it’s a clever solution, it introduces a fundamental design concern: dbt shouldn’t be responsible for managing sources. Sources are meant to be stable, pre-existing references that dbt reads from — not objects that dbt creates or controls. When dbt starts owning source definitions, the line between infrastructure and transformation blurs, making pipelines harder to reason about, debug, and hand off to other teams. Models should run and transform; sources should simply exist.

Why Hive?

Apache Hive has been around since 2008, originally developed at Facebook to bring SQL-like querying to Hadoop’s distributed file system. At its core, Hive introduced a revolutionary idea: separate the metadata (where the data lives, how it’s structured, what format it’s in) from the data itself. This metadata layer — the Hive Metastore — became the backbone of the Hadoop ecosystem and eventually the de facto standard for Data Lake table management, adopted by virtually every major distributed query engine, including Spark, Presto, and later Trino.

Over time, more modern table formats emerged to address Hive’s limitations. Apache Iceberg, for example, was designed to handle the pain points that Hive struggled with at scale: full ACID transactions, reliable schema evolution, time travel, and efficient handling of billions of files. Iceberg is undeniably powerful — and if you’re building a large-scale, write-heavy lakehouse with complex update and delete patterns, it’s likely the better choice.

But here’s the thing: Hive’s simplicity is a feature, not a bug. Iceberg’s richness comes with real operational overhead — more configuration, more moving parts, and a steeper learning curve. When your goal is straightforward S3 file ingestion without complex upserts or time travel requirements, Iceberg can feel like bringing a sledgehammer to crack a nut. Hive’s loss in the feature war is our gain in the simplicity game.

This is where Hive shines. By leveraging Hive Metastore alongside AWS Glue Catalog as the metadata layer, we can define, manage, and transform S3 data in a consistent, code-first way — entirely within dbt. No console gymnastics, no permission rabbit holes. Sources remain untouched and independently managed, while dbt focuses on what it does best: running transformations.

That said, Hive isn’t a silver bullet:

• It adds an extra layer to your stack, which means more components to monitor and maintain

• Query performance on very large, unpartitioned datasets can lag behind native warehouse solutions

• It works best when you’re already operating within the AWS ecosystem — outside of it, the setup overhead may not be worth it

• Schema evolution (adding/changing columns) requires careful handling to avoid breaking downstream models

If you’re already working in AWS and want to manage your ingestion logic cleanly in dbt, Hive is a natural, battle-tested fit. Let’s walk through how we set it all up.

Hive-Trino Setup

With the “why” out of the way, let’s get our hands dirty. The setup boils down to two things: telling Trino how to talk to Hive via AWS Glue, and giving the right IAM permissions to make it all work.

1. Adding the Hive Catalog to Trino

Trino uses catalog configuration files to connect to different data sources. To wire up Hive with AWS Glue as the metastore, add the following catalog:

s3_source_files_hive: |
    connector.name=hive
    hive.metastore=glue
    hive.metastore.glue.aws-access-key=${aws_access_key_hive}
    hive.metastore.glue.aws-secret-key=${aws_secret_key_hive}
    hive.metastore.glue.region=us-east-1
    fs.native-s3.enabled=true
    s3.aws-access-key=${aws_access_key_hive}
    s3.aws-secret-key=${aws_secret_key_hive}
    s3.region=us-east-1
    s3.path-style-access=true

A few things worth noting here:

• hive.metastore=glue is the key line — it tells Trino to use AWS Glue Catalog instead of a self-hosted Hive Metastore, which saves you from spinning up and maintaining yet another service

• fs.native-s3.enabled=true enables Trino’s native S3 filesystem, which is more performant and reliable than the legacy Hadoop-based S3 driver

• Credentials are injected via environment variables, keeping secrets out of your config files — make sure these are properly set in your Trino deployment

2. IAM Policy

The catalog configuration alone isn’t enough — the IAM user referenced by those credentials needs the right permissions to both read/write from S3 and interact with the Glue Catalog. Attach the following policy:

{
    “Version”: “2012-10-17”,
    “Statement”: [
        {
            “Sid”: “S3DataReadWrite”,
            “Effect”: “Allow”,
            “Action”: [
                “s3:GetBucketLocation”,
                “s3:ListBucket”,
                “s3:GetObject”,
                “s3:PutObject”,
                “s3:DeleteObject”
            ],
            “Resource”: [
                “arn:aws:s3:::source-files-hive”,
                “arn:aws:s3:::source-files-hive/*”
            ]
        },
        {
            “Sid”: “GlueMetadataReadWrite”,
            “Effect”: “Allow”,
            “Action”: [
                “glue:GetDatabase”,
                “glue:GetDatabases”,
                “glue:CreateDatabase”,
                “glue:GetTable”,
                “glue:GetTables”,
                “glue:CreateTable”,
                “glue:UpdateTable”,
                “glue:DeleteTable”,
                “glue:GetPartition”,
                “glue:GetPartitions”,
                “glue:CreatePartition”,
                “glue:BatchCreatePartition”,
                “glue:DeletePartition”,
                “glue:BatchDeletePartition”
            ],
            “Resource”: [
                “arn:aws:glue:us-east-1:<account_id>:catalog”,
                “arn:aws:glue:us-east-1:<account_id>:database/s3_source_files_hive”,
                “arn:aws:glue:us-east-1:<account_id>:table/s3_source_files_hive/*”
            ]
        }
    ]
}

The policy is split into two clear responsibilities:

• S3DataReadWrite — grants access to the source-files-hive bucket, which is where your raw source files will be initially dropped. Trino needs read access to query them and write access to manage any intermediate outputs

• GlueMetadataReadWrite — grants Trino the ability to interact with Glue Catalog: reading and creating databases, managing tables, and handling partitions. Notice the resources are intentionally scoped to the s3_source_files_hive database only — following the principle of least privilege, we don’t hand out catalog-wide permissions

💡 Tip: Replace <account_id> with your actual AWS account ID before applying the policy. If you’re managing this via Terraform or CloudFormation, this is a good candidate for a variable substitution.

With the catalog registered and IAM permissions in place, Trino can now discover and query any Hive tables registered in your Glue Catalog. Next, let’s look at how dbt fits into the picture.

dbt Integration

Now that Trino can talk to Glue and read from S3, we need to bridge the last gap: making dbt aware of our Hive tables and using them as sources for transformations. The elegant part of this approach is that we don’t register tables manually — we create them dynamically using a dbt macro, triggered as a pre_hook before the model runs.

1. The Macro

Add the following macro to your dbt Trino project:

{% macro create_external_hive_table(model_name, schema_definition, s3_location) %}
    {% set target_relation = “s3_source_files_hive.s3_source_files_hive.” ~ model_name %}
    {% set sql %}
        DROP TABLE IF EXISTS {{ target_relation }};
        CREATE TABLE {{ target_relation }} (
            {{ schema_definition }}
        )
        WITH (
            external_location = ‘{{ s3_location }}’,
            format = ‘CSV’,
            skip_header_line_count = 1
        )
    {% endset %}
    {% if execute %}
        {% do run_query(sql) %}
        {{ log(”Created Hive Table: “ ~ target_relation, info=True) }}
    {% endif %}
    {{ return(”SELECT 1”) }}
{% endmacro %}

Let’s break down what’s happening here:

• target_relation — builds the fully qualified table name in the format catalog.database.table, pointing directly at our s3_source_files_hive catalog and database

• DROP TABLE IF EXISTS — ensures idempotency. Every time the model runs, the Hive table is recreated fresh. This is intentional: if the S3 files or schema change, the table definition stays in sync

• CREATE TABLE ... WITH (...) — this is a Hive external table definition. The key properties are:

  • external_location — the S3 path where the raw files live. Hive doesn’t move or copy the data, it simply points to it

  • format = ‘CSV’ — tells Hive how to parse the files. This can be swapped for ORC, PARQUET, JSON, etc. depending on your source files

  • skip_header_line_count = 1 — skips the first row of the CSV, assuming it contains column headers

• {% if execute %} — a dbt guard that prevents the macro from running during the parsing/compilation phase, only firing during actual execution

• run_query(sql) — executes the DDL statement against Trino

• return(”SELECT 1”) — macros used in pre_hook must return a valid SQL string. Returning SELECT 1 is a clean no-op that satisfies this requirement without side effects

2. The Model

With the macro in place, here’s how a dbt model using it looks:

{%- set hive_table = “table_name” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                column1 VARCHAR,
                column2 VARCHAR,
                column3 VARCHAR,
                column4 VARCHAR”,
            s3_location=”s3://source-files-hive/table_folder/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

Here’s the flow this model follows when dbt runs it:

1. pre_hook fires — before anything else, create_external_hive_table is called, dropping and recreating the Hive external table in Glue Catalog, pointed at the S3 location

2. The model runs — dbt executes the SELECT statement, reading directly from the freshly registered Hive table, which transparently reads the CSV files from S3

3. current_date as ingested_at — a small but important addition: we stamp every ingested row with the current date, giving us a simple audit trail of when the data was loaded

💡 Tip: Notice that the model name is defined once as a variable (hive_table) and reused in both the config block and the FROM clause. This keeps things DRY — if you rename the table, you change it in one place only.

This pattern keeps the separation of concerns intact: the macro handles infrastructure (table registration), and the model handles transformation. dbt does what dbt is supposed to do — and nothing more.

A Real-World Example: School Network Data Ingestion

To make this concrete, consider a large school network – say, 50+ schools across a district – already operating within AWS. Their data team is responsible for centralizing student, staff, and assessment data from a patchwork of source systems: a Student Information System, an HR platform, and several third-party assessment providers.

None of these systems share a common integration standard. What they do all share: a scheduled CSV export, dropped into S3.

Here’s what that might look like in practice:

s3://source-files-hive/

├── students/

├── staff/

└── assessments/

Each folder receives nightly file drops from the respective source system. The data team defines one dbt model per source, each invoking the same create_external_hive_table macro:

sql{%- set hive_table = “students” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                student_id VARCHAR,
                first_name VARCHAR,
                last_name VARCHAR,
                date_of_birth VARCHAR,
                enrollment_status VARCHAR,
                school_id VARCHAR”,
            s3_location=”s3://source-files-hive/students/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

sql{%- set hive_table = “staff” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                staff_id VARCHAR,
                full_name VARCHAR,
                role VARCHAR,
                department VARCHAR,
                school_id VARCHAR,
                employment_status VARCHAR”,
            s3_location=”s3://source-files-hive/staff/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

sql{%- set hive_table = “assessments” -%}
{{
    config(
        alias=hive_table,
        pre_hook=create_external_hive_table(
            model_name=hive_table,
            schema_definition=”
                assessment_id VARCHAR,
                student_id VARCHAR,
                subject VARCHAR,
                score VARCHAR,
                max_score VARCHAR,
                assessment_date VARCHAR,
                school_id VARCHAR”,
            s3_location=”s3://source-files-hive/assessments/”
        )
    )
}}
select
    *,
    current_date as ingested_at
from
    s3_source_files_hive.s3_source_files_hive.{{ hive_table }}

The result: three clean, independently managed source tables, registered automatically on each dbt run, with zero manual intervention. The data team can onboard a new source system – say, a library platform or a special education tracking tool – by adding a single model file and an S3 folder. No console access required, no infrastructure tickets, no cross-team dependency on a cloud engineer to wire up permissions.

Downstream, dbt models join students, staff, and assessments to produce analytics-ready datasets: attendance summaries, assessment score trends by school, staff-to-student ratios. The ingestion layer stays untouched – it simply exists, and the transformation layer builds on top of it.

Summary

Data ingestion doesn’t have to be complicated. In this post, we walked through a clean, practical approach to bringing raw S3 files into a queryable Data Lake — without overengineering the pipeline or blurring the lines between infrastructure and transformation.

Here’s what we covered:

• Why Hive over alternatives — Hive’s maturity and simplicity make it a natural fit for straightforward S3 ingestion. Where Iceberg wins on features, Hive wins on pragmatism. When you don’t need ACID transactions or time travel, there’s no reason to pay the complexity tax

• Why not external tables or dbt-redshift packages — external tables introduce permission headaches and lack standardization across warehouses. The dbt external tables package, while clever, violates a core dbt principle: sources should exist independently, not be managed by the transformation layer

• Trino + AWS Glue Catalog — by configuring a Hive catalog in Trino and pointing it at Glue, we get a fully managed, serverless metastore without spinning up any additional infrastructure. A tightly scoped IAM policy keeps things secure and follows the principle of least privilege

• Dynamic table creation with dbt macros — instead of manually registering Hive tables, a single reusable macro handles the DDL as a pre_hook, keeping table definitions in sync with your S3 files on every run

• Clean separation of concerns — the macro owns infrastructure, the model owns transformation. dbt stays in its lane, and your pipeline stays maintainable

The result is a lightweight, reproducible, and version-controlled ingestion pipeline that lives entirely in your dbt project — no console gymnastics, no permission rabbit holes, no overengineering.

If you’re already running Trino on AWS and want a simple on-ramp for S3 file ingestion, this stack is hard to beat. As always, the best pipeline is the one your team can actually understand, maintain, and extend — and this one fits that bill.

Introduction

Like many Charter Management Organizations (CMOs) across the U.S., Uncommon Schools relies on PowerSchool as its Student Information System (SIS) -- the system of record for student enrollment, demographics, attendance, grades, and scheduling. While PowerSchool is critical for day-to-day school operations, it wasn’t designed for fast, large-scale analytics use cases.

Under the hood, PowerSchool stores its data in an Oracle database, which makes extracting large tables -- like Students -- surprisingly slow when using conventional approaches.

In this post, I’ll walk through how we reduced ingestion time for the Students table from nearly 50 minutes to under a minute. We achieved this by migrating from a custom Python-based pipeline to Trino, enabling significantly faster data access and simpler ingestion logic. Along the way, I’ll share a few practical techniques to speed up data extraction -- not just in Oracle, but in any database system that supports procedural logic.

This work builds on ideas we explored in our earlier post, Trino + dbt: Simplifying ELT with Pure SQL, where we showed how combining Trino and dbt can simplify ingestion, improve reliability, and unify ingestion with transformation workflows. Here, we take that foundation further -- focusing on performance, real-world constraints, and how to make ingestion not just simpler, but dramatically faster.

The Original Pipeline: Where We Started

Our initial approach was built around a containerized extraction layer. We created a custom Docker image based on an Oracle Instant Client setup, which allowed us to connect directly to the PowerSchool Oracle database and run extraction queries. This image included Python, SQL*Plus, and a set of internal scripts used to execute parameterized queries and export data.

We orchestrated everything with Airflow, where each table ingestion was defined as a separate task in a DAG. Each task would spin up a container (via ECS/EKS) using our custom operator, ensuring that extraction workloads didn’t run on Airflow workers themselves but instead executed in isolated pods. This design gave us scalability and parallelism—but also introduced operational complexity.

To control what data gets extracted, we relied on SQL templates. Each table had its own query template, dynamically parameterized (e.g., by PowerSchool instance), and executed inside the container. For example, the Students table template included not only direct column selections, but also calls to Oracle PL/SQL functions like ps_customfields.getStudentsCF.

These functions are part of PowerSchool’s internal logic layer and allow access to data that isn’t physically stored in the base tables. While this gave us flexibility and completeness, it also came at a significant performance cost—each function call is effectively an additional computation executed per row.

Over time, several issues with this approach became clear:

• Lack of lineage and validation: Changes to SQL templates were difficult to track and validate against downstream models. A simple column removal could silently break transformations later in the pipeline.

• Operational overhead: Each table required its own Airflow task and container. With ~30 tables, this meant dozens of parallel pods, making logs harder to read and increasing pressure on orchestration infrastructure.

• Performance bottlenecks: The biggest issue—queries were slow. Very slow. Extracting the Students table for a single instance could take up to 50–60 minutes, largely due to heavy use of PL/SQL functions and row-by-row processing.

In short, while the pipeline worked, it wasn’t scalable, transparent, or fast enough for our needs.

The New Approach: Fast Ingestion with Trino

To address the limitations of our original pipeline, we leaned into the approach we described in our previous post, Trino + dbt: Simplifying ELT with Pure SQL. The core idea is simple but powerful: move ingestion logic into SQL, unify it with transformation workflows in dbt, and let Trino handle execution across systems.

This shift solved several problems at once.

First, unpredictable production issues became much easier to manage. By bringing ingestion into dbt, we gained lineage, dependency tracking, and the ability to validate changes before they reach production. Instead of discovering issues at runtime, we can now catch them during development or CI – exactly the gap we highlighted in our earlier work.

Second, we eliminated the overhead of “millions of tasks and pods.” Instead of running one Airflow task per table (and spawning dozens of containers), we now have a single Airflow task that triggers a dbt run. Within that run, dbt executes multiple models concurrently using threads – all inside a single pod. This dramatically simplifies orchestration, improves observability, and reduces load on both Airflow and our container infrastructure.

But the biggest win came from query performance.

In the new approach, the ingestion logic is written as a dbt model that uses Trino’s system.query to push down execution directly to Oracle. Instead of relying heavily on row-by-row PL/SQL function calls, we restructured the query to:

Replace expensive function calls where possible with joins (e.g., leveraging studentcorefields, vwstudentcorefieldsuc, and other tables instead of repeatedly calling ps_customfields.getStudentsCF)

The result is a much more efficient execution plan: fewer function calls, better use of joins, and improved parallelism across databases.

Combined with Trino’s ability to execute queries concurrently and dbt’s threaded execution model, this reduced the ingestion time for the Students table from nearly an hour to under a minute.

Just as importantly, the pipeline is now simpler, more maintainable, and fully transparent. Instead of spreading logic across Docker images, Airflow tasks, and SQL templates, everything lives in version-controlled dbt models—fully integrated with the rest of the data platform.

To tie everything together operationally, we also introduced a lightweight custom Airflow operator that runs dbt + Trino workloads inside a single Kubernetes pod:

Summary

Uncommon Schools’ data team cut ingestion time for PowerSchool student records from nearly an hour to under a minute by replacing a custom Python/Docker pipeline with Trino and dbt. The original setup relied on containerized Oracle extractions, row-by-row PL/SQL function calls, and one Airflow task per table — functional, but slow, hard to maintain, and operationally expensive. The new approach moves ingestion logic into dbt models, uses Trino to push queries directly down to Oracle, and replaces costly function calls with efficient joins. The result is a faster, simpler, and fully version-controlled pipeline where lineage, validation, and transformation all live in one place.

When we first introduced Airflow Schedule Insights, we told the story of Dr. Movie, a cinema chain struggling with a “scheduling black box.” They had DAGs that silently refused to run, resource bottlenecks from over-parallelization, and zero visibility into when their next asset-based trigger would actually fire.

That story helped explain the problem.

But with Airflow 3, we decided to evolve the example environment.

Instead of fictional cinema DAGs, we now ship example educational DAGs that mimic real academic data platform dependency patterns. These DAGs are not production pipelines — they simply replicate common educational orchestration structures and asset relationships so users can test complex scheduling scenarios in a realistic setting.

In other words:
They copy educational DAG dependency patterns — not institutional data.

This makes the demo environment more relatable for universities and training organizations, while remaining lightweight and safe to use as a sandbox.

And alongside that shift, Schedule Insights v0.2.0 is officially out — fully migrated, dependency-light, and optimized for Airflow 3.

pip install airflow-schedule-insights

👉 GitHub: https://github.com/ponderedw/airflow-schedule-insights

What Changed in v0.2.0 (Airflow 3 Migration)

Under the hood, 22 files changed. If you’re a plugin developer, these are the silent Airflow 3 metadata changes you need to understand.

1️⃣ The Great Renaming: Datasets → Assets

Airflow 3 formalized the shift from “datasets” to “assets.”

Old:

task_outlet_dataset_reference

New:

task_outlet_asset_reference

If your plugin queries the metadata DB directly, this rename will break you.

2️⃣ Logic Flip: is_active → is_stale

Airflow 2:

DagModel.is_active == True

Airflow 3:

DagModel.is_stale.is_(False)

The column is gone. The logic is inverted. If you don’t update this filter, your visibility logic silently fails.

3️⃣ Asset Dependencies Are Trees, Not Lists

Airflow 3 introduces conditional asset expression9s with nested AND / OR logic.

Example:

(Asset_A AND (Asset_B OR Asset_C))

This is no longer a flat dependency list.

We implemented a recursive evaluator that parses the new asset_expression JSON structure so Schedule Insights can still predict:

• When a DAG will run

• What it’s waiting on

• Which branch of logic is currently blocking execution

Without this, “next run” predictions are simply wrong.

4️⃣ Better Performance, Fewer Dependencies

We removed pandas and numpy.

Run duration medians are now computed directly in SQL using percentile_cont via SQLAlchemy.

Result:

• Faster installs

• Lower memory usage

• Cleaner dependency graph

Example Educational DAGs (Dependency Patterns Only)

For Airflow 3, the repository now includes an Academic Data Platform example DAG suite.

These DAGs simulate patterns such as:

• Ingestion → normalization → transformation → consolidation

• Dataset/asset-triggered execution

• Hybrid cron + asset scheduling

• Conditional AND/OR asset logic

• Reporting layers dependent on multiple upstream assets

They exist purely to demonstrate:

• Complex asset expressions

• Multi-layered DAG orchestration

• Real-world dependency graphs

No real institutional integrations.
No sensitive data.
Just realistic dependency topology.

This makes Schedule Insights far more meaningful to demonstrate than the original cinema example, especially when working with Airflow 3’s asset trees.

Core Features (Now Tested on Complex Asset Graphs)

🛠 The Next Run Table — No More Silent Failures

Asset-driven DAGs often “don’t run” for subtle reasons:

• Paused DAGs

• Waiting on asset expressions

• Hybrid cron + dataset logic

• Unmet upstream conditions

The Next Run table explains exactly why:

• “This DAG is paused and will not run.”

• “No schedule or dependencies found.”

• “Waiting for upstream Asset/DAG triggers.”

📊 The Schedule Impact Simulator

Want to change:

*/25 → */5

Before committing?

Simulate it.

The UI highlights impacted future runs in purple and shows how the change propagates through downstream asset dependencies.

This is especially useful in complex multi-layer DAG graphs.

🕒 Timezone-Aware Gantt Charts

The plugin detects your local timezone and renders:

• Historical runs

• Predicted runs

• Gantt charts

No more UTC mental math.

🔌 Programmatic Insights via API

We exposed the scheduling engine via FastAPI endpoints:

• get_future_runs_json

• get_next_future_run_json

Your internal tooling can now query predicted Airflow behavior programmatically.

Visualizing Structure with airflow-dag-dependencies

To complement Schedule Insights, you can use our companion plugin:

airflow-dag-dependencies

Schedule Insights tells you when a DAG will run.
airflow-dag-dependencies shows you how everything connects.

Together they provide:

• Structural visibility (dependency graph)

• Temporal visibility (future execution prediction)

Especially powerful with Airflow 3’s conditional asset trees.

Why This Matters at Scale

As more teams contribute DAGs to a shared Airflow environment — as is the case at Uncommon Schools, where multiple teams orchestrate data across a large network of schools — scheduling complexity compounds quickly. Each team adds their own timing requirements, and as run durations drift, overlap and resource contention become increasingly hard to spot from any single vantage point. Schedule Insights gives you a cross-team view — so you can see potential collisions before they become incidents. And when a DAG pile-up does occur, the historical run data lets you work backward and reconstruct what likely triggered the cascade.

Try It Today

Spin it up locally:

git clone https://github.com/ponderedw/airflow-schedule-insights
cd airflow-schedule-insights
docker compose -f local_airflow_3/docker-compose.yml up

Then navigate to:

Browse → Schedule Insights

You’ll see the plugin running against realistic educational dependency patterns — not fictional cinema workflows.

Airflow 3 made orchestration more powerful.
Schedule Insights makes it understandable.

Introduction

In our previous post, we explored how to bridge the gap between complex data structures and end-users by grounding natural-language queries in a Cube-governed semantic layer. By using RAG, we enabled anyone—from teachers to board members—to query data without needing to touch SQL, GraphQL, or REST APIs.

But while answering a question with text is a great start, data is often best understood visually. Even for those who can write code, the goal is often the same: getting to the insight as quickly as possible. We asked ourselves: Can we move beyond simple text responses and visualize this data directly within the chat interface?

The answer is a resounding yes.

In this post, we’re introducing a new capability that turns conversational insights into immediate, actionable BI charts. We’ve implemented this by leveraging Streamlit’s native charting capabilities combined with custom logic for parsing the AI’s response.

Of course, we aren’t alone in this pursuit—the entire industry is racing toward “conversational BI.” This is simply our take on the problem. Our hope is that this specific implementation provides a blueprint or some inspiration for bringing similar capabilities to your own organization. If you have a different approach or wider experience in this space, we’d love for you to share your thoughts in the comments.

You can find the updated code in the same repository here:

👉 GitHub: https://github.com/ponderedw/dbt-to-cube

Let’s dive into how we move from a simple chat response to a fully rendered chart.

One quick note before we dive in: all of the examples and visualizations in this post use synthetic sample data generated as part of the demo environment.

Quickstart

Ready to see it in action? You can get the entire environment running in just a few steps.

1. Clone the repository: git clone https://github.com/ponderedw/dbt-to-cube

2. Configure your environment: Create your .private.env file from the .private.env.template as explained here. This ensures your LLM provider and credentials are ready to go.

3. Launch the stack: Simply run just all. This automated workflow handles the heavy lifting: spinning up the database, generating mock data, running dbt models, and ingesting metrics into both Cube and Superset. (You can read more about the underlying architecture here).

4. Access the UI: Head over to http://localhost:8501/, enter your STREAMLIT_PASSWORD, and you’re ready to chat with your data.

From Questions to Visuals

Start by getting familiar with the metadata the RAG layer has indexed:

“Hey. What measures and dimensions do we have?”

Once the AI confirms the available metrics, it’s time to move beyond text. Let’s ask for a specific visualization:

“Can you visualize the total number of course enrollments by difficulty level using a pie chart?”

The result? A fully rendered BI graph in just 11 seconds.

Let’s try a different metric and a more dynamic chart type:

“Return the number of students with excellent performance across academic years and visualize it using a line chart.”

In seconds, you receive an amazing, interactive line chart—mapping complex trends without you ever touching a visualization library.

More to Explore

The system is flexible enough to handle various chart types and comparison logic:

• Area Charts: “Can you create an area chart showing the number of students with excellent performance compared to the total number of enrolled students across semesters?”
  

• Scatter Charts: “Can you visualize the relationship between course difficulty level and average course GPA using a scatter chart?”
  

How does it work?

To enable these visualizations, we didn’t need to reinvent the wheel. We kept the same agent-based architecture and Streamlit frontend but introduced two key modifications to the workflow:

1. Enhancing the Prompt (The “Brain”)

We updated our LangChain prompt to give the agent “visual awareness.” It now understands how to structure its output so the frontend knows when a chart is requested.

• Where to look: You can find the specific instructions we added in cube-to-rag/app/server/chat.py.

2. Intelligent Response Parsing (The “Bridge”)

Once the AI generates a response, the Streamlit application needs to know whether to display a paragraph of text or render a dynamic chart. We updated the handling logic to parse the AI’s response and trigger the appropriate Streamlit chart component.

• Where to look: The updated logic is located in streamlit/Main.py.

By separating the “intent” (in the prompt) from the “rendering” (in Streamlit), we maintain a clean separation of concerns while delivering a seamless user experience.

Conclusion

This experiment marks a significant step in making data truly accessible. By moving from simple text-based answers to dynamic BI visualizations, we’ve closed the loop between asking a question and gaining a visual insight.

The beauty of this approach lies in its governance. Because the AI is still grounded in the Cube semantic layer, the charts aren’t just “hallucinated” guesses—they are visual representations of verified business logic. Whether it’s a school administrator tracking student performance or a developer looking for a quick metric check, the interface provides a “hands-free” BI experience that stays accurate at scale.

What’s Next?

While this prototype demonstrates the power of combining RAG with Streamlit, there is still much to explore:

• Embedded BI Dashboards: Moving beyond single charts to embed fully functional, pre-built dashboards (like those from Apache Superset) directly into the chat when a high-level overview is requested.

• Advanced Interactivity: Allowing users to drill down into chart elements through further conversation or “chatting with the legend.”

• Complex Multi-Step Reasoning: Improving how the agent handles requests that require joining multiple cubes or complex pre-aggregations.

We believe that the future of BI isn’t just a static dashboard; it’s a conversation that adapts to the user’s needs in real-time. We hope this project helps you kickstart your own journey into conversational analytics.

Join the Conversation

We’d love to hear your thoughts!

• Did this setup work for your data?

• Do you prefer a different library for parsing AI-generated charts?

• What’s the biggest challenge you’ve faced with Text-to-Chart workflows?

Drop a comment below, or better yet, open an issue or PR on the GitHub repository.

Introduction

In one of our previous posts, we showed how dbt, Cube, and Superset can work together to automate and standardize the semantic layer. The response from the community was overwhelmingly positive but it also raised a recurring question: why introduce Cube between dbt and Superset instead of connecting BI tools directly to the database?

The answer is consistency at scale. A single, shared semantic layer ensures that metrics, dimensions, and business logic are defined once and reused everywhere – across BI tools, APIs, and downstream applications – rather than being reimplemented in each consumer.

This naturally led us to a bigger idea. If Cube already exposes a governed semantic layer through SQL, REST, and GraphQL, why force natural-language analytics back into brittle English-to-SQL translations? What if users could converse directly with the semantic layer instead?

In this post, we explore how to turn Cube analytics into a natural-language interface using RAG. We use RAG to store and retrieve Cube metrics and dimensions, allowing each user query to be grounded in the correct semantic context before any query is generated or executed. This keeps conversations aligned with predefined business logic while remaining flexible and intuitive for users.

Let’s walk through how it works.

👉 GitHub: https://github.com/ponderedw/dbt-to-cube

Quickstart

1. Clone the repository:
   👉 https://github.com/ponderedw/dbt-to-cube

2. Copy .private.env.template to .private.env and configure the required secrets for your LLM provider.

3. just all

Once everything is up and running:

• Open:

http://localhost:8000/

• and navigate to the cube_schemas collection.
  You’ll see a set of vectors representing Cube metadata – dimensions, measures, and cubes.
  These embeddings are generated using the cube-to-rag library (https://pypi.org/project/cube-to-rag/), which we use to extract and embed Cube schema metadata into the vector store.
  

• Next, open

http://localhost:8501/

• If you’ve set STREAMLIT_PASSWORD in .private.env, use it to log in.
  

You’re all set, start asking questions and explore your Cube analytics through natural language.

Talking to Your Data

Let’s start simple:
“What metrics and dimensions do we have?”

Great. Now let’s explore the data itself:
“Show all available course names.”

Nice. Let’s narrow it down:
“What metrics are available for Linear Algebra?”

Now for a concrete business question:
“How many enrollments do we have across all courses?”

And finally, let’s push it a bit further with a more complex query involving filters and breakdowns:
“What is the student engagement score for the Data Structures and Algorithms course, broken down by semester?”

And just like that, everything works. Metrics stay consistent, dimensions are respected, and the conversation maps cleanly to the semantic layer

Conclusion

This experiment shows one possible direction for moving from static semantic layers to truly conversational analytics. By grounding natural-language queries in Cube’s governed semantic layer using RAG, we avoid many of the pitfalls of naive text-to-SQL approaches: inconsistent metrics, broken joins, and logic that drifts away from business definitions. Instead, users can explore data conversationally while staying aligned with a single source of truth.

That said, this is not the solution – it’s our attempt at solving a real and growing problem. There are open questions around performance, UX, evaluation, security, and how this approach compares to alternatives such as agent-driven query planning, stricter semantic parsers, or hybrid BI experiences. We’re certain there are improvements to be made, edge cases we haven’t covered, and ideas we haven’t considered.

We’d love to hear what others think about this: What works? What doesn’t? What would you do differently? If you’re experimenting with similar approaches – or taking a completely different path – your feedback, suggestions, and alternatives are more than welcome.

Introduction

Most teams have been there. You start storing company documentation as Word files or PDFs because it’s quick, familiar, and temporary. A proof of concept, a contract, a report – throw it into a folder and move on. But over time, that “temporary” storage often becomes a critical production knowledge base.

At that point, problems emerge: documents are hard to search, relationships are implicit, extracting structured data is manual, and automation becomes painful.

A common approach today is “just put everything into RAG.” While retrieval helps with search, it doesn’t solve the core issue: the information itself is still unstructured. If the content were normalized into a machine-friendly format (like JSON with well-defined schemas), working with it would be faster, more reliable, and would open new workflows – analytics, validation, integrations, and automation.

That gap is what inspired this experiment. I wanted a way to turn business documents into structured, queryable data using LLMs. The result is StructuDoc, an open-source proof of concept that combines document parsing, image understanding, schema discovery, and prompt tracking into a single pipeline. It can move PDFs and DOCX files into structured datasets, identify common schemas, and store everything in S3.

This post introduces StructuDoc, explains how it works, and shares the reasoning behind its design. It’s offered in a spirit of sharing rather than as a commercial product – we’d love your feedback, alternative approaches you’ve tried, or contributions to the project.

• GitHub: https://github.com/ponderedw/StructuDoc

• Docker Hub: https://hub.docker.com/repository/docker/pondered/structudoc/general

Quickstart with StructuDoc

StructuDoc makes it easy to turn PDFs and DOCX files into structured JSON, discover common schemas, and even analyze images using AI. Here’s how to get started in minutes:

1. Clone the repository

git clone https://github.com/ponderedw/StructuDoc.git
cd StructuDoc

2. Set up environment variables

Copy the example file:

cp .env.example .env

Edit .env to configure your local storage (MinIO) or S3, and optionally your LLM provider keys:

ENV=local
SOURCE_BUCKET=minio/source-bucket
MINIO_HOST=http://localhost:9000
MINIO_SECURE=false
AWS_ACCESS_KEY_ID=admin
AWS_SECRET_ACCESS_KEY=password
LLM_MODEL=Bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0

3. Start the application (this file can be executed independently of the repository)

docker compose up -f docker-compose-prod.yml

• Streamlit UI:

 http://localhost:8501

• FastAPI backend:

 http://localhost:8080

• MinIO console (login: admin/password):

 http://localhost:9001

4. Upload and process documents

1. Go to Upload Source Files in the UI.

2. Upload DOCX or PDF files.

3. StructuDoc will automatically:

   • Convert PDFs to Markdown

   • Extract images

   • Store all files in structured folders

5. Parse and analyze with AI

• Navigate to Parse Files With LLM:

• Analyze images with AI prompts

• And remember to save those descriptions to your s3 bucket:

• Discover common schemas across multiple documents

• Parse your documents to JSON

Now you have structured, queryable data from your documents – ready for testing, analysis, or building pipelines!

Conclusion

StructuDoc is an open-source experiment for turning PDFs and DOCX files into structured, queryable data. Instead of relying solely on RAG for search, it lets you extract JSON, discover common schemas, and analyze images with AI. This isn’t a fully polished product – it’s a proof of concept born from real challenges we’ve faced, shared in the hope it might help others or spark new ideas. We’d love to hear your feedback, learn about alternative approaches you’ve tried, or see contributions that improve StructuDoc.

GitHub: https://github.com/ponderedw/StructuDoc
Docker Hub: https://hub.docker.com/repository/docker/pondered/structudoc/general

Introduction

dbt has become the backbone of modern analytics engineering, and one of its best features is CI: the ability to compare a developer branch with main, rebuild only the changed models in a temporary schema, and use production data from unchanged models. It’s an amazing way to catch issues before they hit production.

But what about incremental models and snapshots? Metrics like Grade Point Average whose day-by-day variations are captured in a dbt snapshot are then tracked and rendered in various contexts over time. This per-student metric is calculated in one environment (production) each day to ensure a clear source of truth. But for CI…that’s where things get tricky.

Both incremental models and snapshots have different logic for initial runs vs. upsert runs. In CI, everything is always built from scratch – so your CI may pass, only for the query to fail the moment it meets existing data or objects in production.

To solve this for Uncommon Schools, we built a tiny but powerful solution – and now we’re sharing it with everyone:
📦 dbt-incremental-ci (https://pypi.org/project/dbt-incremental-ci/)

Quickstart

Get up and running in under 5 minutes.

dbt-incremental-ci copies your production incremental models and snapshots into your CI schema so you can test against real production data, not empty tables.

Installation

pip install dbt-incremental-ci

Option 1: Using a Local Manifest File

Use this approach if your production manifest.json is stored in S3, GCS, or available locally.

Step 1: Download your production manifest

# From S3
aws s3 cp s3://my-dbt-artifacts/prod/manifest.json ./prod_manifest.json

# From GCS

gsutil cp gs://my-dbt-artifacts/prod/manifest.json ./prod_manifest.json

Step 2: Copy production tables into your CI schema

dbt-incremental-ci \
  --prod-manifest-path ./prod_manifest.json \
  --dbt-project-dir . \
  --database-uri “postgresql://user:pass@host:5432/db” \
  --ci-schema “ci_pr_123” \
  --threads 4

Step 3: Run dbt with the copied production data

dbt build --select state:modified+ --defer --state ./prod_manifest.json

Step 4: Run your tests

dbt test --select state:modified+ --defer --state ./prod_manifest.json

You’re now running incremental models and snapshots against actual production state—not empty CI tables.

Option 2: Using dbt Cloud

If you’re on dbt Cloud, you don’t need to download the manifest—the tool fetches it automatically from the API.

Step 1: Set your credentials

export DBT_CLOUD_API_TOKEN=”your_token”
export DBT_CLOUD_ACCOUNT_ID=”12345”

Step 2: Copy production tables into your CI schema

dbt-incremental-ci \
  --dbt-cloud-job-id “67890” \
  --dbt-project-dir . \
  --database-uri “$DATABASE_URI” \
  --ci-schema “ci_pr_123” \
  --threads 4

The tool will automatically:

• Connect to the dbt Cloud API

• Fetch the latest successful production run

• Download the manifest

• Copy the required tables

Step 3: Run dbt

dbt build --select state:modified+ --defer

Step 4: Run your tests

dbt test --select state:modified+ --defer

What Just Happened?

The tool:

1. Identified changed models using state:modified+

2. Filtered to incremental models and snapshots that already exist in production

3. Copied those production tables (with data!) into your CI schema

4. Preserved custom schemas (e.g., prod_marts → ci_pr_123_marts)

This ensures your CI pipeline behaves like a real incremental run—catching bugs that only appear when existing data is present.

Preview Before Running (Dry Run)

Want to see what will be copied without actually doing anything?

dbt-incremental-ci \
  --prod-manifest-path ./prod_manifest.json \
  --dbt-project-dir . \
  --database-uri “postgresql://user:pass@host:5432/db” \
  --ci-schema “ci_pr_123” \
  --dry-run \
  --verbose

This prints exactly which tables would be copied and the SQL that would run.

Final Thoughts

Incremental models and snapshots are some of the most powerful parts of dbt – but they’re also uniquely tricky to validate in CI. Full rebuilds simply don’t reflect how these models behave in production, which means some of the most painful bugs can slip past your pipeline undetected.

dbt-incremental-ci is our attempt to fix that gap. It’s a lightweight, practical solution that brings real production state into your CI runs so you can test incremental logic the way it actually runs in the real world.

But this isn’t the final word – far from it.
This is our first take at solving the problem, and we know the community will have ideas we haven’t thought of yet. If you have a better approach, want to improve this one, or see opportunities to make it smarter, faster, or more general – we’d love your help.

👉 PyPI: https://pypi.org/project/dbt-incremental-ci/
👉 GitHub: https://github.com/ponderedw/dbt-incremental-ci – issues, discussions, and PRs are all very welcome.

Introduction

A couple of weeks ago, we shared several blog posts about different ways we’re making data more accessible and interactive: our Superset MCP (link), loading dbt metadata into GraphDB (link), and an amazing AI-powered UI plugin for Airflow that lets you interact with your workflows directly in the UI (link).

Today, I want to bring all three together and introduce superset-chat - a library that helps you manage both your dbt and dashboard metadata right in the Superset UI, powered by AI, Superset MCP, and our Falkor DB plugin. You can check it out on GitHub: superset-mcp-plugins.

If you’ve ever felt a disconnect between BI and transformation logic, this library might resonate. It can answer questions like, “Which dashboards depend on source X?” - even when those sources aren’t referenced directly. While still a POC, superset-chat has already helped us organize educational metadata, and we’re excited to share it with the community. Maybe you’ll help improve it or come up with something entirely new!

Quickstart

You can try superset-chat in two ways:

1. Use your existing Superset

If you already have a Superset deployment, you can integrate the plugin directly:

• Install the plugin:

pip install superset-chat

• Update your superset_config.py to register the plugin:

FLASK_APP_MUTATOR = lambda app: init_custom_views(app)
def init_custom_views(app):

    from superset_chat.ai_superset_assistant import AISupersetAssistantView

    appbuilder = app.appbuilder
    appbuilder.add_view(
        AISupersetAssistantView,
        “AI Superset Assistant”,
        icon=”fa-robot”,
        category=”Custom Tools”
    )

• Make sure you have the following environment variables set for your deployment:

LLM_MODEL_ID=bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0  # Must start with ‘bedrock’, ‘anthropic’, or ‘openai’
# ANTHROPIC_API_KEY=...              # Required if using Anthropic
AWS_ACCESS_KEY_ID=’...’             # Required if using Bedrock
AWS_SECRET_ACCESS_KEY=’...’         # Required if using Bedrock
SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://postgres:postgres@postgres:5432/postgres
SUPERSET_API_URL=http://localhost:8088
SUPERSET_USERNAME=superset_admin
SUPERSET_PASSWORD=superset
GRAPH_DB=falkordb                     # Optional: enable GraphDB connection
GRAPH_HOST=...                        # Optional: if using a remote GraphDB
GRAPH_USER=...
GRAPH_PASSWORD=...

Restart Superset, log in, and open AI Superset Assistant under Custom Tools.

2. Run the local deploy

If you don’t have a Superset deployment, you can use our preconfigured local setup:

• Clone the repository:

git clone https://github.com/ponderedw/superset-mcp-plugins.git
cd superset-mcp-plugins

• Copy the environment template and configure credentials:

cp template.env .env

• The minimal .env for local deploy (we use FalkorDB in this tutorial) can be:

LLM_MODEL_ID=bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0
# ANTHROPIC_API_KEY=...
AWS_ACCESS_KEY_ID=’...’
AWS_SECRET_ACCESS_KEY=’...’
SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://postgres:postgres@postgres:5432/postgres
SUPERSET_API_URL=http://localhost:8088
SUPERSET_USERNAME=superset_admin
SUPERSET_PASSWORD=superset
GRAPH_DB=falkordb

• Run Superset with superset-chat:

just run_superset

• Navigate to your Superset instance (usually http://localhost:8088) and open AI Superset Assistant under Custom Tools.

This local deploy is a fully working Superset + superset-chat environment, so you can start experimenting even if you don’t have your own deployment.

Using superset-chat: Examples

If you’re using our project, we’ve included a dbt project that was automatically loaded into FalkorDB, along with Superset charts and dashboards that were imported into Superset. You can explore all of them in the UI as well as in dbt docs.

Now, let’s try out the plugin with a few example queries:

1. Superset-related question:

“Quick question – what’s our current count of Superset dashboards and charts?”

As expected, superset-chat lists all the Superset metadata correctly – dashboards and charts – just like in the UI. Cool!

2. dbt-related question:
   
   

“Hey! Do you know how many dbt models we have in total?”

We can check the answer in dbt docs, and it’s completely correct. Awesome!

3. Combined dbt + Superset question:

“Could you help me find all the dbt sources (not models) used in the Student Academic Standing chart?”

Amazing! superset-chat traced the chart all the way back to the underlying database tables. This shows how we can connect Superset dashboards directly to dbt metadata seamlessly.

Connecting BI and Transformation Metadata with AI

Connecting BI and Transformation Metadata with AI
With superset-chat, you can finally see your BI layer and your transformation logic in one place, without manually tracing dependencies or opening multiple tools. By unifying dbt lineage and Superset metadata, the plugin allows you to:

• Ask natural-language questions spanning both dashboards and data models.

• Trace dashboards and charts back to the underlying source tables.

• Verify dbt lineage directly from Superset without leaving the UI.

• Quickly get insights that previously required multiple queries or digging through documentation.

Even as a POC, superset-chat already shows how AI can bridge the gap between data transformation and business intelligence. It’s also beginning to influence some of our internal workflows - for example, in our work with Uncommon Schools we’re exploring how the assistant can accelerate outage protocols, particularly when on-call responders aren’t familiar with the specific code paths involved. This kind of scenario highlights the value of unifying metadata and making it accessible through natural language.

Whether you’re exploring dashboards, analyzing dbt models, or troubleshooting lineage issues, this tool turns what was once a fragmented process into a single, interactive experience. We’re excited to share this with the community and invite you to try it, suggest improvements, or build on it for your own workflows. It’s a step toward truly connected, AI-powered data exploration.

Introduction

Rebuilding the same semantic layer over and over across different BI tools has always felt… unnecessarily painful. Most dashboards still reach directly into databases, each with their own idea of what a “metric” is, so you end up maintaining the same logic in three or four places. Even tools built to solve this – like Cube.js – don’t really solve the end-to-end problem of getting consistent metrics into your BI layer.

Out of curiosity (and a bit of frustration), I started tinkering with a small proof of concept to see whether dbt, Cube.js, and Superset could be wired together into a single automated flow. The result is dbt-to-cube:

https://github.com/ponderedw/dbt-to-cube

It’s very much a work in progress, but it already connects dbt → Cube.js → Superset and keeps metrics in sync without any manual steps. Sharing it here in case others are experimenting with similar setups – I’d genuinely love feedback, ideas, or contributions from anyone who gives it a try.

Quickstart

See dbt-cube-sync in action! This walkthrough demonstrates a full semantic layer pipeline—from dbt transformations to BI dashboards—without any manual metric duplication.

Already Have dbt, Cube.js, and Superset?

If your stack is already deployed, you don’t need Docker. Just install dbt-cube-sync and point it to your existing deployment:

pip install dbt-cube-sync

# Convert dbt models to Cube.js schemas

dbt-cube-sync dbt-to-cube \

  --manifest /path/to/manifest.json \

  --catalog /path/to/catalog.json \

  --output /path/to/cube/output

# Sync Cube.js schemas to Superset

dbt-cube-sync cube-to-bi superset \

  --cube-files /path/to/cube/output \

  --url http://your-superset:8088 \

  --username admin \

  --password admin \

  --cube-connection-name Cube

This workflow works with Superset, Tableau, or PowerBI, giving you full control over your production-ready semantic layer.

Starting from Scratch? Spin Up Everything Locally

If you want a full demo stack including sample data, all services run locally in Docker: PostgreSQL, dbt, Cube.js, Superset, and dbt-cube-sync.

Prerequisites

• Docker & Docker Compose

• Git

• ~10 minutes

Step 1: Clone & Start the Stack

git clone https://github.com/ponderedw/dbt-to-cube
cd dbt-to-cube
docker-compose up --build

This will:

1. Initialize PostgreSQL with sample data

2. Run dbt transformations on 46 pre-built models

3. Start Cube.js API serving your semantic layer

4. Configure Superset with admin access and database connection

5. Automatically generate Cube.js schemas from dbt and sync them to Superset

Step 2: Explore Your Data Stack

Once the logs show “Pipeline completed successfully!”, you can access:

• Superset: http://localhost:8088 — login: admin/admin

  • 46 datasets auto-created from dbt models

  • Column metadata and metrics like Average GPA and Pass Rate

• dbt Docs: http://localhost:8080 — full DAG and column-level documentation

• Cube.js API: http://localhost:4000 — REST and GraphQL endpoints

Step 3: Watch the Pipeline in Action

docker-compose logs -f dbt-cube-sync

The pipeline runs in three automated phases:

1. dbt Parsing – reads manifest.json & catalog.json, finds models, and maps dependencies

2. Cube.js Schema Generation – converts dbt models into Cube.js dimensions and measures

3. BI Tool Sync – pushes datasets and metrics to your BI tool with proper metadata

Everything happens automatically for all models—no manual intervention required.

✅ What You’ve Built

In minutes, you now have a unified semantic layer pipeline:

1. Single source of truth: Metrics defined once in dbt

2. Automatic propagation: dbt → Cube.js → BI tools

3. Cross-tool flexibility: Works with Superset, Tableau, PowerBI

4. Developer-friendly: Version-controlled, testable, and documented

Replace sample data with your warehouse and dbt models, and you have a production-ready semantic layer – no metric inconsistencies across tools.

The Takeaway

This little experiment convinced me that an automated semantic layer doesn’t have to be complicated. Define metrics once in dbt, let a small tool translate them to Cube.js, and push them into your BI tool of choice – no more re-implementing the same logic across dashboards.

If you end up trying out dbt-to-cube, I’d be really interested to hear what works, what breaks, or what you’d want it to support next. It’s still a POC, but with feedback from others running into the same problems, it could grow into something genuinely useful for the community.

Always happy to chat if you’re experimenting with semantic layers or have ideas on making this whole workflow smoother!

Introduction

At Ponder, we’ve developed several plugins for Airflow 2.* and recently began migrating them to Airflow 3. Among our most valuable tools are Airflow Schedule Insights and Airflow Chat.

During the migration process, we discovered that Airflow 3.1.1 removed the DAG Dependencies tab that was present in Airflow 2.

Rather than wait for this feature to return, we saw an opportunity: we could build a plugin to restore this functionality, help the community migrate to Airflow 3 more smoothly, demonstrate how to create plugins for Airflow 3, and even add improvements we found missing in Airflow 2.

The result is airflow-dag-dependencies. In this post, we’ll walk you through how we built it and how it helps us manage our many educational DAGs.

Quick Start

To use our plugin, simply install airflow-dag-dependencies in your Airflow environment. If you don’t have an environment set up, clone our GitHub repository and run docker compose up.

After a few minutes, once Airflow is running, navigate to http://localhost:8080/ and log in with username “airflow” and password “airflow”. You’ll find a dozen educational DAGs — load_class_attendance, load_course_evaluations, and others.

Some are triggered by datasets, others by DAGs, and some run on cron schedules. The challenge? Understanding how they all connect.

This is where our plugin shines. Navigate to the Browse menu and click on DAG Dependencies.

You’ll immediately see all the relationships visualized clearly. Notice how transform_program_effectiveness has two dependencies: a cron schedule and a dataset trigger.

You can also spot that load_curriculum_catalog and load_student_enrollment_data have no dependencies or schedules—they won’t run automatically.

And you’ll see that transform_academic_performance is triggered by load_course_evaluations.

Enhanced Filtering

But we didn’t stop at recreating Airflow 2’s functionality—we added the filtering capabilities we always wished it had. Want to see only the direct downstream dependencies of transform_student_feedback_analysis and all its upstream dependencies? Simply enter +transform_student_feedback_analysis+1 in the search bar. The graph instantly filters to show only the relevant connections.

The search syntax is intuitive: use + before a DAG name to include upstream dependencies, after to include downstream, or both for the full picture. Add a number (like +dag_name+2) to control how many levels deep to traverse. You can also search multiple DAGs at once using commas, or use @exact_name for precise matching.

That’s the power of airflow-dag-dependencies!

Building Your First Airflow 3 Plugin: A Technical Deep Dive

Now that we’ve seen the plugin in action, here is how to build it from scratch. This walkthrough highlights the key differences between Airflow 2 and Airflow 3 plugin development and shows how to create your own plugin.

Plugin Structure: Airflow 3 vs Airflow 2

One of the most significant changes in Airflow 3 is how plugins are structured and registered.

Airflow 2 Plugin Structure

from airflow.plugins_manager import AirflowPlugin
from flask import Blueprint

from flask_admin import BaseView

class MyAirflow2Plugin(AirflowPlugin):

    name = “my_plugin”

    flask_blueprints = [my_blueprint]  # Flask blueprints

    admin_views = [MyAdminView]        # Flask-Admin views

    menu_links = [my_menu_link]        # Menu items

Airflow 3 Plugin Structure

from airflow.plugins_manager import AirflowPlugin
class MyAirflow3Plugin(AirflowPlugin):

    name = “my_plugin”

    fastapi_apps = [my_fastapi_app]     # FastAPI applications

    external_views = [my_external_view] # External view definitions

Key Changes in Airflow 3

• FastAPI replaces Flask for web applications, improving performance, async support, and automatic API documentation.
• External views replace Flask-Admin views, giving more flexible UI integration.
• Menu items are now part of external views rather than separate menu link definitions.

Anatomy of the DAG Dependencies Plugin

Package Structure

airflow_dag_dependencies/

├── __init__.py

└── plugins/

├── __init__.py

└── dags_dependencies_plugin.py

FastAPI Application

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI()

@app.get(”/”, response_class=HTMLResponse)

async def dag_dependencies_graph():

    # Return HTML with embedded D3.js visualization

    return html_content

@app.get(”/api/dag-dependencies”)

async def get_dag_dependencies():

    # Return JSON data for DAG relationships

    return {”nodes”: nodes, “links”: links}

Plugin Registration

FastAPI app configuration

fastapi_app_with_metadata = {
    “app”: app,
    “url_prefix”: “/dags_dependencies”,
    “name”: “DAG Dependencies App”
}

External view for menu integration

dag_dependencies_external_view = {
    “name”: “DAG Dependencies”,
    “href”: “/dags_dependencies/”,
    “destination”: “nav”,
    “category”: “browse”,
    “url_route”: “dag_dependencies”
}

Plugin class

class DagDependenciesPlugin(AirflowPlugin):
    name = “dags_dependencies_plugin”
    fastapi_apps = [fastapi_app_with_metadata]
    external_views = [dag_dependencies_external_view]

Data Access: Using DagBag

The plugin accesses Airflow’s DAG metadata through DagBag.

from airflow.models import DagBag
async def get_dag_dependencies():

    dagbag = DagBag(include_examples=False)

    nodes = []

    for dag_id, dag in dagbag.dags.items():

        nodes.append({

            “id”: dag_id,

            “type”: determine_dag_type(dag),

            “schedule”: get_schedule_info(dag)

        })

    links = analyze_dag_relationships(dagbag)

    return {”nodes”: nodes, “links”: links}

Frontend Options: Embedded or External

Airflow 3 supports multiple frontend approaches:

• Embedded HTML/JS (current plugin)
• Standalone React components (future tutorial)
• Serving static assets for more advanced UIs

Packaging for Distribution

Example pyproject.toml configuration:

[tool.poetry]
name = “airflow-dag-dependencies”

version = “0.1.1a1”

description = “An Apache Airflow plugin that provides interactive visualization of DAG dependencies”

authors = [”Ponder”]

[tool.poetry.dependencies]

python = “>=3.12, <3.14”

apache-airflow = “^3.1.0”

[tool.poetry.plugins.”airflow.plugins”]

dag-dependencies-plugin = “airflow_dag_dependencies.plugins.dags_dependencies_plugin:DagDependenciesPlugin”

What’s Next?

The next tutorial will cover:

• Building React components that integrate with Airflow’s UI
• Handling authentication and permissions

Airflow 3’s FastAPI foundation opens the door to more capable, modern, and faster plugin development. More powerful integrations are now possible than ever.

Conclusion

Airflow 3 marks a real shift in how developers extend the platform. FastAPI, external views, and the new plugin model make it possible to build tools that are faster, more flexible, and much easier to maintain. Rebuilding the DAG Dependencies feature gave us a chance to explore these new capabilities in depth and turn a missing Airflow 2 feature into something more capable than before.

The airflow-dag-dependencies plugin restores a critical part of the Airflow UI, adds filtering and search features that help teams understand complex pipelines, and acts as a practical example of how to build real plugins in Airflow 3. If you are migrating from Airflow 2 or planning to develop your own extensions, this approach will give you a strong foundation.

There is still more room to grow. React-based UI components, richer interactions with Airflow’s new frontend APIs, and improved permission handling are all on the way. Airflow 3 opened the door to a modern plugin ecosystem, and this is only the start.

Introduction

At Ponder, we first introduced Airflow MCP and its plugin to make working with DAGs and pipelines smoother. But along the way, we realized something: managing Superset dashboards and their metadata can be just as challenging—if not more so.

Take education as an example. Schools and educational institutions manage dozens—sometimes hundreds—of dashboards tracking everything from student performance and attendance to resource allocation and budget utilization. Multiple BI developers work constantly on these dashboards, making frequent updates and refinements so stakeholders can access the most current data to improve education and help pupils succeed. The result? A powerful ecosystem of insights—but also an overwhelming maze of metadata.

Questions like “Who changed the attendance dashboard last week?”, “Which table powers our graduation rate metrics?”, or “What’s the data source behind this enrollment forecast?” can quickly become a time sink. And it’s not just schools—any organization with a complex BI environment faces the same challenge.

That’s why we built Superset MCP—so you can explore and manage Superset metadata through natural language, directly from your Claude code. No need to wrestle with the UI, no need to dig through layers of metadata. Just type your question, and get clear answers in plain language.

You can check out the project on GitHub, pull the image from Docker Hub, or install it from PyPI.

Quickstart

1. Run Superset Locally

If you don’t already have a Superset deployment but still want to try out Superset MCP, simply clone our GitHub repository and start it with Docker Compose:

git clone https://github.com/ponderedw/superset-mcp
cd superset-mcp
docker compose up

Once it’s up, open your browser and visit:

http://localhost:8088/

Login credentials for the example environment:

• Username: superset_admin

• Password: superset
  
  

2. Using Superset MCP in Claude Desktop

1. Open Claude Desktop.

2. Navigate to: Settings → Developer → Edit Config.

3. Add the following entry to your MCP servers:

{
  “mcpServers”: {
    “superset”: {
      “args”: [
        “run”,
        “-i”,
        “--rm”,
        “-e”,
        “SUPERSET_API_URL”,
        “-e”,
        “SUPERSET_USERNAME”,
        “-e”,
        “SUPERSET_PASSWORD”,
        “pondered/superset-mcp:latest”
      ],
      “command”: “docker”,
      “env”: {
        “SUPERSET_API_URL”: “http://host.docker.internal:8088”,
        “SUPERSET_USERNAME”: “superset_admin”,
        “SUPERSET_PASSWORD”: “superset”
      }
    }
  }
}

Note: The Superset URL and credentials above match our local example environment.

4. Restart Claude Code.

3. Using Superset MCP with LangChain

To integrate with LangChain:

1. Install our PyPI package:

pip install superset-mcp-server

📦 PyPI link

2. Add the following MCP server configuration to your tools:

from langchain_mcp_adapters.client import MultiServerMCPClient
import os

mcps = {
    “SupersetMCP”: {
        “command”: “python”,
        “args”: [”-m”, “superset_mcp_server.mcp_server”],
        “transport”: “stdio”,
        “env”: {
            k: v for k, v in {
                “SUPERSET_API_URL”: os.getenv(”SUPERSET_API_URL”),
                “SUPERSET_USERNAME”: os.getenv(”SUPERSET_USERNAME”),
                “SUPERSET_PASSWORD”: os.getenv(”SUPERSET_PASSWORD”),
            }.items() if v is not None
        }
    }
}

client = MultiServerMCPClient(mcps)
mcp_tools = await client.get_tools()

Now you’re ready to simply ask questions about your Superset instance in natural language—and let MCP handle the rest!

Talking to Superset MCP

Let’s connect to our example Superset deployment and build a simple dashboard using the data automatically ingested into our Postgres database.

First, we’ll add a database connection — specify “postgres” as both the host and the database name, and use “superset” for the login and password.

Once connected, we can create a few simple charts from the database tables and organize them into an E-commerce Performance Dashboard.

Finally, don’t forget to publish your dashboard so it’s ready to use!

You can explore all the available tools in Claude Desktop — and more will be coming soon.

Now, let’s start asking some interesting questions:

• How many dashboards and charts do we have?

  

  

  Great! We get the exact number of dashboards and charts in our environment.

• Who are the most active users?

  

  

  
  Perfect — not only do we see the top active user, but we also get an Activity Timeline, which makes it much easier to understand what’s happening in our cluster.

• What is the source table for Customer-Product Network?

  

  Excellent — we now have the precise database object name.

• When was the Revenue by Country chart last changed, and by whom?

  

  Spot on — we’ve got the correct answer here as well.

Pretty cool, right? Keep exploring and help us uncover even more capabilities of Superset MCP!

Conclusion — Managing Metadata the Smart Way

Superset MCP turns the often painful task of navigating dashboard metadata into a fast, intuitive conversation. Whether you’re an educational administrator tracking changes to student performance dashboards, a data analyst identifying which tables feed into critical reports, or a team lead trying to understand who’s been most active in your BI environment—you can now skip the clicks, menus, and manual searches. Just ask in plain language, and get precise, actionable answers.

In education settings where BI teams constantly update dashboards to provide stakeholders with real-time insights for improving outcomes, Superset MCP becomes invaluable. But the benefits extend far beyond schools—any business with complex dashboard environments, whether in healthcare, finance, retail, or manufacturing, can leverage this natural language interface to cut through the metadata complexity.

Because it works entirely through natural language, you can easily delegate these tasks to analysts, project managers, principals, department heads, or any non-technical users—empowering more people in your team to get the information they need without relying on developers or admins.

We’re excited to see how the community will use Superset MCP—and we’d love your feedback. Try it out, share your ideas, and help us shape the next set of capabilities.

🔗 Resources:

• GitHub Repository

• Docker Hub Image

• PyPI Package

  Thanks for reading Data & AI Engineering @ Ponder! Subscribe for free to receive new posts and support our work.

Introduction

In education data teams, our analytics models often grow organically alongside organizational needs. As school networks expand their data infrastructure—adding new student information systems, assessment platforms, and operational tools—their dbt projects can become increasingly complex. What starts as a clean set of staging models quickly evolves into an intricate web of dependencies that becomes difficult to navigate and maintain.

In previous blog posts—dbt-to-English and English-to-SQL—we demonstrated how LLMs can assist with understanding and generating dbt metadata. However, these tools fall short when it comes to answering complex questions about dbt’s project-wide structure and dependencies. For instance, what is the most widely reused source across models (recursively)? Or, how many seed datasets are used by models materialized as tables?

We’ve seen this challenge firsthand. At one school network, their analytics catalog grew from a handful of models serving basic enrollment reporting to hundreds of interconnected models supporting everything from attendance analytics to intervention tracking to college persistence dashboards. This organic growth led to inadvertent circular dependencies that made it increasingly difficult to refactor models or optimize their orchestration. The team needed a way to visualize and understand these dependencies before they could untangle them.

That’s when we had a realization: dbt’s structure is naturally graph-shaped, making it an excellent fit for graph databases. This inspired us to launch a new project—dbt-kg—which shows how to upload dbt metadata (from manifest.json and catalog.json) into a graph database and leverage it in an AI-powered chat. While our demo uses FalkorDB, the project is also compatible with Neo4j—just update the configuration settings accordingly.

Quickstart

To get started, clone our GitHub repository:
https://github.com/ponderedw/dbt-kg

Next, copy the environment template and configure it:

cp template.env .env

In your .env file, set the LLM_MODEL_ID according to the provider you want to use:

LLM_MODEL_ID=’bedrock:...’
LLM_MODEL_ID=’anthropic:...’
LLM_MODEL_ID=’openai:...’

The prefix (bedrock, anthropic, or openai) determines which LLM provider will be used.
Be sure to include the appropriate credentials as well:

• AWS_SECRET_ACCESS_KEY for Bedrock

• ANTHROPIC_API_KEY for Anthropic

• OPENAI_API_KEY for OpenAI

Then, choose your graph database by setting the GRAPH_DB variable. This can be either falkordb or neo4j. For this demo, we’ll be using FalkorDB.

If your database requires authentication, provide values for GRAPH_USER and GRAPH_PASSWORD.

⚠️ Our default FalkorDB setup does not use authentication, so these can remain commented out. If you’re using our default Neo4j setup, uncomment those variables and set the default credentials accordingly.

Now, run the setup:

just all

Once everything is set up, open your browser and go to:

 http://localhost:8501/

Navigate to the “Load DBT Manifest” page from the sidebar. You’ll be prompted to upload your manifest.json and catalog.json files. If you have your own, upload them; otherwise, use the sample files located in DbtExampleProject/target within the repo.

Upon successful upload, you’ll see a confirmation message.

Finally, open your graph database UI at:

 http://localhost:3000/

Run the following query to view your dbt Knowledge Graph:

MATCH (a)-[b]-(c)
RETURN a, b, c

Congratulations! Your dbt Knowledge Graph is now up and running. 🎉

Interacting with Your dbt Graph via Chat

Let’s return to

 http://localhost:8501/

—this time, we’ll stay on the Main page and begin chatting with our dbt Knowledge Graph.

We’ll start by asking a high-level question about source table usage:

Prompt:
“Can you find the source tables that are used by the most models, including indirect dependencies through the entire dependency chain?”

The system correctly identifies both source tables from our example project and ranks them by usage in descending order. Perfect!

Next, let’s try something more challenging:

Prompt:
“Please identify the model with the highest number of dependent models, including transitive dependencies. This model must be materialized as a table. Return all the information you have about this table.”

This is tricky—because in our project, there are no models materialized as table.

Impressively, the LLM handles this gracefully by identifying the closest match: a view instead, and clearly stating the limitation. That’s a solid response.

Let’s follow up with another prompt to dig deeper:

Prompt:
“Can you also list indirect dependent models for stg_orders?”

Now we’re talking. The model returns the full dependency chain. Mission accomplished.

From One Monolithic Job to Smart Job Grouping

In many real-world workflows, especially when integrating with orchestrators like Airflow, we don’t want to run the entire dbt graph as a single job. Instead, we aim to break it into smaller, manageable jobs (DAGs) that can trigger downstream non-dbt processes as early as possible. But there’s a catch: circular dependencies often make this kind of separation complex and error-prone.

Let’s ask the LLM to help with that:

Prompt:
“Can you analyze our dbt dependency graph and create a job separation strategy? I need to group models, seeds, and snapshots into separate jobs that eliminate circular dependencies while defining clear inter-job dependencies.”

And it delivers! The model intelligently separates our graph into four distinct jobs and describes the cross-job dependencies clearly.

Summary

By leveraging the graph-shaped nature of dbt projects, the dbt-kg initiative demonstrates how graph databases like FalkorDB and Neo4j can supercharge the way we interact with dbt metadata. With an intuitive setup, seamless integration of LLMs, and a simple web-based interface, users can upload their dbt artifacts and start asking complex, project-wide questions in natural language.

For education data teams in particular, this approach addresses a common challenge: as schools and networks scale their analytics capabilities, their dbt projects naturally grow more complex. The AI-powered chat not only handles nuanced prompts about source usage, dependencies, and materializations but also assists in advanced tasks like intelligent job grouping to optimize orchestration workflows—exactly the kind of refactoring that the school network mentioned earlier needed to break apart their circular dependencies and create a more maintainable structure.

Whether you’re supporting a single school, a district, or a multi-state network, transforming static metadata into a dynamic, queryable knowledge graph makes data engineering more intelligent, conversational, and manageable. This is especially valuable in education, where data teams often wear multiple hats and need efficient ways to understand and maintain increasingly sophisticated analytics infrastructure.

Introduction

After recently launching our new library, airflow-chat—a plugin that enables seamless LLM chat interactions within your Airflow instance via our Airflow MCP—we quickly turned our attention to an important question: how do we effectively observe and log these conversations?

Imagine a scenario where a user triggers an unintended action due to safe mode being disabled, leading to a production error. How do you trace the issue? Who initiated the command? What was the exact prompt that caused the failure? Or more generally—how do you evaluate LLM interactions over time?

To address these questions, we turned to Langfuse, an elegant open-source tool purpose-built for developing, monitoring, evaluating, and debugging LLM applications. In this blog post, we’ll walk you through integrating Langfuse into your LangChain workflow—and specifically, how we’ve already done this within the airflow-chat plugin. You can follow along using our repo: https://github.com/ponderedw/airflow-mcp-plugins.

Quickstart

Let’s walk through a quick setup to see how Langfuse integrates with the airflow-chat plugin.

1. Clone the Repository and Set Up Locally

Start by cloning our GitHub repository:

git clone https://github.com/ponderedw/airflow-mcp-plugins

Then follow the setup steps outlined in our airflow-chat blog post to get the plugin running in your local environment.

Note: Running both Airflow and Langfuse locally can be resource-intensive. If your machine starts lagging, consider connecting to pre-deployed instances of Airflow and Langfuse for a smoother experience.

To launch both locally, run:

just airflow_langfuse

Once the containers are up, head to

 http://localhost:3000/

and sign up for a new Langfuse account.

2. Create an Organization, Project, and API Key

After signing up:

• Create a new Organization

  

• Create a Project under that organization

  

• Generate an API key

  

Copy the credentials and add them to your Airflow environment variables. If you’re using our GitHub setup, you can simply paste them into the airflow-plugin.env file:

LANGFUSE_SECRET_KEY=sk-lf-053f1073-789b-4705-8daa-23b825829d3d
LANGFUSE_PUBLIC_KEY=pk-lf-25851900-8b27-45f7-b7a0-617321e988cf
LANGFUSE_HOST=http://host.docker.internal:3000

3. Restart and Explore

Restart your environment:

just airflow_langfuse

Now, navigate to the Airflow UI, interact with the airflow-chat plugin, and ask a few test questions.

Then head over to the Langfuse dashboard. Open the "Tracing → Traces" tab:

Voilà—your entire chat session is logged, complete with trace-level details.

Just like that, observability is in place!

Going Further with Langfuse

Langfuse offers much more than just trace logging—it provides powerful tools to evaluate and improve your LLM application.

Users
Navigate to the Users tab to see a list of application users. You’ll notice a user named "airflow"—this represents interactions made through the airflow-chat plugin. Click on it to view all associated sessions. This is a great way to monitor how your team is using the plugin over time.

Cost Tracking
Langfuse makes it easy to track costs. In the Traces tab, you’ll see session-level costs, while the Home tab gives you an overview of model-level expenses. For deeper insights, head over to the Dashboards tab and explore the built-in Langfuse Cost Dashboard, or create a custom dashboard with your own cost graphs.

If you suspect that token prices are incorrect, go to Settings → Models, and either “Add model definition” or “Clone” an existing one to customize pricing.

Evaluations
Langfuse supports two types of evaluations: Human Annotation and LLM-as-a-Judge.

• Human Annotation
  Go to the Scores tab and create a new score. Next, create an Annotation Queue, and then add traces to the queue via the Traces tab. You can now manually evaluate each session. While this method offers fine-grained control, it can be time-consuming.

  

  

  

• LLM-as-a-Judge
  To automate evaluations, go to Settings → LLM Connection and add a model connection.
  Then create a custom evaluator by navigating to Create Custom Evaluator → Create New Evaluator, and set up a Default Evaluation Model.

  

  Choose a metric—such as Relevance—and specify whether you want to evaluate existing traces or only new ones.

  

  

  For inputs and outputs, map query to input and generation to output.

  

  Now, go back to Airflow and ask a question. Check the Evaluation Logs and you’ll see a score and comment like:

“The generation directly and comprehensively answers the user's query by listing all the DAGs, providing their names, and offering additional context about their purpose and scheduling. The response is focused, informative, and adds significant value to the user's understanding of the available DAGs in the Airflow environment.”

This demonstrates how Langfuse helps you evaluate, measure, and continuously improve your LLM-powered apps.

Setting Up Langfuse for Your LangChain Application

Integrating Langfuse into your LangChain app is straightforward. The configuration is handled in the airflow_chat/plugins/app/server/llm.py file, specifically within the get_user_chat_config function:

def get_user_chat_config(session_id: str, username: str = None) -> dict:
    chat_config = {
        'configurable': {'thread_id': session_id},
        'recursion_limit': 100
    }

    if os.environ.get('LANGFUSE_HOST'):
        langfuse_handler = CallbackHandler(
            user_id=username if username else session_id,
            session_id=str(session_id),
            public_key=os.environ.get('LANGFUSE_PUBLIC_KEY'),
            secret_key=os.environ.get('LANGFUSE_SECRET_KEY'),
            host=os.environ.get('LANGFUSE_HOST')
        )
        chat_config['callbacks'] = [langfuse_handler]

    return chat_config

This function builds the chat configuration and conditionally attaches the Langfuse callback handler if the environment variables are present.

The resulting configuration (chat_config) is then passed to the astream_events method like this:

async for event in self._agent.astream_events(
    {"messages": [HumanMessage(content=message)]},
    config=chat_session,
    version='v2',
)

The _agent object itself is initialized using the create_react_agent utility from LangGraph:

from langgraph.prebuilt import create_react_agent

self._agent = create_react_agent(
    self._llm,
    tools,
    checkpointer=checkpointer,
    prompt=PROMPT_MESSAGE
)

All of this logic can be found in the airflow_chat/plugins/app/server/llm.py file. This setup enables seamless event streaming and tracking via Langfuse within your LangChain-based application.

Summary

By combining the airflow-chat plugin with Langfuse, we’ve created a robust observability stack for LLM-powered workflows within Airflow. From real-time trace logging and cost tracking to advanced evaluation methods like LLM-as-a-judge, Langfuse brings clarity and control to your AI interactions. Whether you're debugging unexpected prompts, monitoring usage, or analyzing performance, Langfuse helps you iterate with confidence. For a deeper dive into all the features Langfuse offers, visit https://langfuse.com.

Introduction

Not long ago, we introduced our MCP for Airflow—designed to make operating DAGs and workflows as intuitive as chatting with a colleague. But there was a catch: it depended on Claude Desktop, which some organizations are unable or unwilling to adopt due to concerns around sharing data with Anthropic—or because of other operational, compliance, or policy-related reasons.

That got us thinking: what if we brought the same conversational interface directly into the Airflow UI?

That’s how the airflow-chat plugin came to life.

Under the hood, the plugin still leverages our existing MCP. You can run it fully embedded inside the Airflow webserver, or flexibly configure it in one of two other ways:

• As a plugin running inside the Airflow webserver (simplest setup)

• Using the plugin alongside a separate MCP service

• Or in a fully decoupled three-process architecture: Airflow webserver plugin → FastAPI server → MCP service

You also have full control over the LLM backend. MCP supports multiple providers, so you can choose to run it with Amazon Bedrock, Anthropic, or OpenAI—whichever best suits your infrastructure and policies.

This makes it easier than ever to integrate natural language interfaces into your pipelines—entirely within your stack and control.

Explore the project here:

• PyPI: airflow-chat

• GitHub: airflow-mcp-plugins

Quickstart

If you don’t have your own Airflow cluster, feel free to use the one provided in our GitHub repository.

Let’s start with the simplest setup. To integrate our plugin, add the following to your Airflow environment:

airflow-chat==0.1.0a8

Note: Version 0.1.0a8 is the latest at the time of writing. Check our PyPI repository for updates before installing.

🛠 Required Environment Variables

Add the following environment variables to your cluster configuration:

LLM_MODEL_ID=anthropic:claude-3-5-haiku-20241022
AIRFLOW_ASSISTENT_AI_CONN=airflow:airflow@http://localhost:8080
ANTHROPIC_API_KEY=your_anthropic_key

AIRFLOW_ASSISTENT_AI_CONN Explained

This connection string is used by the AI Assistant plugin to communicate with your Airflow instance. It follows this structure:

AIRFLOW_ASSISTENT_AI_CONN=<username>:<password>@http://localhost:8080

• Replace <username> and <password> with any valid Airflow credentials you want the assistant to use.

• Do not change localhost:8080, as this address is required for internal routing within the containerized environment where the plugin operates.

🔁 Model Flexibility:
You can switch LLM_MODEL_ID to use models from Bedrock or OpenAI. Simply change the prefix accordingly:

• For Bedrock: bedrock:...

• For OpenAI: openai:...

Be sure to set the correct corresponding API credentials:

• Anthropic: ANTHROPIC_API_KEY

• Bedrock: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

• OpenAI: OPENAI_API_KEY

⚙️ Additional Configuration

You can also configure optional Airflow MCP environment variables such as:

• AIRFLOW_INSIGHTS_MODE

• POST_MODE

We explain these options in detail in this blog post.

🧪 Running with Our Default Environment

If you’re using our pre-configured environment:

cp airflow-plugin-test.env airflow-plugin.env

Then open airflow-plugin.env, fill in all required values, and run:

just airflow

🧭 Accessing the Assistant

Once your cluster is running, open

 http://localhost:8080

, then navigate to:

Tools -> AI Chat Assistant

You’ll now see the Airflow Chat Assistant in action! Try asking:

"What Airflow DAGs do we have, and which ones are paused?"

🎉 Everything should work beautifully out of the box!

🔐 Access Control

Access to the AI plugin is restricted to:

• Admins, or

• Any user assigned the AIRFLOW_AI role.

Other Deployment Strategies

Let’s take a look at how our simple deployment works—and then walk through how to modify it depending on your needs.

🧩 The Basic Setup

Our lightweight Airflow Chat plugin runs both the Langchain modules and Airflow MCP locally inside the Airflow Webserver. This setup works out-of-the-box with minimal configuration.

But what if you don’t want to manage environment variables inside the Airflow instance under the same user? No problem—you can split the deployment and run MCP externally.

🚀 Option 1: Externalize the MCP Server

To run MCP as a separate component:

1. Remove the embedded MCP server from your Airflow Webserver setup.

Add these environment variables to your Airflow instance (you can use airflow-plugin.env in our repo):

TRANSPORT_TYPE=sse
mcp_host=host.docker.internal:8001
MCP_TOKEN=test_token

📌 Note: mcp_host must be updated if you're deploying the MCP server on your own infrastructure.

Remove the Airflow connection variable: AIRFLOW_ASSISTENT_AI_CONN

2. If you're using our repository, just run:

cp airflow-mcp-test.env airflow-mcp.env

Then, customize the following variables as needed (no changes required if using the default setup in our repo):

LLM_MODEL_ID=antropic:claude-3-5-haiku-20241022
AIRFLOW_ASSISTENT_AI_CONN=airflow:airflow@http://airflow-webserver:8080
TOKEN=test_token
TRANSPORT_TYPE=sse

✅ Ensure that TOKEN and MCP_TOKEN have the same value to establish a secure connection between Airflow and the remote MCP server.

To deploy:

• Run just prod if you're using our repo, or

• Use the Docker image: hipposysai/airflow-mcp:latest

• Refer to docker-compose.prod.yml for an example configuration.

🖥️ Option 2: Use the Plugin as a UI Only (Separate Langchain Logic)

If you prefer to fully decouple Langchain logic from the Airflow Webserver (and just use the plugin as a UI layer), update your airflow-plugin.env as follows:

INTERNAL_AI_ASSISTANT_SERVER=false
FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'
BACKEND_URL=http://chat_fastapi:8080

Then, for the FastAPI/Langchain service (using the hipposysai/airflow-mcp-fastapi image), define:

LLM_MODEL_ID=antropic:claude-3-5-haiku-20241022
SECRET_KEY='ThisIsATempSecretForLocalEnvs.ReplaceInProd.'
FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'
TRANSPORT_TYPE=sse
MCP_TOKEN=test_token
mcp_host=mcp_sse_server:8000
POSTGRES_USER='airflow'
POSTGRES_PASSWORD='airflow'
POSTGRES_DB='postgres'

To apply this config:

cp langchain-fastapi-test.env langchain-fastapi.env
just prod

⚖️ Option 3: Separate Backend but Share MCP

Want to offload backend logic from your Airflow Webserver without introducing a separate MCP service? You can:

1. Remove TRANSPORT_TYPE=sse from langchain-fastapi.env (or whichever file defines your backend).

2. Copy all environment variables from your MCP service (airflow-mcp.env) to your backend environment.

This option gives you better performance isolation without significantly increasing deployment complexity.

Summary

The airflow-chat plugin brings a conversational AI interface directly into the Airflow UI, enabling users to inspect DAGs, query task states, and operate pipelines using natural language. Built on top of the existing MCP architecture, it offers flexible deployment options—from running entirely inside the Airflow Webserver to more decoupled setups involving external MCP and FastAPI services. The plugin supports multiple LLM backends (Anthropic, Bedrock, OpenAI), giving teams full control over model providers and data handling. With minimal configuration, customizable environment files, and built-in access controls, airflow-chat makes AI-powered orchestration both accessible and secure—right where data pipelines live.

Introduction

A few weeks ago, we introduced Airflow MCP from Hipposys. Since then, we've fine-tuned its capabilities to better align with real-world user needs. In this post, we’ll walk you through three major updates: Safe Mode for secure operation, enhanced controls to manage DAG state, and a new way to predict upcoming DAG runs using our Airflow Schedule Insights plugin.

Run with Confidence: Safe Mode

We realized that many users—especially those working with production Airflow instances—need to be certain that MCP won’t inadvertently alter DAG states. Even if your Airflow user has permission to run, pause, or delete DAGs, you may want to use MCP strictly for read-only access. That’s why Safe Mode is now the default behavior.

In Safe Mode, only informational (GET) functions are available, ensuring you can inspect your Airflow environment without risk. To enable write capabilities (e.g. triggering or modifying DAGs), you can explicitly opt in by setting the POST_MODE environment variable.

To do this, add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "airflow_mcp": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "airflow_api_url",
        "-e",
        "airflow_username",
        "-e",
        "airflow_password",
        "-e",
        "POST_MODE",
        "hipposysai/airflow-mcp:latest"
      ],
      "command": "docker",
      "env": {
        "airflow_api_url": "http://host.docker.internal:8088/api/v1",
        "airflow_username": "airflow",
        "airflow_password": "airflow",
        "POST_MODE": "true"
      }
    }
  }
}

Once enabled, you’ll have access to all functions—including state-changing operations.

Until then, rest assured: by default, MCP operates in a safe, read-only mode, so you can explore and monitor without risking production disruptions or unscheduled runs.

Pause DAGs Before the Storm

We've all been there: an unexpected (or even expected) outage hits, and you need to urgently pause all Airflow DAGs. So you go through them one by one… but wait—some DAGs were already paused. Should you manually note them down just to restore their state afterward? That’s tedious—and error-prone.

Not anymore.

Thanks to the new change_dags_pause_status method, this process is now streamlined and reliable.

Note: The Airflow cluster I'm working with is deployed from our GitHub repo: hipposys-ltd/airflow-mcp.

Right now, here’s our current state:

Let’s use MCP to prepare for the outage.
Start with the following prompt:

"Hi, please pause all Airflow DAGs, but before doing so, list the DAGs that are currently paused. We’ll use that list to remember which ones need to be unpaused tomorrow."

As you can see, all DAGs have been successfully paused.

Now comes the tricky part: restoring normal operations without unpausing DAGs that were paused before the outage.

Run:

"Great. Now please unpause the DAGs, but make sure to leave the ones that were already paused as they are."

And just like that—done.
It only took two prompts to handle a full-cluster pause and restore cycle, cleanly and safely.

When Will It Run?

Let’s start with something simple: finding out when load_ticket_sales will run next.

To do that, we’ll need to update claude_desktop_config.json once again by enabling Airflow Insights Mode:

{
  "mcpServers": {
    "airflow_mcp": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "airflow_api_url",
        "-e",
        "airflow_username",
        "-e",
        "airflow_password",
        "-e",
        "POST_MODE",
        "-e",
        "AIRFLOW_INSIGHTS_MODE",
        "hipposysai/airflow-mcp:latest"
      ],
      "command": "docker",
      "env": {
        "airflow_api_url": "http://host.docker.internal:8088/api/v1",
        "airflow_username": "airflow",
        "airflow_password": "airflow",
        "POST_MODE": "true",
        "AIRFLOW_INSIGHTS_MODE": "true"
      }
    }
  }
}

⚠️ Note: Make sure the Airflow Schedule Insights plugin is installed in your Airflow instance for this to work.

Now, restart Claude Desktop and ask:

“When is the next run for load_ticket_sales (in EST)?”

After a short wait, you’ll get a detailed answer including the start time, end time, and expected duration—and it’s accurate. But that’s easy—load_ticket_sales is a scheduled DAG.

Let’s go a step further and try something more dynamic.

Event-Driven DAG Prediction

Now let’s look at an event-driven DAG with no fixed schedule: transform_customer_sentiment.

This DAG is triggered when:

• either transform_sales_aggregator and load_customer_feedback complete,

• or load_ticket_sales finishes on its own.
  

  

According to Claude, the next trigger will come from load_ticket_sales. Let’s wait and see…

And yes—it was triggered just one second later than predicted and finished two seconds later than expected. Even better, Claude correctly identified that load_ticket_sales was the trigger.

Querying Multiple DAGs at Once

Now let’s try asking about several DAGs in one go:

“When are the next runs for load_movie_inventory, load_box_office_reports, and transform_loyalty_program_analysis (in UTC)?”

Claude replies—none of them will run, and that’s correct!

With just a few configuration changes, you can now monitor and predict future DAG runs directly from Claude Desktop—scheduled or event-driven. No guesswork, no surprises.

Summary: Next-Level DAG Observability and Control

With the latest updates to Airflow MCP, Claude Desktop evolves from a smart interface into a powerful control and observability tool for Airflow users. You can now operate in Safe Mode for peace of mind in production, pause and restore DAG states safely during outages, and even predict when DAGs—scheduled or event-driven—will run next, thanks to the new Airflow Schedule Insights integration. Whether you're running a complex pipeline or preparing for maintenance, these upgrades ensure you're always one step ahead—with just a couple of prompts.

When building a RAG system, developers often turn to vector databases—like ChromaDB or Milvus—for storing embeddings. While popular, these solutions come with notable drawbacks:

• Proprietary options can be expensive, and even open-source databases typically require maintaining a full cluster.

• Managing infrastructure—permissions, monitoring, uptime—adds significant operational complexity.

• Scaling vector databases for large workloads isn't always straightforward or cost-effective.

• Most solutions lack native integration with AWS services like Amazon Bedrock Knowledge Bases.

To address these limitations, AWS recently introduced S3 Vector Buckets—a lightweight, scalable, and fully managed way to store and retrieve vector embeddings directly within Amazon S3. While AWS provides excellent documentation on using this with their Agent ecosystem (official guide), we created a demo project showing how to integrate S3 Vector Buckets with LangChain.

In this post, we'll walk you through the setup, using real-world data from the NYC City Planning website, and demonstrate how to extract useful insights with natural language queries.

Setting Up the Vector Bucket

Start by navigating to AWS S3, selecting Vector Buckets from the left-hand menu, and creating a new bucket.

Great — now we have an S3 vector bucket ready to store embeddings. But how do we search through those vectors? For that, we need to create a vector index.

When configuring the index, set the dimensions value to match the number of values returned by your embedding model. Since we’re using the amazon.titan-embed-text-v2:0 embedding model from Amazon Bedrock, which produces vectors of 1024 dimensions, set the dimension accordingly.

Next, you’ll need to choose a similarity measure — either Euclidean Distance or Cosine Similarity. We’ll use Cosine Similarity, as it aligns better with how Titan embeddings are designed: they capture meaning in the direction of the vector rather than its magnitude, which is exactly what Cosine Similarity measures.

Perfect! With both the vector bucket and index set up, you're now ready to run your repository and start interacting with your vector store.

Using the Vector Bucket

To get started, clone our demo repository locally:

git clone https://github.com/hipposys-ltd/s3-vector-bucket
cd s3-vector-bucket
cp template.env .env

This will generate a .env file where you’ll configure all necessary environment variables:

DEPLOY_ENV='local'
# Postgres

POSTGRES_USER='postgres'

POSTGRES_PASSWORD='postgres'

POSTGRES_DB='postgres'

# LLM Model

LLM_MODEL_ID='bedrock:...' # or use 'anthropic:...' or 'openai:...'

# Secrets (replace in production)

SECRET_KEY='ThisIsATempSecretForLocalEnvs.ReplaceInProd.'

FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'

# AWS Credentials

AWS_ACCESS_KEY_ID='...'

AWS_SECRET_ACCESS_KEY='...'

AWS_DEFAULT_REGION='...'

# Optional Keys

# ANTHROPIC_API_KEY='...'

# OPENAI_API_KEY='<YOUR OPENAI API KEY>'

# Vector Store

bucket='test-vector-bucket'

index='test-vector-index'

# Optional: Custom embedding model

# embedding_model='...'

To use Bedrock for both embedding and LLM, set LLM_MODEL_ID to a string starting with bedrock: and ensure your AWS credentials are configured.
If you prefer using models like Claude (Anthropic) or OpenAI, set the respective API key and adjust LLM_MODEL_ID accordingly (e.g., anthropic:... or openai:...).

Once configured, run:

just all

Then open your browser and navigate to

http://localhost:8501

You’ll find three tabs:

• Main – a chat interface to interact with your vector store

• Load File – to upload PDF documents and populate your vector index

• See Vectors – to explore and manage uploaded embeddings

Initially, the vector store will be empty, so let’s add some data.
Download the prepared NYC CPC dataset from:
https://hipposys-public-assets.s3.us-east-1.amazonaws.com/newsletter-assets/NYC+Planning.zip

Extract the PDF files into the data/ folder. Then, go to the Load File tab and upload them.

After processing is complete, check the See Vectors tab—you’ll see the loaded data! You can even manage the embeddings: delete individual rows, entire documents, or wipe all vectors.

Now, switch to the Main tab and try a query like:

What are the changing conditions of the Special Hudson Yards District?

You should receive a highly relevant response—your setup is working!

How It Works

The core logic of our vector integration lives in app/vector/__init__.py.

Connecting to the S3 Vector Store

We use the boto3 SDK to interact with AWS’s new S3 vector store:

self.s3vectors = boto3.client('s3vectors', region_name='us-east-1')

This client handles all interactions—storing, searching, listing, and deleting vectors in the configured S3 bucket.

Searching the Vector Bucket

The search method performs the following steps:

1. Embeds the input text using an embedding model (e.g., Amazon Titan).

2. Sends the resulting vector to the S3 bucket for similarity search.

3. Returns the top matches with metadata and distance.

def search(self, input_text):
    request = json.dumps({"inputText": input_text})
    response = self.bedrock.invoke_model(modelId=self.embedding_model, body=request)
    embedding = json.loads(response["body"].read())["embedding"]

    query = self.s3vectors.query_vectors(
        vectorBucketName=self.bucket,
        indexName=self.index,
        queryVector={"float32": embedding},
        topK=3,
        returnDistance=True,
        returnMetadata=True
    )
    return query["vectors"]

Listing and Deleting Vectors

You can also view or clean up your vector store using the following methods:

def list_vectors(self):
    query = self.s3vectors.list_vectors(
        vectorBucketName=self.bucket,
        indexName=self.index,
        returnMetadata=True
    )
    return [{**vector['metadata'], 'key': vector['key']} for vector in query['vectors']]

def delete_vectors(self, vector_keys: list):
    self.s3vectors.delete_vectors(
        vectorBucketName=self.bucket,
        indexName=self.index,
        keys=vector_keys
    )

Uploading Data to the Vector Store

The upload_file function (also in the same module) handles:

• Loading text from files

• Generating embeddings

• Storing them into the S3 vector bucket

This process is triggered when you upload documents via the Load File tab in the UI.

LangChain Integration

To integrate this logic into your LangChain agent, we wrap the search method as a LangChain Tool inside app/server/chat.py:

retriever = VectorDB()
tool = Tool(
    name="Data_Retriever",
    func=retriever.search,
    description="Searches S3 vector bucket for similar embeddings and returns matching results",
)
tools = [tool]

This enables your LangChain agent to retrieve semantically relevant data from your S3 vector bucket in response to user queries.

Conclusion

AWS S3 Vector Buckets offer a simple and scalable alternative to traditional vector databases—no clusters, no heavy maintenance, and no additional services to run. Combined with LangChain, you can now build RAG pipelines that are lightweight, cost-effective, and easy to set up.

Our demo showed how to embed and store documents, query them semantically, and retrieve meaningful answers—all using S3 as your vector store. Whether you're working with Amazon Bedrock, Anthropic, or OpenAI models, this setup provides a flexible and developer-friendly foundation for building intelligent applications.

Introduction

“English-to-SQL” is a hot topic right now—and for good reason. It’s the dream of every data analyst, data engineer, and data-savvy team: imagine asking a question in plain English and having an LLM not only understand it, but also run the exact SQL query needed to get the answer from your database or data lake.

There are many ways to implement English-to-SQL, and each approach comes with different tradeoffs in complexity, accuracy, and contextual awareness. What we’re showcasing here is just one practical and powerful path—focused on giving the LLM rich, structured context using dbt and LangChain.

Our dbt-to-English project laid the foundation by giving LLMs access to your dbt metadata, real SQL object names, and custom prompts. Now, we’ve taken it further: by integrating LangChain’s QuerySQLDatabaseTool, your LLM-powered assistant can not only understand your data environment, but actually connect to your database and run live SQL queries based on natural language inputs.

If you haven’t read our previous blog post about dbt-to-English, we recommend starting there—it sets the stage for what’s next.

Let’s explore how this new feature works in practice.

Getting Started: Connecting to the Database and Generating Fictional Sample Data

Let’s begin with a simple example to understand how the system works.

1. Configure Your Environment
Update your private.env file by adding a new key from the private.env-template file, for example:

DATABASE_URI=...

You can either use your own database or connect to the provided local one:

DATABASE_URI=postgresql://postgres:postgres@postgres:5432/postgres

2. Launch the Environment
Run the dbt-to-English environment (GitHub repo) using Docker:

docker compose up --build

3. Run dbt to generate source tables and transformation models.

 cd DbtExampleProject
 dbt build

4. Enable Database Connection in UI
In the app interface, select the catalog and manifest files. Then proceed to the System Prompt section, where a new checkbox labeled "Connect to the Database?" appears below the prompt text. Check this box.

Now add the following instruction to your prompt:

8. Execute a query to retrieve only the first 5 rows (no more) from the table, using PostgreSQL dialect. You must use your retrieval tool for this step!
   After analyzing the structure of the results (including column names, data types, and format patterns), generate entirely fictional placeholder data that illustrates the table schema only.
   Do not include any actual database values, even if they appear to be examples.

5. Run the Script
Let’s use the stg_customers model for this test. The system connects to the database via the provided DATABASE_URI, runs a query to fetch the first 5 rows, analyzes the schema, and produces fake—but structurally accurate—data.

6. Verify Query Execution
Inspect the PostgreSQL logs to confirm that the query was executed. The logs show that the tool did connect and access the table, which confirms the integration is functioning correctly.

7. Compare with Actual Data
To verify the accuracy of the AI's schema understanding, run the same query manually:

SELECT * FROM dbt.stg_customers sc LIMIT 5;

As expected, the tool generated entirely fictional data that mirrors the structure of the actual records. This proves that the placeholder data is schema-faithful and safe for demonstration or documentation purposes. Data analysts, product managers, or other non-technical users can now explore data structures confidently, without any risk of exposing real production data.

Advanced Validation Use Case

Now, let’s explore a more advanced and practical use case—identifying outdated dbt descriptions.

1. Add a Description with Known Issues

We’ll begin by editing the schema.yml file to add a description for the order_count_per_customer model. Intentionally include two incorrect statements and one correct one. For example:

“The following table is always empty but consists of four columns. Its data is generated using a SQL GROUP BY clause.”

2. Refresh Metadata

Restart the docker compose process to ensure the catalog and manifest files are regenerated with the updated model description:

docker compose down

docker compose up --build

Then, reselect the updated catalog and manifest files in the interface.

3. Update the Prompt for Comparison

Modify the prompt to instruct the LLM to perform a structured validation:

Compare the actual behavior of the current model with its dbt description (documentation).
If needed, use the QuerySQLDatabaseTool with the PostgreSQL dialect to validate the implementation.
Return your findings in a table with two columns:

• Description Item – a specific claim or element from the dbt model's description

• Status – either Correct if the behavior matches the description, or Incorrect if it does not

The table must include only the items that can be explicitly verified as correct or incorrect.
Avoid listing any assumptions, inferred logic, or undocumented behaviors.

Make sure to check the "Connect to the Database" checkbox before running the model.

4. Run the LLM

Execute the process.

As observed, the output table is highly accurate. The LLM correctly interpreted the dbt description, connected to the database, and validated each point:

• It verified that the table is not empty.

• It confirmed the presence of four columns.

• It validated the use of a SQL GROUP BY clause.

This confirms that the LLM can reliably detect outdated or incorrect documentation by cross-referencing dbt metadata with actual model behavior—providing a valuable tool for maintaining accurate, trustworthy data documentation.

How It Works Under the Hood

Let’s break down how this functionality is implemented from a technical perspective.

The core logic resides in the following file within our repository:
fastapi/server/tr_dbt_to_english.py

To enable database interaction, we utilize the QuerySQLDatabaseTool from the langchain_community package.

This tool is initialized with a SQLDatabase object, which can be constructed using the SQLDatabase.from_uri() method.

In addition to setting up the database connection, we define the LLM Chat instance—this serves as the "brain" behind the operation.

Once the QuerySQLDatabaseTool is ready, we configure a ReAct agent (LangChain tutorial). This agent integrates the LLM and the tools it can use (such as querying the database).

Finally, we use self.react_agent.stream to stream the model’s response, allowing it to dynamically interact with the database and return results in real time.

Conclusion: From Metadata to Meaningful Queries

What began as a way to help LLMs understand dbt metadata has now evolved into a practical, end-to-end system for transforming natural language into real, executable SQL—accurately and safely. By combining the structure of dbt, the flexibility of LangChain, and the power of live database connectivity, we've created a workflow that not only interprets intent but validates it against real data sources. Whether you're verifying outdated documentation, previewing schema with fictional data, or simply trying to bridge the gap between humans and databases, this integration shows what's possible when metadata, modeling, and AI work together. And best of all—it’s reproducible, extensible, and ready for your data team to explore.

Subscribe now

Introduction

DBT is a powerful tool for transforming data and managing analytics engineering workflows. But let’s face it—while it’s great for developers, it can be tough for analysts, product managers, or even other engineers to understand exactly what’s happening inside a specific model. The SQL can get dense, the logic isn't always obvious, and while DBT Docs and DBT Cloud’s Explore are helpful, they fall short when it comes to things like query-level lineage or automatic explanations of what a model is actually doing.

That’s why we built DBT to English—a tool that gives you LLM-powered insights into your DBT project. Whether you're using Bedrock or Anthropic, you can plug in your preferred LLM and get clean, contextual, human-friendly interpretations of your models, SQL logic, metadata, and lineage. Plus, it’s fully customizable: tweak the system prompt to get documentation that fits your team’s voice and style.

In this blog post, we’ll walk you through how to get started with the tool and show you how it can bring clarity and accessibility to even the most complex DBT projects.

Running Locally

To get started, clone the repository: https://github.com/hipposys-ltd/dbt-to-english

Then, copy the environment template file and create your local .env configuration:

cp private.env-template private.env

In your .env file, set the LLM_MODEL variable depending on the provider you're using:

• For Anthropic models, start the value with Anthropic: and provide your ANTHROPIC_API_KEY.

• For Amazon Bedrock, start with Bedrock: and include your AWS credentials and connection settings.

Once configured, build and start the app with Docker:

docker compose up --build

After it’s running, open your browser and go to http://localhost:8501/ to access the Streamlit interface.

DBT to English Translation

When you open the UI, you'll see a form that asks for two files: catalog.json and manifest.json.
If you don’t have your own DBT project, you can use sample files from our repository:
DbtExampleProject/target/catalog.json and DbtExampleProject/target/manifest.json.

Once you've selected the files, click on “Parse JSON Files.”
This will reveal two additional fields:

1. A field to select the nodes (models) you want to parse or generate documentation for.

2. A System Prompt field that defines the instruction sent to the LLM.

   

Choose a few models and leave the System Prompt as-is for now, then click “Run LLM.”

You’ll receive a detailed, plain-English explanation of your selected model.

• First, you'll see a natural language description—no SQL syntax or complex calculations.

  

• Next, an interactive graph will be generated. Unlike dbt Docs or dbt Cloud, this graph includes not just model-level lineage but also internal relationships like CTEs and calculations.

  

• Following that, SQL Logic & Calculations and Columns Table will show how each column was derived, including its source tables.

  

  

• Finally, you'll see a Dependencies Table with brief explanations for each dependency.

Pretty cool, right?

Want to change the output?

Perfect! That’s exactly what the System Prompt is for. Let’s go back to it and take a closer look.

The response you received was directly shaped by the System Prompt. Feel free to experiment—change the structure, add your own instructions or formatting, rearrange the output, etc.

Important: If you modify the diagram-related instructions, please keep the example section as-is.
Our UI relies on that example to recognize the diagram portion in the LLM response and render it as an interactive graph.

Conclusion

"DBT to English" is our way of making DBT projects more accessible, understandable, and human-friendly—no matter your technical background. By leveraging the power of large language models, we’ve opened the door for teams to explore data models, understand complex SQL logic, and visualize lineage at a much more granular level—all through a clean, conversational interface.

But this is just the beginning.

We enthusiastically invite the community to contribute to this open-source initiative—whether you're interested in adding new features, supporting more LLM providers, improving prompts, or enhancing the UI. Your feedback and contributions will help shape the future of how we understand and interact with modern data stacks.

Visit our GitHub repository at https://github.com/hipposys-ltd/dbt-to-english to join the project, submit pull requests, or share your ideas. Let’s make DBT more approachable!