When we first introduced Airflow Schedule Insights, we told the story of Dr. Movie, a cinema chain struggling with a “scheduling black box.” They had DAGs that silently refused to run, resource bottlenecks from over-parallelization, and zero visibility into when their next asset-based trigger would actually fire.

That story helped explain the problem.

But with Airflow 3, we decided to evolve the example environment.

Instead of fictional cinema DAGs, we now ship example educational DAGs that mimic real academic data platform dependency patterns. These DAGs are not production pipelines — they simply replicate common educational orchestration structures and asset relationships so users can test complex scheduling scenarios in a realistic setting.

In other words:
They copy educational DAG dependency patterns — not institutional data.

This makes the demo environment more relatable for universities and training organizations, while remaining lightweight and safe to use as a sandbox.

And alongside that shift, Schedule Insights v0.2.0 is officially out — fully migrated, dependency-light, and optimized for Airflow 3.

pip install airflow-schedule-insights

👉 GitHub: https://github.com/ponderedw/airflow-schedule-insights

What Changed in v0.2.0 (Airflow 3 Migration)

Under the hood, 22 files changed. If you’re a plugin developer, these are the silent Airflow 3 metadata changes you need to understand.

1️⃣ The Great Renaming: Datasets → Assets

Airflow 3 formalized the shift from “datasets” to “assets.”

Old:

task_outlet_dataset_reference

New:

task_outlet_asset_reference

If your plugin queries the metadata DB directly, this rename will break you.

2️⃣ Logic Flip: is_active → is_stale

Airflow 2:

DagModel.is_active == True

Airflow 3:

DagModel.is_stale.is_(False)

The column is gone. The logic is inverted. If you don’t update this filter, your visibility logic silently fails.

3️⃣ Asset Dependencies Are Trees, Not Lists

Airflow 3 introduces conditional asset expression9s with nested AND / OR logic.

Example:

(Asset_A AND (Asset_B OR Asset_C))

This is no longer a flat dependency list.

We implemented a recursive evaluator that parses the new asset_expression JSON structure so Schedule Insights can still predict:

• When a DAG will run

• What it’s waiting on

• Which branch of logic is currently blocking execution

Without this, “next run” predictions are simply wrong.

4️⃣ Better Performance, Fewer Dependencies

We removed pandas and numpy.

Run duration medians are now computed directly in SQL using percentile_cont via SQLAlchemy.

Result:

• Faster installs

• Lower memory usage

• Cleaner dependency graph

Example Educational DAGs (Dependency Patterns Only)

For Airflow 3, the repository now includes an Academic Data Platform example DAG suite.

These DAGs simulate patterns such as:

• Ingestion → normalization → transformation → consolidation

• Dataset/asset-triggered execution

• Hybrid cron + asset scheduling

• Conditional AND/OR asset logic

• Reporting layers dependent on multiple upstream assets

They exist purely to demonstrate:

• Complex asset expressions

• Multi-layered DAG orchestration

• Real-world dependency graphs

No real institutional integrations.
No sensitive data.
Just realistic dependency topology.

This makes Schedule Insights far more meaningful to demonstrate than the original cinema example, especially when working with Airflow 3’s asset trees.

Core Features (Now Tested on Complex Asset Graphs)

🛠 The Next Run Table — No More Silent Failures

Asset-driven DAGs often “don’t run” for subtle reasons:

• Paused DAGs

• Waiting on asset expressions

• Hybrid cron + dataset logic

• Unmet upstream conditions

The Next Run table explains exactly why:

• “This DAG is paused and will not run.”

• “No schedule or dependencies found.”

• “Waiting for upstream Asset/DAG triggers.”

📊 The Schedule Impact Simulator

Want to change:

*/25 → */5

Before committing?

Simulate it.

The UI highlights impacted future runs in purple and shows how the change propagates through downstream asset dependencies.

This is especially useful in complex multi-layer DAG graphs.

🕒 Timezone-Aware Gantt Charts

The plugin detects your local timezone and renders:

• Historical runs

• Predicted runs

• Gantt charts

No more UTC mental math.

🔌 Programmatic Insights via API

We exposed the scheduling engine via FastAPI endpoints:

• get_future_runs_json

• get_next_future_run_json

Your internal tooling can now query predicted Airflow behavior programmatically.

Visualizing Structure with airflow-dag-dependencies

To complement Schedule Insights, you can use our companion plugin:

airflow-dag-dependencies

Schedule Insights tells you when a DAG will run.
airflow-dag-dependencies shows you how everything connects.

Together they provide:

• Structural visibility (dependency graph)

• Temporal visibility (future execution prediction)

Especially powerful with Airflow 3’s conditional asset trees.

Why This Matters at Scale

As more teams contribute DAGs to a shared Airflow environment — as is the case at Uncommon Schools, where multiple teams orchestrate data across a large network of schools — scheduling complexity compounds quickly. Each team adds their own timing requirements, and as run durations drift, overlap and resource contention become increasingly hard to spot from any single vantage point. Schedule Insights gives you a cross-team view — so you can see potential collisions before they become incidents. And when a DAG pile-up does occur, the historical run data lets you work backward and reconstruct what likely triggered the cascade.

Try It Today

Spin it up locally:

git clone https://github.com/ponderedw/airflow-schedule-insights
cd airflow-schedule-insights
docker compose -f local_airflow_3/docker-compose.yml up

Then navigate to:

Browse → Schedule Insights

You’ll see the plugin running against realistic educational dependency patterns — not fictional cinema workflows.

Airflow 3 made orchestration more powerful.
Schedule Insights makes it understandable.

Introduction

In our previous post, we explored how to bridge the gap between complex data structures and end-users by grounding natural-language queries in a Cube-governed semantic layer. By using RAG, we enabled anyone—from teachers to board members—to query data without needing to touch SQL, GraphQL, or REST APIs.

But while answering a question with text is a great start, data is often best understood visually. Even for those who can write code, the goal is often the same: getting to the insight as quickly as possible. We asked ourselves: Can we move beyond simple text responses and visualize this data directly within the chat interface?

The answer is a resounding yes.

In this post, we’re introducing a new capability that turns conversational insights into immediate, actionable BI charts. We’ve implemented this by leveraging Streamlit’s native charting capabilities combined with custom logic for parsing the AI’s response.

Of course, we aren’t alone in this pursuit—the entire industry is racing toward “conversational BI.” This is simply our take on the problem. Our hope is that this specific implementation provides a blueprint or some inspiration for bringing similar capabilities to your own organization. If you have a different approach or wider experience in this space, we’d love for you to share your thoughts in the comments.

You can find the updated code in the same repository here:

👉 GitHub: https://github.com/ponderedw/dbt-to-cube

Let’s dive into how we move from a simple chat response to a fully rendered chart.

One quick note before we dive in: all of the examples and visualizations in this post use synthetic sample data generated as part of the demo environment.

Quickstart

Ready to see it in action? You can get the entire environment running in just a few steps.

1. Clone the repository: git clone https://github.com/ponderedw/dbt-to-cube

2. Configure your environment: Create your .private.env file from the .private.env.template as explained here. This ensures your LLM provider and credentials are ready to go.

3. Launch the stack: Simply run just all. This automated workflow handles the heavy lifting: spinning up the database, generating mock data, running dbt models, and ingesting metrics into both Cube and Superset. (You can read more about the underlying architecture here).

4. Access the UI: Head over to http://localhost:8501/, enter your STREAMLIT_PASSWORD, and you’re ready to chat with your data.

From Questions to Visuals

Start by getting familiar with the metadata the RAG layer has indexed:

“Hey. What measures and dimensions do we have?”

Once the AI confirms the available metrics, it’s time to move beyond text. Let’s ask for a specific visualization:

“Can you visualize the total number of course enrollments by difficulty level using a pie chart?”

The result? A fully rendered BI graph in just 11 seconds.

Let’s try a different metric and a more dynamic chart type:

“Return the number of students with excellent performance across academic years and visualize it using a line chart.”

In seconds, you receive an amazing, interactive line chart—mapping complex trends without you ever touching a visualization library.

More to Explore

The system is flexible enough to handle various chart types and comparison logic:

• Area Charts: “Can you create an area chart showing the number of students with excellent performance compared to the total number of enrolled students across semesters?”
  

• Scatter Charts: “Can you visualize the relationship between course difficulty level and average course GPA using a scatter chart?”
  

How does it work?

To enable these visualizations, we didn’t need to reinvent the wheel. We kept the same agent-based architecture and Streamlit frontend but introduced two key modifications to the workflow:

1. Enhancing the Prompt (The “Brain”)

We updated our LangChain prompt to give the agent “visual awareness.” It now understands how to structure its output so the frontend knows when a chart is requested.

• Where to look: You can find the specific instructions we added in cube-to-rag/app/server/chat.py.

2. Intelligent Response Parsing (The “Bridge”)

Once the AI generates a response, the Streamlit application needs to know whether to display a paragraph of text or render a dynamic chart. We updated the handling logic to parse the AI’s response and trigger the appropriate Streamlit chart component.

• Where to look: The updated logic is located in streamlit/Main.py.

By separating the “intent” (in the prompt) from the “rendering” (in Streamlit), we maintain a clean separation of concerns while delivering a seamless user experience.

Conclusion

This experiment marks a significant step in making data truly accessible. By moving from simple text-based answers to dynamic BI visualizations, we’ve closed the loop between asking a question and gaining a visual insight.

The beauty of this approach lies in its governance. Because the AI is still grounded in the Cube semantic layer, the charts aren’t just “hallucinated” guesses—they are visual representations of verified business logic. Whether it’s a school administrator tracking student performance or a developer looking for a quick metric check, the interface provides a “hands-free” BI experience that stays accurate at scale.

What’s Next?

While this prototype demonstrates the power of combining RAG with Streamlit, there is still much to explore:

• Embedded BI Dashboards: Moving beyond single charts to embed fully functional, pre-built dashboards (like those from Apache Superset) directly into the chat when a high-level overview is requested.

• Advanced Interactivity: Allowing users to drill down into chart elements through further conversation or “chatting with the legend.”

• Complex Multi-Step Reasoning: Improving how the agent handles requests that require joining multiple cubes or complex pre-aggregations.

We believe that the future of BI isn’t just a static dashboard; it’s a conversation that adapts to the user’s needs in real-time. We hope this project helps you kickstart your own journey into conversational analytics.

Join the Conversation

We’d love to hear your thoughts!

• Did this setup work for your data?

• Do you prefer a different library for parsing AI-generated charts?

• What’s the biggest challenge you’ve faced with Text-to-Chart workflows?

Drop a comment below, or better yet, open an issue or PR on the GitHub repository.

Introduction

In one of our previous posts, we showed how dbt, Cube, and Superset can work together to automate and standardize the semantic layer. The response from the community was overwhelmingly positive but it also raised a recurring question: why introduce Cube between dbt and Superset instead of connecting BI tools directly to the database?

The answer is consistency at scale. A single, shared semantic layer ensures that metrics, dimensions, and business logic are defined once and reused everywhere – across BI tools, APIs, and downstream applications – rather than being reimplemented in each consumer.

This naturally led us to a bigger idea. If Cube already exposes a governed semantic layer through SQL, REST, and GraphQL, why force natural-language analytics back into brittle English-to-SQL translations? What if users could converse directly with the semantic layer instead?

In this post, we explore how to turn Cube analytics into a natural-language interface using RAG. We use RAG to store and retrieve Cube metrics and dimensions, allowing each user query to be grounded in the correct semantic context before any query is generated or executed. This keeps conversations aligned with predefined business logic while remaining flexible and intuitive for users.

Let’s walk through how it works.

👉 GitHub: https://github.com/ponderedw/dbt-to-cube

Quickstart

1. Clone the repository:
   👉 https://github.com/ponderedw/dbt-to-cube

2. Copy .private.env.template to .private.env and configure the required secrets for your LLM provider.

3. just all

Once everything is up and running:

• Open:

http://localhost:8000/

• and navigate to the cube_schemas collection.
  You’ll see a set of vectors representing Cube metadata – dimensions, measures, and cubes.
  These embeddings are generated using the cube-to-rag library (https://pypi.org/project/cube-to-rag/), which we use to extract and embed Cube schema metadata into the vector store.
  

• Next, open

http://localhost:8501/

• If you’ve set STREAMLIT_PASSWORD in .private.env, use it to log in.
  

You’re all set, start asking questions and explore your Cube analytics through natural language.

Talking to Your Data

Let’s start simple:
“What metrics and dimensions do we have?”

Great. Now let’s explore the data itself:
“Show all available course names.”

Nice. Let’s narrow it down:
“What metrics are available for Linear Algebra?”

Now for a concrete business question:
“How many enrollments do we have across all courses?”

And finally, let’s push it a bit further with a more complex query involving filters and breakdowns:
“What is the student engagement score for the Data Structures and Algorithms course, broken down by semester?”

And just like that, everything works. Metrics stay consistent, dimensions are respected, and the conversation maps cleanly to the semantic layer

Conclusion

This experiment shows one possible direction for moving from static semantic layers to truly conversational analytics. By grounding natural-language queries in Cube’s governed semantic layer using RAG, we avoid many of the pitfalls of naive text-to-SQL approaches: inconsistent metrics, broken joins, and logic that drifts away from business definitions. Instead, users can explore data conversationally while staying aligned with a single source of truth.

That said, this is not the solution – it’s our attempt at solving a real and growing problem. There are open questions around performance, UX, evaluation, security, and how this approach compares to alternatives such as agent-driven query planning, stricter semantic parsers, or hybrid BI experiences. We’re certain there are improvements to be made, edge cases we haven’t covered, and ideas we haven’t considered.

We’d love to hear what others think about this: What works? What doesn’t? What would you do differently? If you’re experimenting with similar approaches – or taking a completely different path – your feedback, suggestions, and alternatives are more than welcome.

Introduction

Most teams have been there. You start storing company documentation as Word files or PDFs because it’s quick, familiar, and temporary. A proof of concept, a contract, a report – throw it into a folder and move on. But over time, that “temporary” storage often becomes a critical production knowledge base.

At that point, problems emerge: documents are hard to search, relationships are implicit, extracting structured data is manual, and automation becomes painful.

A common approach today is “just put everything into RAG.” While retrieval helps with search, it doesn’t solve the core issue: the information itself is still unstructured. If the content were normalized into a machine-friendly format (like JSON with well-defined schemas), working with it would be faster, more reliable, and would open new workflows – analytics, validation, integrations, and automation.

That gap is what inspired this experiment. I wanted a way to turn business documents into structured, queryable data using LLMs. The result is StructuDoc, an open-source proof of concept that combines document parsing, image understanding, schema discovery, and prompt tracking into a single pipeline. It can move PDFs and DOCX files into structured datasets, identify common schemas, and store everything in S3.

This post introduces StructuDoc, explains how it works, and shares the reasoning behind its design. It’s offered in a spirit of sharing rather than as a commercial product – we’d love your feedback, alternative approaches you’ve tried, or contributions to the project.

• GitHub: https://github.com/ponderedw/StructuDoc

• Docker Hub: https://hub.docker.com/repository/docker/pondered/structudoc/general

Quickstart with StructuDoc

StructuDoc makes it easy to turn PDFs and DOCX files into structured JSON, discover common schemas, and even analyze images using AI. Here’s how to get started in minutes:

1. Clone the repository

git clone https://github.com/ponderedw/StructuDoc.git
cd StructuDoc

2. Set up environment variables

Copy the example file:

cp .env.example .env

Edit .env to configure your local storage (MinIO) or S3, and optionally your LLM provider keys:

ENV=local
SOURCE_BUCKET=minio/source-bucket
MINIO_HOST=http://localhost:9000
MINIO_SECURE=false
AWS_ACCESS_KEY_ID=admin
AWS_SECRET_ACCESS_KEY=password
LLM_MODEL=Bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0

3. Start the application (this file can be executed independently of the repository)

docker compose up -f docker-compose-prod.yml

• Streamlit UI:

 http://localhost:8501

• FastAPI backend:

 http://localhost:8080

• MinIO console (login: admin/password):

 http://localhost:9001

4. Upload and process documents

1. Go to Upload Source Files in the UI.

2. Upload DOCX or PDF files.

3. StructuDoc will automatically:

   • Convert PDFs to Markdown

   • Extract images

   • Store all files in structured folders

5. Parse and analyze with AI

• Navigate to Parse Files With LLM:

• Analyze images with AI prompts

• And remember to save those descriptions to your s3 bucket:

• Discover common schemas across multiple documents

• Parse your documents to JSON

Now you have structured, queryable data from your documents – ready for testing, analysis, or building pipelines!

Conclusion

StructuDoc is an open-source experiment for turning PDFs and DOCX files into structured, queryable data. Instead of relying solely on RAG for search, it lets you extract JSON, discover common schemas, and analyze images with AI. This isn’t a fully polished product – it’s a proof of concept born from real challenges we’ve faced, shared in the hope it might help others or spark new ideas. We’d love to hear your feedback, learn about alternative approaches you’ve tried, or see contributions that improve StructuDoc.

GitHub: https://github.com/ponderedw/StructuDoc
Docker Hub: https://hub.docker.com/repository/docker/pondered/structudoc/general

Introduction

dbt has become the backbone of modern analytics engineering, and one of its best features is CI: the ability to compare a developer branch with main, rebuild only the changed models in a temporary schema, and use production data from unchanged models. It’s an amazing way to catch issues before they hit production.

But what about incremental models and snapshots? Metrics like Grade Point Average whose day-by-day variations are captured in a dbt snapshot are then tracked and rendered in various contexts over time. This per-student metric is calculated in one environment (production) each day to ensure a clear source of truth. But for CI…that’s where things get tricky.

Both incremental models and snapshots have different logic for initial runs vs. upsert runs. In CI, everything is always built from scratch – so your CI may pass, only for the query to fail the moment it meets existing data or objects in production.

To solve this for Uncommon Schools, we built a tiny but powerful solution – and now we’re sharing it with everyone:
📦 dbt-incremental-ci (https://pypi.org/project/dbt-incremental-ci/)

Quickstart

Get up and running in under 5 minutes.

dbt-incremental-ci copies your production incremental models and snapshots into your CI schema so you can test against real production data, not empty tables.

Installation

pip install dbt-incremental-ci

Option 1: Using a Local Manifest File

Use this approach if your production manifest.json is stored in S3, GCS, or available locally.

Step 1: Download your production manifest

# From S3
aws s3 cp s3://my-dbt-artifacts/prod/manifest.json ./prod_manifest.json

# From GCS

gsutil cp gs://my-dbt-artifacts/prod/manifest.json ./prod_manifest.json

Step 2: Copy production tables into your CI schema

dbt-incremental-ci \
  --prod-manifest-path ./prod_manifest.json \
  --dbt-project-dir . \
  --database-uri “postgresql://user:pass@host:5432/db” \
  --ci-schema “ci_pr_123” \
  --threads 4

Step 3: Run dbt with the copied production data

dbt build --select state:modified+ --defer --state ./prod_manifest.json

Step 4: Run your tests

dbt test --select state:modified+ --defer --state ./prod_manifest.json

You’re now running incremental models and snapshots against actual production state—not empty CI tables.

Option 2: Using dbt Cloud

If you’re on dbt Cloud, you don’t need to download the manifest—the tool fetches it automatically from the API.

Step 1: Set your credentials

export DBT_CLOUD_API_TOKEN=”your_token”
export DBT_CLOUD_ACCOUNT_ID=”12345”

Step 2: Copy production tables into your CI schema

dbt-incremental-ci \
  --dbt-cloud-job-id “67890” \
  --dbt-project-dir . \
  --database-uri “$DATABASE_URI” \
  --ci-schema “ci_pr_123” \
  --threads 4

The tool will automatically:

• Connect to the dbt Cloud API

• Fetch the latest successful production run

• Download the manifest

• Copy the required tables

Step 3: Run dbt

dbt build --select state:modified+ --defer

Step 4: Run your tests

dbt test --select state:modified+ --defer

What Just Happened?

The tool:

1. Identified changed models using state:modified+

2. Filtered to incremental models and snapshots that already exist in production

3. Copied those production tables (with data!) into your CI schema

4. Preserved custom schemas (e.g., prod_marts → ci_pr_123_marts)

This ensures your CI pipeline behaves like a real incremental run—catching bugs that only appear when existing data is present.

Preview Before Running (Dry Run)

Want to see what will be copied without actually doing anything?

dbt-incremental-ci \
  --prod-manifest-path ./prod_manifest.json \
  --dbt-project-dir . \
  --database-uri “postgresql://user:pass@host:5432/db” \
  --ci-schema “ci_pr_123” \
  --dry-run \
  --verbose

This prints exactly which tables would be copied and the SQL that would run.

Final Thoughts

Incremental models and snapshots are some of the most powerful parts of dbt – but they’re also uniquely tricky to validate in CI. Full rebuilds simply don’t reflect how these models behave in production, which means some of the most painful bugs can slip past your pipeline undetected.

dbt-incremental-ci is our attempt to fix that gap. It’s a lightweight, practical solution that brings real production state into your CI runs so you can test incremental logic the way it actually runs in the real world.

But this isn’t the final word – far from it.
This is our first take at solving the problem, and we know the community will have ideas we haven’t thought of yet. If you have a better approach, want to improve this one, or see opportunities to make it smarter, faster, or more general – we’d love your help.

👉 PyPI: https://pypi.org/project/dbt-incremental-ci/
👉 GitHub: https://github.com/ponderedw/dbt-incremental-ci – issues, discussions, and PRs are all very welcome.

Introduction

A couple of weeks ago, we shared several blog posts about different ways we’re making data more accessible and interactive: our Superset MCP (link), loading dbt metadata into GraphDB (link), and an amazing AI-powered UI plugin for Airflow that lets you interact with your workflows directly in the UI (link).

Today, I want to bring all three together and introduce superset-chat - a library that helps you manage both your dbt and dashboard metadata right in the Superset UI, powered by AI, Superset MCP, and our Falkor DB plugin. You can check it out on GitHub: superset-mcp-plugins.

If you’ve ever felt a disconnect between BI and transformation logic, this library might resonate. It can answer questions like, “Which dashboards depend on source X?” - even when those sources aren’t referenced directly. While still a POC, superset-chat has already helped us organize educational metadata, and we’re excited to share it with the community. Maybe you’ll help improve it or come up with something entirely new!

Quickstart

You can try superset-chat in two ways:

1. Use your existing Superset

If you already have a Superset deployment, you can integrate the plugin directly:

• Install the plugin:

pip install superset-chat

• Update your superset_config.py to register the plugin:

FLASK_APP_MUTATOR = lambda app: init_custom_views(app)
def init_custom_views(app):

    from superset_chat.ai_superset_assistant import AISupersetAssistantView

    appbuilder = app.appbuilder
    appbuilder.add_view(
        AISupersetAssistantView,
        “AI Superset Assistant”,
        icon=”fa-robot”,
        category=”Custom Tools”
    )

• Make sure you have the following environment variables set for your deployment:

LLM_MODEL_ID=bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0  # Must start with ‘bedrock’, ‘anthropic’, or ‘openai’
# ANTHROPIC_API_KEY=...              # Required if using Anthropic
AWS_ACCESS_KEY_ID=’...’             # Required if using Bedrock
AWS_SECRET_ACCESS_KEY=’...’         # Required if using Bedrock
SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://postgres:postgres@postgres:5432/postgres
SUPERSET_API_URL=http://localhost:8088
SUPERSET_USERNAME=superset_admin
SUPERSET_PASSWORD=superset
GRAPH_DB=falkordb                     # Optional: enable GraphDB connection
GRAPH_HOST=...                        # Optional: if using a remote GraphDB
GRAPH_USER=...
GRAPH_PASSWORD=...

Restart Superset, log in, and open AI Superset Assistant under Custom Tools.

2. Run the local deploy

If you don’t have a Superset deployment, you can use our preconfigured local setup:

• Clone the repository:

git clone https://github.com/ponderedw/superset-mcp-plugins.git
cd superset-mcp-plugins

• Copy the environment template and configure credentials:

cp template.env .env

• The minimal .env for local deploy (we use FalkorDB in this tutorial) can be:

LLM_MODEL_ID=bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0
# ANTHROPIC_API_KEY=...
AWS_ACCESS_KEY_ID=’...’
AWS_SECRET_ACCESS_KEY=’...’
SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://postgres:postgres@postgres:5432/postgres
SUPERSET_API_URL=http://localhost:8088
SUPERSET_USERNAME=superset_admin
SUPERSET_PASSWORD=superset
GRAPH_DB=falkordb

• Run Superset with superset-chat:

just run_superset

• Navigate to your Superset instance (usually http://localhost:8088) and open AI Superset Assistant under Custom Tools.

This local deploy is a fully working Superset + superset-chat environment, so you can start experimenting even if you don’t have your own deployment.

Using superset-chat: Examples

If you’re using our project, we’ve included a dbt project that was automatically loaded into FalkorDB, along with Superset charts and dashboards that were imported into Superset. You can explore all of them in the UI as well as in dbt docs.

Now, let’s try out the plugin with a few example queries:

1. Superset-related question:

“Quick question – what’s our current count of Superset dashboards and charts?”

As expected, superset-chat lists all the Superset metadata correctly – dashboards and charts – just like in the UI. Cool!

2. dbt-related question:
   
   

“Hey! Do you know how many dbt models we have in total?”

We can check the answer in dbt docs, and it’s completely correct. Awesome!

3. Combined dbt + Superset question:

“Could you help me find all the dbt sources (not models) used in the Student Academic Standing chart?”

Amazing! superset-chat traced the chart all the way back to the underlying database tables. This shows how we can connect Superset dashboards directly to dbt metadata seamlessly.

Connecting BI and Transformation Metadata with AI

Connecting BI and Transformation Metadata with AI
With superset-chat, you can finally see your BI layer and your transformation logic in one place, without manually tracing dependencies or opening multiple tools. By unifying dbt lineage and Superset metadata, the plugin allows you to:

• Ask natural-language questions spanning both dashboards and data models.

• Trace dashboards and charts back to the underlying source tables.

• Verify dbt lineage directly from Superset without leaving the UI.

• Quickly get insights that previously required multiple queries or digging through documentation.

Even as a POC, superset-chat already shows how AI can bridge the gap between data transformation and business intelligence. It’s also beginning to influence some of our internal workflows - for example, in our work with Uncommon Schools we’re exploring how the assistant can accelerate outage protocols, particularly when on-call responders aren’t familiar with the specific code paths involved. This kind of scenario highlights the value of unifying metadata and making it accessible through natural language.

Whether you’re exploring dashboards, analyzing dbt models, or troubleshooting lineage issues, this tool turns what was once a fragmented process into a single, interactive experience. We’re excited to share this with the community and invite you to try it, suggest improvements, or build on it for your own workflows. It’s a step toward truly connected, AI-powered data exploration.

Introduction

Rebuilding the same semantic layer over and over across different BI tools has always felt… unnecessarily painful. Most dashboards still reach directly into databases, each with their own idea of what a “metric” is, so you end up maintaining the same logic in three or four places. Even tools built to solve this – like Cube.js – don’t really solve the end-to-end problem of getting consistent metrics into your BI layer.

Out of curiosity (and a bit of frustration), I started tinkering with a small proof of concept to see whether dbt, Cube.js, and Superset could be wired together into a single automated flow. The result is dbt-to-cube:

https://github.com/ponderedw/dbt-to-cube

It’s very much a work in progress, but it already connects dbt → Cube.js → Superset and keeps metrics in sync without any manual steps. Sharing it here in case others are experimenting with similar setups – I’d genuinely love feedback, ideas, or contributions from anyone who gives it a try.

Quickstart

See dbt-cube-sync in action! This walkthrough demonstrates a full semantic layer pipeline—from dbt transformations to BI dashboards—without any manual metric duplication.

Already Have dbt, Cube.js, and Superset?

If your stack is already deployed, you don’t need Docker. Just install dbt-cube-sync and point it to your existing deployment:

pip install dbt-cube-sync

# Convert dbt models to Cube.js schemas

dbt-cube-sync dbt-to-cube \

  --manifest /path/to/manifest.json \

  --catalog /path/to/catalog.json \

  --output /path/to/cube/output

# Sync Cube.js schemas to Superset

dbt-cube-sync cube-to-bi superset \

  --cube-files /path/to/cube/output \

  --url http://your-superset:8088 \

  --username admin \

  --password admin \

  --cube-connection-name Cube

This workflow works with Superset, Tableau, or PowerBI, giving you full control over your production-ready semantic layer.

Starting from Scratch? Spin Up Everything Locally

If you want a full demo stack including sample data, all services run locally in Docker: PostgreSQL, dbt, Cube.js, Superset, and dbt-cube-sync.

Prerequisites

• Docker & Docker Compose

• Git

• ~10 minutes

Step 1: Clone & Start the Stack

git clone https://github.com/ponderedw/dbt-to-cube
cd dbt-to-cube
docker-compose up --build

This will:

1. Initialize PostgreSQL with sample data

2. Run dbt transformations on 46 pre-built models

3. Start Cube.js API serving your semantic layer

4. Configure Superset with admin access and database connection

5. Automatically generate Cube.js schemas from dbt and sync them to Superset

Step 2: Explore Your Data Stack

Once the logs show “Pipeline completed successfully!”, you can access:

• Superset: http://localhost:8088 — login: admin/admin

  • 46 datasets auto-created from dbt models

  • Column metadata and metrics like Average GPA and Pass Rate

• dbt Docs: http://localhost:8080 — full DAG and column-level documentation

• Cube.js API: http://localhost:4000 — REST and GraphQL endpoints

Step 3: Watch the Pipeline in Action

docker-compose logs -f dbt-cube-sync

The pipeline runs in three automated phases:

1. dbt Parsing – reads manifest.json & catalog.json, finds models, and maps dependencies

2. Cube.js Schema Generation – converts dbt models into Cube.js dimensions and measures

3. BI Tool Sync – pushes datasets and metrics to your BI tool with proper metadata

Everything happens automatically for all models—no manual intervention required.

✅ What You’ve Built

In minutes, you now have a unified semantic layer pipeline:

1. Single source of truth: Metrics defined once in dbt

2. Automatic propagation: dbt → Cube.js → BI tools

3. Cross-tool flexibility: Works with Superset, Tableau, PowerBI

4. Developer-friendly: Version-controlled, testable, and documented

Replace sample data with your warehouse and dbt models, and you have a production-ready semantic layer – no metric inconsistencies across tools.

The Takeaway

This little experiment convinced me that an automated semantic layer doesn’t have to be complicated. Define metrics once in dbt, let a small tool translate them to Cube.js, and push them into your BI tool of choice – no more re-implementing the same logic across dashboards.

If you end up trying out dbt-to-cube, I’d be really interested to hear what works, what breaks, or what you’d want it to support next. It’s still a POC, but with feedback from others running into the same problems, it could grow into something genuinely useful for the community.

Always happy to chat if you’re experimenting with semantic layers or have ideas on making this whole workflow smoother!

Introduction

At Ponder, we’ve developed several plugins for Airflow 2.* and recently began migrating them to Airflow 3. Among our most valuable tools are Airflow Schedule Insights and Airflow Chat.

During the migration process, we discovered that Airflow 3.1.1 removed the DAG Dependencies tab that was present in Airflow 2.

Rather than wait for this feature to return, we saw an opportunity: we could build a plugin to restore this functionality, help the community migrate to Airflow 3 more smoothly, demonstrate how to create plugins for Airflow 3, and even add improvements we found missing in Airflow 2.

The result is airflow-dag-dependencies. In this post, we’ll walk you through how we built it and how it helps us manage our many educational DAGs.

Quick Start

To use our plugin, simply install airflow-dag-dependencies in your Airflow environment. If you don’t have an environment set up, clone our GitHub repository and run docker compose up.

After a few minutes, once Airflow is running, navigate to http://localhost:8080/ and log in with username “airflow” and password “airflow”. You’ll find a dozen educational DAGs — load_class_attendance, load_course_evaluations, and others.

Some are triggered by datasets, others by DAGs, and some run on cron schedules. The challenge? Understanding how they all connect.

This is where our plugin shines. Navigate to the Browse menu and click on DAG Dependencies.

You’ll immediately see all the relationships visualized clearly. Notice how transform_program_effectiveness has two dependencies: a cron schedule and a dataset trigger.

You can also spot that load_curriculum_catalog and load_student_enrollment_data have no dependencies or schedules—they won’t run automatically.

And you’ll see that transform_academic_performance is triggered by load_course_evaluations.

Enhanced Filtering

But we didn’t stop at recreating Airflow 2’s functionality—we added the filtering capabilities we always wished it had. Want to see only the direct downstream dependencies of transform_student_feedback_analysis and all its upstream dependencies? Simply enter +transform_student_feedback_analysis+1 in the search bar. The graph instantly filters to show only the relevant connections.

The search syntax is intuitive: use + before a DAG name to include upstream dependencies, after to include downstream, or both for the full picture. Add a number (like +dag_name+2) to control how many levels deep to traverse. You can also search multiple DAGs at once using commas, or use @exact_name for precise matching.

That’s the power of airflow-dag-dependencies!

Building Your First Airflow 3 Plugin: A Technical Deep Dive

Now that we’ve seen the plugin in action, here is how to build it from scratch. This walkthrough highlights the key differences between Airflow 2 and Airflow 3 plugin development and shows how to create your own plugin.

Plugin Structure: Airflow 3 vs Airflow 2

One of the most significant changes in Airflow 3 is how plugins are structured and registered.

Airflow 2 Plugin Structure

from airflow.plugins_manager import AirflowPlugin
from flask import Blueprint

from flask_admin import BaseView

class MyAirflow2Plugin(AirflowPlugin):

    name = “my_plugin”

    flask_blueprints = [my_blueprint]  # Flask blueprints

    admin_views = [MyAdminView]        # Flask-Admin views

    menu_links = [my_menu_link]        # Menu items

Airflow 3 Plugin Structure

from airflow.plugins_manager import AirflowPlugin
class MyAirflow3Plugin(AirflowPlugin):

    name = “my_plugin”

    fastapi_apps = [my_fastapi_app]     # FastAPI applications

    external_views = [my_external_view] # External view definitions

Key Changes in Airflow 3

• FastAPI replaces Flask for web applications, improving performance, async support, and automatic API documentation.
• External views replace Flask-Admin views, giving more flexible UI integration.
• Menu items are now part of external views rather than separate menu link definitions.

Anatomy of the DAG Dependencies Plugin

Package Structure

airflow_dag_dependencies/

├── __init__.py

└── plugins/

├── __init__.py

└── dags_dependencies_plugin.py

FastAPI Application

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI()

@app.get(”/”, response_class=HTMLResponse)

async def dag_dependencies_graph():

    # Return HTML with embedded D3.js visualization

    return html_content

@app.get(”/api/dag-dependencies”)

async def get_dag_dependencies():

    # Return JSON data for DAG relationships

    return {”nodes”: nodes, “links”: links}

Plugin Registration

FastAPI app configuration

fastapi_app_with_metadata = {
    “app”: app,
    “url_prefix”: “/dags_dependencies”,
    “name”: “DAG Dependencies App”
}

External view for menu integration

dag_dependencies_external_view = {
    “name”: “DAG Dependencies”,
    “href”: “/dags_dependencies/”,
    “destination”: “nav”,
    “category”: “browse”,
    “url_route”: “dag_dependencies”
}

Plugin class

class DagDependenciesPlugin(AirflowPlugin):
    name = “dags_dependencies_plugin”
    fastapi_apps = [fastapi_app_with_metadata]
    external_views = [dag_dependencies_external_view]

Data Access: Using DagBag

The plugin accesses Airflow’s DAG metadata through DagBag.

from airflow.models import DagBag
async def get_dag_dependencies():

    dagbag = DagBag(include_examples=False)

    nodes = []

    for dag_id, dag in dagbag.dags.items():

        nodes.append({

            “id”: dag_id,

            “type”: determine_dag_type(dag),

            “schedule”: get_schedule_info(dag)

        })

    links = analyze_dag_relationships(dagbag)

    return {”nodes”: nodes, “links”: links}

Frontend Options: Embedded or External

Airflow 3 supports multiple frontend approaches:

• Embedded HTML/JS (current plugin)
• Standalone React components (future tutorial)
• Serving static assets for more advanced UIs

Packaging for Distribution

Example pyproject.toml configuration:

[tool.poetry]
name = “airflow-dag-dependencies”

version = “0.1.1a1”

description = “An Apache Airflow plugin that provides interactive visualization of DAG dependencies”

authors = [”Ponder”]

[tool.poetry.dependencies]

python = “>=3.12, <3.14”

apache-airflow = “^3.1.0”

[tool.poetry.plugins.”airflow.plugins”]

dag-dependencies-plugin = “airflow_dag_dependencies.plugins.dags_dependencies_plugin:DagDependenciesPlugin”

What’s Next?

The next tutorial will cover:

• Building React components that integrate with Airflow’s UI
• Handling authentication and permissions

Airflow 3’s FastAPI foundation opens the door to more capable, modern, and faster plugin development. More powerful integrations are now possible than ever.

Conclusion

Airflow 3 marks a real shift in how developers extend the platform. FastAPI, external views, and the new plugin model make it possible to build tools that are faster, more flexible, and much easier to maintain. Rebuilding the DAG Dependencies feature gave us a chance to explore these new capabilities in depth and turn a missing Airflow 2 feature into something more capable than before.

The airflow-dag-dependencies plugin restores a critical part of the Airflow UI, adds filtering and search features that help teams understand complex pipelines, and acts as a practical example of how to build real plugins in Airflow 3. If you are migrating from Airflow 2 or planning to develop your own extensions, this approach will give you a strong foundation.

There is still more room to grow. React-based UI components, richer interactions with Airflow’s new frontend APIs, and improved permission handling are all on the way. Airflow 3 opened the door to a modern plugin ecosystem, and this is only the start.

Introduction

At Ponder, we first introduced Airflow MCP and its plugin to make working with DAGs and pipelines smoother. But along the way, we realized something: managing Superset dashboards and their metadata can be just as challenging—if not more so.

Take education as an example. Schools and educational institutions manage dozens—sometimes hundreds—of dashboards tracking everything from student performance and attendance to resource allocation and budget utilization. Multiple BI developers work constantly on these dashboards, making frequent updates and refinements so stakeholders can access the most current data to improve education and help pupils succeed. The result? A powerful ecosystem of insights—but also an overwhelming maze of metadata.

Questions like “Who changed the attendance dashboard last week?”, “Which table powers our graduation rate metrics?”, or “What’s the data source behind this enrollment forecast?” can quickly become a time sink. And it’s not just schools—any organization with a complex BI environment faces the same challenge.

That’s why we built Superset MCP—so you can explore and manage Superset metadata through natural language, directly from your Claude code. No need to wrestle with the UI, no need to dig through layers of metadata. Just type your question, and get clear answers in plain language.

You can check out the project on GitHub, pull the image from Docker Hub, or install it from PyPI.

Quickstart

1. Run Superset Locally

If you don’t already have a Superset deployment but still want to try out Superset MCP, simply clone our GitHub repository and start it with Docker Compose:

git clone https://github.com/ponderedw/superset-mcp
cd superset-mcp
docker compose up

Once it’s up, open your browser and visit:

http://localhost:8088/

Login credentials for the example environment:

• Username: superset_admin

• Password: superset
  
  

2. Using Superset MCP in Claude Desktop

1. Open Claude Desktop.

2. Navigate to: Settings → Developer → Edit Config.

3. Add the following entry to your MCP servers:

{
  “mcpServers”: {
    “superset”: {
      “args”: [
        “run”,
        “-i”,
        “--rm”,
        “-e”,
        “SUPERSET_API_URL”,
        “-e”,
        “SUPERSET_USERNAME”,
        “-e”,
        “SUPERSET_PASSWORD”,
        “pondered/superset-mcp:latest”
      ],
      “command”: “docker”,
      “env”: {
        “SUPERSET_API_URL”: “http://host.docker.internal:8088”,
        “SUPERSET_USERNAME”: “superset_admin”,
        “SUPERSET_PASSWORD”: “superset”
      }
    }
  }
}

Note: The Superset URL and credentials above match our local example environment.

4. Restart Claude Code.

3. Using Superset MCP with LangChain

To integrate with LangChain:

1. Install our PyPI package:

pip install superset-mcp-server

📦 PyPI link

2. Add the following MCP server configuration to your tools:

from langchain_mcp_adapters.client import MultiServerMCPClient
import os

mcps = {
    “SupersetMCP”: {
        “command”: “python”,
        “args”: [”-m”, “superset_mcp_server.mcp_server”],
        “transport”: “stdio”,
        “env”: {
            k: v for k, v in {
                “SUPERSET_API_URL”: os.getenv(”SUPERSET_API_URL”),
                “SUPERSET_USERNAME”: os.getenv(”SUPERSET_USERNAME”),
                “SUPERSET_PASSWORD”: os.getenv(”SUPERSET_PASSWORD”),
            }.items() if v is not None
        }
    }
}

client = MultiServerMCPClient(mcps)
mcp_tools = await client.get_tools()

Now you’re ready to simply ask questions about your Superset instance in natural language—and let MCP handle the rest!

Talking to Superset MCP

Let’s connect to our example Superset deployment and build a simple dashboard using the data automatically ingested into our Postgres database.

First, we’ll add a database connection — specify “postgres” as both the host and the database name, and use “superset” for the login and password.

Once connected, we can create a few simple charts from the database tables and organize them into an E-commerce Performance Dashboard.

Finally, don’t forget to publish your dashboard so it’s ready to use!

You can explore all the available tools in Claude Desktop — and more will be coming soon.

Now, let’s start asking some interesting questions:

• How many dashboards and charts do we have?

  

  

  Great! We get the exact number of dashboards and charts in our environment.

• Who are the most active users?

  

  

  
  Perfect — not only do we see the top active user, but we also get an Activity Timeline, which makes it much easier to understand what’s happening in our cluster.

• What is the source table for Customer-Product Network?

  

  Excellent — we now have the precise database object name.

• When was the Revenue by Country chart last changed, and by whom?

  

  Spot on — we’ve got the correct answer here as well.

Pretty cool, right? Keep exploring and help us uncover even more capabilities of Superset MCP!

Conclusion — Managing Metadata the Smart Way

Superset MCP turns the often painful task of navigating dashboard metadata into a fast, intuitive conversation. Whether you’re an educational administrator tracking changes to student performance dashboards, a data analyst identifying which tables feed into critical reports, or a team lead trying to understand who’s been most active in your BI environment—you can now skip the clicks, menus, and manual searches. Just ask in plain language, and get precise, actionable answers.

In education settings where BI teams constantly update dashboards to provide stakeholders with real-time insights for improving outcomes, Superset MCP becomes invaluable. But the benefits extend far beyond schools—any business with complex dashboard environments, whether in healthcare, finance, retail, or manufacturing, can leverage this natural language interface to cut through the metadata complexity.

Because it works entirely through natural language, you can easily delegate these tasks to analysts, project managers, principals, department heads, or any non-technical users—empowering more people in your team to get the information they need without relying on developers or admins.

We’re excited to see how the community will use Superset MCP—and we’d love your feedback. Try it out, share your ideas, and help us shape the next set of capabilities.

🔗 Resources:

• GitHub Repository

• Docker Hub Image

• PyPI Package

  Thanks for reading Data & AI Engineering @ Ponder! Subscribe for free to receive new posts and support our work.

Introduction

In education data teams, our analytics models often grow organically alongside organizational needs. As school networks expand their data infrastructure—adding new student information systems, assessment platforms, and operational tools—their dbt projects can become increasingly complex. What starts as a clean set of staging models quickly evolves into an intricate web of dependencies that becomes difficult to navigate and maintain.

In previous blog posts—dbt-to-English and English-to-SQL—we demonstrated how LLMs can assist with understanding and generating dbt metadata. However, these tools fall short when it comes to answering complex questions about dbt’s project-wide structure and dependencies. For instance, what is the most widely reused source across models (recursively)? Or, how many seed datasets are used by models materialized as tables?

We’ve seen this challenge firsthand. At one school network, their analytics catalog grew from a handful of models serving basic enrollment reporting to hundreds of interconnected models supporting everything from attendance analytics to intervention tracking to college persistence dashboards. This organic growth led to inadvertent circular dependencies that made it increasingly difficult to refactor models or optimize their orchestration. The team needed a way to visualize and understand these dependencies before they could untangle them.

That’s when we had a realization: dbt’s structure is naturally graph-shaped, making it an excellent fit for graph databases. This inspired us to launch a new project—dbt-kg—which shows how to upload dbt metadata (from manifest.json and catalog.json) into a graph database and leverage it in an AI-powered chat. While our demo uses FalkorDB, the project is also compatible with Neo4j—just update the configuration settings accordingly.

Quickstart

To get started, clone our GitHub repository:
https://github.com/ponderedw/dbt-kg

Next, copy the environment template and configure it:

cp template.env .env

In your .env file, set the LLM_MODEL_ID according to the provider you want to use:

LLM_MODEL_ID=’bedrock:...’
LLM_MODEL_ID=’anthropic:...’
LLM_MODEL_ID=’openai:...’

The prefix (bedrock, anthropic, or openai) determines which LLM provider will be used.
Be sure to include the appropriate credentials as well:

• AWS_SECRET_ACCESS_KEY for Bedrock

• ANTHROPIC_API_KEY for Anthropic

• OPENAI_API_KEY for OpenAI

Then, choose your graph database by setting the GRAPH_DB variable. This can be either falkordb or neo4j. For this demo, we’ll be using FalkorDB.

If your database requires authentication, provide values for GRAPH_USER and GRAPH_PASSWORD.

⚠️ Our default FalkorDB setup does not use authentication, so these can remain commented out. If you’re using our default Neo4j setup, uncomment those variables and set the default credentials accordingly.

Now, run the setup:

just all

Once everything is set up, open your browser and go to:

 http://localhost:8501/

Navigate to the “Load DBT Manifest” page from the sidebar. You’ll be prompted to upload your manifest.json and catalog.json files. If you have your own, upload them; otherwise, use the sample files located in DbtExampleProject/target within the repo.

Upon successful upload, you’ll see a confirmation message.

Finally, open your graph database UI at:

 http://localhost:3000/

Run the following query to view your dbt Knowledge Graph:

MATCH (a)-[b]-(c)
RETURN a, b, c

Congratulations! Your dbt Knowledge Graph is now up and running. 🎉

Interacting with Your dbt Graph via Chat

Let’s return to

 http://localhost:8501/

—this time, we’ll stay on the Main page and begin chatting with our dbt Knowledge Graph.

We’ll start by asking a high-level question about source table usage:

Prompt:
“Can you find the source tables that are used by the most models, including indirect dependencies through the entire dependency chain?”

The system correctly identifies both source tables from our example project and ranks them by usage in descending order. Perfect!

Next, let’s try something more challenging:

Prompt:
“Please identify the model with the highest number of dependent models, including transitive dependencies. This model must be materialized as a table. Return all the information you have about this table.”

This is tricky—because in our project, there are no models materialized as table.

Impressively, the LLM handles this gracefully by identifying the closest match: a view instead, and clearly stating the limitation. That’s a solid response.

Let’s follow up with another prompt to dig deeper:

Prompt:
“Can you also list indirect dependent models for stg_orders?”

Now we’re talking. The model returns the full dependency chain. Mission accomplished.

From One Monolithic Job to Smart Job Grouping

In many real-world workflows, especially when integrating with orchestrators like Airflow, we don’t want to run the entire dbt graph as a single job. Instead, we aim to break it into smaller, manageable jobs (DAGs) that can trigger downstream non-dbt processes as early as possible. But there’s a catch: circular dependencies often make this kind of separation complex and error-prone.

Let’s ask the LLM to help with that:

Prompt:
“Can you analyze our dbt dependency graph and create a job separation strategy? I need to group models, seeds, and snapshots into separate jobs that eliminate circular dependencies while defining clear inter-job dependencies.”

And it delivers! The model intelligently separates our graph into four distinct jobs and describes the cross-job dependencies clearly.

Summary

By leveraging the graph-shaped nature of dbt projects, the dbt-kg initiative demonstrates how graph databases like FalkorDB and Neo4j can supercharge the way we interact with dbt metadata. With an intuitive setup, seamless integration of LLMs, and a simple web-based interface, users can upload their dbt artifacts and start asking complex, project-wide questions in natural language.

For education data teams in particular, this approach addresses a common challenge: as schools and networks scale their analytics capabilities, their dbt projects naturally grow more complex. The AI-powered chat not only handles nuanced prompts about source usage, dependencies, and materializations but also assists in advanced tasks like intelligent job grouping to optimize orchestration workflows—exactly the kind of refactoring that the school network mentioned earlier needed to break apart their circular dependencies and create a more maintainable structure.

Whether you’re supporting a single school, a district, or a multi-state network, transforming static metadata into a dynamic, queryable knowledge graph makes data engineering more intelligent, conversational, and manageable. This is especially valuable in education, where data teams often wear multiple hats and need efficient ways to understand and maintain increasingly sophisticated analytics infrastructure.

Introduction

After recently launching our new library, airflow-chat—a plugin that enables seamless LLM chat interactions within your Airflow instance via our Airflow MCP—we quickly turned our attention to an important question: how do we effectively observe and log these conversations?

Imagine a scenario where a user triggers an unintended action due to safe mode being disabled, leading to a production error. How do you trace the issue? Who initiated the command? What was the exact prompt that caused the failure? Or more generally—how do you evaluate LLM interactions over time?

To address these questions, we turned to Langfuse, an elegant open-source tool purpose-built for developing, monitoring, evaluating, and debugging LLM applications. In this blog post, we’ll walk you through integrating Langfuse into your LangChain workflow—and specifically, how we’ve already done this within the airflow-chat plugin. You can follow along using our repo: https://github.com/ponderedw/airflow-mcp-plugins.

Quickstart

Let’s walk through a quick setup to see how Langfuse integrates with the airflow-chat plugin.

1. Clone the Repository and Set Up Locally

Start by cloning our GitHub repository:

git clone https://github.com/ponderedw/airflow-mcp-plugins

Then follow the setup steps outlined in our airflow-chat blog post to get the plugin running in your local environment.

Note: Running both Airflow and Langfuse locally can be resource-intensive. If your machine starts lagging, consider connecting to pre-deployed instances of Airflow and Langfuse for a smoother experience.

To launch both locally, run:

just airflow_langfuse

Once the containers are up, head to

 http://localhost:3000/

and sign up for a new Langfuse account.

2. Create an Organization, Project, and API Key

After signing up:

• Create a new Organization

  

• Create a Project under that organization

  

• Generate an API key

  

Copy the credentials and add them to your Airflow environment variables. If you’re using our GitHub setup, you can simply paste them into the airflow-plugin.env file:

LANGFUSE_SECRET_KEY=sk-lf-053f1073-789b-4705-8daa-23b825829d3d
LANGFUSE_PUBLIC_KEY=pk-lf-25851900-8b27-45f7-b7a0-617321e988cf
LANGFUSE_HOST=http://host.docker.internal:3000

3. Restart and Explore

Restart your environment:

just airflow_langfuse

Now, navigate to the Airflow UI, interact with the airflow-chat plugin, and ask a few test questions.

Then head over to the Langfuse dashboard. Open the "Tracing → Traces" tab:

Voilà—your entire chat session is logged, complete with trace-level details.

Just like that, observability is in place!

Going Further with Langfuse

Langfuse offers much more than just trace logging—it provides powerful tools to evaluate and improve your LLM application.

Users
Navigate to the Users tab to see a list of application users. You’ll notice a user named "airflow"—this represents interactions made through the airflow-chat plugin. Click on it to view all associated sessions. This is a great way to monitor how your team is using the plugin over time.

Cost Tracking
Langfuse makes it easy to track costs. In the Traces tab, you’ll see session-level costs, while the Home tab gives you an overview of model-level expenses. For deeper insights, head over to the Dashboards tab and explore the built-in Langfuse Cost Dashboard, or create a custom dashboard with your own cost graphs.

If you suspect that token prices are incorrect, go to Settings → Models, and either “Add model definition” or “Clone” an existing one to customize pricing.

Evaluations
Langfuse supports two types of evaluations: Human Annotation and LLM-as-a-Judge.

• Human Annotation
  Go to the Scores tab and create a new score. Next, create an Annotation Queue, and then add traces to the queue via the Traces tab. You can now manually evaluate each session. While this method offers fine-grained control, it can be time-consuming.

  

  

  

• LLM-as-a-Judge
  To automate evaluations, go to Settings → LLM Connection and add a model connection.
  Then create a custom evaluator by navigating to Create Custom Evaluator → Create New Evaluator, and set up a Default Evaluation Model.

  

  Choose a metric—such as Relevance—and specify whether you want to evaluate existing traces or only new ones.

  

  

  For inputs and outputs, map query to input and generation to output.

  

  Now, go back to Airflow and ask a question. Check the Evaluation Logs and you’ll see a score and comment like:

“The generation directly and comprehensively answers the user's query by listing all the DAGs, providing their names, and offering additional context about their purpose and scheduling. The response is focused, informative, and adds significant value to the user's understanding of the available DAGs in the Airflow environment.”

This demonstrates how Langfuse helps you evaluate, measure, and continuously improve your LLM-powered apps.

Setting Up Langfuse for Your LangChain Application

Integrating Langfuse into your LangChain app is straightforward. The configuration is handled in the airflow_chat/plugins/app/server/llm.py file, specifically within the get_user_chat_config function:

def get_user_chat_config(session_id: str, username: str = None) -> dict:
    chat_config = {
        'configurable': {'thread_id': session_id},
        'recursion_limit': 100
    }

    if os.environ.get('LANGFUSE_HOST'):
        langfuse_handler = CallbackHandler(
            user_id=username if username else session_id,
            session_id=str(session_id),
            public_key=os.environ.get('LANGFUSE_PUBLIC_KEY'),
            secret_key=os.environ.get('LANGFUSE_SECRET_KEY'),
            host=os.environ.get('LANGFUSE_HOST')
        )
        chat_config['callbacks'] = [langfuse_handler]

    return chat_config

This function builds the chat configuration and conditionally attaches the Langfuse callback handler if the environment variables are present.

The resulting configuration (chat_config) is then passed to the astream_events method like this:

async for event in self._agent.astream_events(
    {"messages": [HumanMessage(content=message)]},
    config=chat_session,
    version='v2',
)

The _agent object itself is initialized using the create_react_agent utility from LangGraph:

from langgraph.prebuilt import create_react_agent

self._agent = create_react_agent(
    self._llm,
    tools,
    checkpointer=checkpointer,
    prompt=PROMPT_MESSAGE
)

All of this logic can be found in the airflow_chat/plugins/app/server/llm.py file. This setup enables seamless event streaming and tracking via Langfuse within your LangChain-based application.

Summary

By combining the airflow-chat plugin with Langfuse, we’ve created a robust observability stack for LLM-powered workflows within Airflow. From real-time trace logging and cost tracking to advanced evaluation methods like LLM-as-a-judge, Langfuse brings clarity and control to your AI interactions. Whether you're debugging unexpected prompts, monitoring usage, or analyzing performance, Langfuse helps you iterate with confidence. For a deeper dive into all the features Langfuse offers, visit https://langfuse.com.

Introduction

Not long ago, we introduced our MCP for Airflow—designed to make operating DAGs and workflows as intuitive as chatting with a colleague. But there was a catch: it depended on Claude Desktop, which some organizations are unable or unwilling to adopt due to concerns around sharing data with Anthropic—or because of other operational, compliance, or policy-related reasons.

That got us thinking: what if we brought the same conversational interface directly into the Airflow UI?

That’s how the airflow-chat plugin came to life.

Under the hood, the plugin still leverages our existing MCP. You can run it fully embedded inside the Airflow webserver, or flexibly configure it in one of two other ways:

• As a plugin running inside the Airflow webserver (simplest setup)

• Using the plugin alongside a separate MCP service

• Or in a fully decoupled three-process architecture: Airflow webserver plugin → FastAPI server → MCP service

You also have full control over the LLM backend. MCP supports multiple providers, so you can choose to run it with Amazon Bedrock, Anthropic, or OpenAI—whichever best suits your infrastructure and policies.

This makes it easier than ever to integrate natural language interfaces into your pipelines—entirely within your stack and control.

Explore the project here:

• PyPI: airflow-chat

• GitHub: airflow-mcp-plugins

Quickstart

If you don’t have your own Airflow cluster, feel free to use the one provided in our GitHub repository.

Let’s start with the simplest setup. To integrate our plugin, add the following to your Airflow environment:

airflow-chat==0.1.0a8

Note: Version 0.1.0a8 is the latest at the time of writing. Check our PyPI repository for updates before installing.

🛠 Required Environment Variables

Add the following environment variables to your cluster configuration:

LLM_MODEL_ID=anthropic:claude-3-5-haiku-20241022
AIRFLOW_ASSISTENT_AI_CONN=airflow:airflow@http://localhost:8080
ANTHROPIC_API_KEY=your_anthropic_key

AIRFLOW_ASSISTENT_AI_CONN Explained

This connection string is used by the AI Assistant plugin to communicate with your Airflow instance. It follows this structure:

AIRFLOW_ASSISTENT_AI_CONN=<username>:<password>@http://localhost:8080

• Replace <username> and <password> with any valid Airflow credentials you want the assistant to use.

• Do not change localhost:8080, as this address is required for internal routing within the containerized environment where the plugin operates.

🔁 Model Flexibility:
You can switch LLM_MODEL_ID to use models from Bedrock or OpenAI. Simply change the prefix accordingly:

• For Bedrock: bedrock:...

• For OpenAI: openai:...

Be sure to set the correct corresponding API credentials:

• Anthropic: ANTHROPIC_API_KEY

• Bedrock: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

• OpenAI: OPENAI_API_KEY

⚙️ Additional Configuration

You can also configure optional Airflow MCP environment variables such as:

• AIRFLOW_INSIGHTS_MODE

• POST_MODE

We explain these options in detail in this blog post.

🧪 Running with Our Default Environment

If you’re using our pre-configured environment:

cp airflow-plugin-test.env airflow-plugin.env

Then open airflow-plugin.env, fill in all required values, and run:

just airflow

🧭 Accessing the Assistant

Once your cluster is running, open

 http://localhost:8080

, then navigate to:

Tools -> AI Chat Assistant

You’ll now see the Airflow Chat Assistant in action! Try asking:

"What Airflow DAGs do we have, and which ones are paused?"

🎉 Everything should work beautifully out of the box!

🔐 Access Control

Access to the AI plugin is restricted to:

• Admins, or

• Any user assigned the AIRFLOW_AI role.

Other Deployment Strategies

Let’s take a look at how our simple deployment works—and then walk through how to modify it depending on your needs.

🧩 The Basic Setup

Our lightweight Airflow Chat plugin runs both the Langchain modules and Airflow MCP locally inside the Airflow Webserver. This setup works out-of-the-box with minimal configuration.

But what if you don’t want to manage environment variables inside the Airflow instance under the same user? No problem—you can split the deployment and run MCP externally.

🚀 Option 1: Externalize the MCP Server

To run MCP as a separate component:

1. Remove the embedded MCP server from your Airflow Webserver setup.

Add these environment variables to your Airflow instance (you can use airflow-plugin.env in our repo):

TRANSPORT_TYPE=sse
mcp_host=host.docker.internal:8001
MCP_TOKEN=test_token

📌 Note: mcp_host must be updated if you're deploying the MCP server on your own infrastructure.

Remove the Airflow connection variable: AIRFLOW_ASSISTENT_AI_CONN

2. If you're using our repository, just run:

cp airflow-mcp-test.env airflow-mcp.env

Then, customize the following variables as needed (no changes required if using the default setup in our repo):

LLM_MODEL_ID=antropic:claude-3-5-haiku-20241022
AIRFLOW_ASSISTENT_AI_CONN=airflow:airflow@http://airflow-webserver:8080
TOKEN=test_token
TRANSPORT_TYPE=sse

✅ Ensure that TOKEN and MCP_TOKEN have the same value to establish a secure connection between Airflow and the remote MCP server.

To deploy:

• Run just prod if you're using our repo, or

• Use the Docker image: hipposysai/airflow-mcp:latest

• Refer to docker-compose.prod.yml for an example configuration.

🖥️ Option 2: Use the Plugin as a UI Only (Separate Langchain Logic)

If you prefer to fully decouple Langchain logic from the Airflow Webserver (and just use the plugin as a UI layer), update your airflow-plugin.env as follows:

INTERNAL_AI_ASSISTANT_SERVER=false
FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'
BACKEND_URL=http://chat_fastapi:8080

Then, for the FastAPI/Langchain service (using the hipposysai/airflow-mcp-fastapi image), define:

LLM_MODEL_ID=antropic:claude-3-5-haiku-20241022
SECRET_KEY='ThisIsATempSecretForLocalEnvs.ReplaceInProd.'
FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'
TRANSPORT_TYPE=sse
MCP_TOKEN=test_token
mcp_host=mcp_sse_server:8000
POSTGRES_USER='airflow'
POSTGRES_PASSWORD='airflow'
POSTGRES_DB='postgres'

To apply this config:

cp langchain-fastapi-test.env langchain-fastapi.env
just prod

⚖️ Option 3: Separate Backend but Share MCP

Want to offload backend logic from your Airflow Webserver without introducing a separate MCP service? You can:

1. Remove TRANSPORT_TYPE=sse from langchain-fastapi.env (or whichever file defines your backend).

2. Copy all environment variables from your MCP service (airflow-mcp.env) to your backend environment.

This option gives you better performance isolation without significantly increasing deployment complexity.

Summary

The airflow-chat plugin brings a conversational AI interface directly into the Airflow UI, enabling users to inspect DAGs, query task states, and operate pipelines using natural language. Built on top of the existing MCP architecture, it offers flexible deployment options—from running entirely inside the Airflow Webserver to more decoupled setups involving external MCP and FastAPI services. The plugin supports multiple LLM backends (Anthropic, Bedrock, OpenAI), giving teams full control over model providers and data handling. With minimal configuration, customizable environment files, and built-in access controls, airflow-chat makes AI-powered orchestration both accessible and secure—right where data pipelines live.

Introduction

A few weeks ago, we introduced Airflow MCP from Hipposys. Since then, we've fine-tuned its capabilities to better align with real-world user needs. In this post, we’ll walk you through three major updates: Safe Mode for secure operation, enhanced controls to manage DAG state, and a new way to predict upcoming DAG runs using our Airflow Schedule Insights plugin.

Run with Confidence: Safe Mode

We realized that many users—especially those working with production Airflow instances—need to be certain that MCP won’t inadvertently alter DAG states. Even if your Airflow user has permission to run, pause, or delete DAGs, you may want to use MCP strictly for read-only access. That’s why Safe Mode is now the default behavior.

In Safe Mode, only informational (GET) functions are available, ensuring you can inspect your Airflow environment without risk. To enable write capabilities (e.g. triggering or modifying DAGs), you can explicitly opt in by setting the POST_MODE environment variable.

To do this, add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "airflow_mcp": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "airflow_api_url",
        "-e",
        "airflow_username",
        "-e",
        "airflow_password",
        "-e",
        "POST_MODE",
        "hipposysai/airflow-mcp:latest"
      ],
      "command": "docker",
      "env": {
        "airflow_api_url": "http://host.docker.internal:8088/api/v1",
        "airflow_username": "airflow",
        "airflow_password": "airflow",
        "POST_MODE": "true"
      }
    }
  }
}

Once enabled, you’ll have access to all functions—including state-changing operations.

Until then, rest assured: by default, MCP operates in a safe, read-only mode, so you can explore and monitor without risking production disruptions or unscheduled runs.

Pause DAGs Before the Storm

We've all been there: an unexpected (or even expected) outage hits, and you need to urgently pause all Airflow DAGs. So you go through them one by one… but wait—some DAGs were already paused. Should you manually note them down just to restore their state afterward? That’s tedious—and error-prone.

Not anymore.

Thanks to the new change_dags_pause_status method, this process is now streamlined and reliable.

Note: The Airflow cluster I'm working with is deployed from our GitHub repo: hipposys-ltd/airflow-mcp.

Right now, here’s our current state:

Let’s use MCP to prepare for the outage.
Start with the following prompt:

"Hi, please pause all Airflow DAGs, but before doing so, list the DAGs that are currently paused. We’ll use that list to remember which ones need to be unpaused tomorrow."

As you can see, all DAGs have been successfully paused.

Now comes the tricky part: restoring normal operations without unpausing DAGs that were paused before the outage.

Run:

"Great. Now please unpause the DAGs, but make sure to leave the ones that were already paused as they are."

And just like that—done.
It only took two prompts to handle a full-cluster pause and restore cycle, cleanly and safely.

When Will It Run?

Let’s start with something simple: finding out when load_ticket_sales will run next.

To do that, we’ll need to update claude_desktop_config.json once again by enabling Airflow Insights Mode:

{
  "mcpServers": {
    "airflow_mcp": {
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "airflow_api_url",
        "-e",
        "airflow_username",
        "-e",
        "airflow_password",
        "-e",
        "POST_MODE",
        "-e",
        "AIRFLOW_INSIGHTS_MODE",
        "hipposysai/airflow-mcp:latest"
      ],
      "command": "docker",
      "env": {
        "airflow_api_url": "http://host.docker.internal:8088/api/v1",
        "airflow_username": "airflow",
        "airflow_password": "airflow",
        "POST_MODE": "true",
        "AIRFLOW_INSIGHTS_MODE": "true"
      }
    }
  }
}

⚠️ Note: Make sure the Airflow Schedule Insights plugin is installed in your Airflow instance for this to work.

Now, restart Claude Desktop and ask:

“When is the next run for load_ticket_sales (in EST)?”

After a short wait, you’ll get a detailed answer including the start time, end time, and expected duration—and it’s accurate. But that’s easy—load_ticket_sales is a scheduled DAG.

Let’s go a step further and try something more dynamic.

Event-Driven DAG Prediction

Now let’s look at an event-driven DAG with no fixed schedule: transform_customer_sentiment.

This DAG is triggered when:

• either transform_sales_aggregator and load_customer_feedback complete,

• or load_ticket_sales finishes on its own.
  

  

According to Claude, the next trigger will come from load_ticket_sales. Let’s wait and see…

And yes—it was triggered just one second later than predicted and finished two seconds later than expected. Even better, Claude correctly identified that load_ticket_sales was the trigger.

Querying Multiple DAGs at Once

Now let’s try asking about several DAGs in one go:

“When are the next runs for load_movie_inventory, load_box_office_reports, and transform_loyalty_program_analysis (in UTC)?”

Claude replies—none of them will run, and that’s correct!

With just a few configuration changes, you can now monitor and predict future DAG runs directly from Claude Desktop—scheduled or event-driven. No guesswork, no surprises.

Summary: Next-Level DAG Observability and Control

With the latest updates to Airflow MCP, Claude Desktop evolves from a smart interface into a powerful control and observability tool for Airflow users. You can now operate in Safe Mode for peace of mind in production, pause and restore DAG states safely during outages, and even predict when DAGs—scheduled or event-driven—will run next, thanks to the new Airflow Schedule Insights integration. Whether you're running a complex pipeline or preparing for maintenance, these upgrades ensure you're always one step ahead—with just a couple of prompts.

When building a RAG system, developers often turn to vector databases—like ChromaDB or Milvus—for storing embeddings. While popular, these solutions come with notable drawbacks:

• Proprietary options can be expensive, and even open-source databases typically require maintaining a full cluster.

• Managing infrastructure—permissions, monitoring, uptime—adds significant operational complexity.

• Scaling vector databases for large workloads isn't always straightforward or cost-effective.

• Most solutions lack native integration with AWS services like Amazon Bedrock Knowledge Bases.

To address these limitations, AWS recently introduced S3 Vector Buckets—a lightweight, scalable, and fully managed way to store and retrieve vector embeddings directly within Amazon S3. While AWS provides excellent documentation on using this with their Agent ecosystem (official guide), we created a demo project showing how to integrate S3 Vector Buckets with LangChain.

In this post, we'll walk you through the setup, using real-world data from the NYC City Planning website, and demonstrate how to extract useful insights with natural language queries.

Setting Up the Vector Bucket

Start by navigating to AWS S3, selecting Vector Buckets from the left-hand menu, and creating a new bucket.

Great — now we have an S3 vector bucket ready to store embeddings. But how do we search through those vectors? For that, we need to create a vector index.

When configuring the index, set the dimensions value to match the number of values returned by your embedding model. Since we’re using the amazon.titan-embed-text-v2:0 embedding model from Amazon Bedrock, which produces vectors of 1024 dimensions, set the dimension accordingly.

Next, you’ll need to choose a similarity measure — either Euclidean Distance or Cosine Similarity. We’ll use Cosine Similarity, as it aligns better with how Titan embeddings are designed: they capture meaning in the direction of the vector rather than its magnitude, which is exactly what Cosine Similarity measures.

Perfect! With both the vector bucket and index set up, you're now ready to run your repository and start interacting with your vector store.

Using the Vector Bucket

To get started, clone our demo repository locally:

git clone https://github.com/hipposys-ltd/s3-vector-bucket
cd s3-vector-bucket
cp template.env .env

This will generate a .env file where you’ll configure all necessary environment variables:

DEPLOY_ENV='local'
# Postgres

POSTGRES_USER='postgres'

POSTGRES_PASSWORD='postgres'

POSTGRES_DB='postgres'

# LLM Model

LLM_MODEL_ID='bedrock:...' # or use 'anthropic:...' or 'openai:...'

# Secrets (replace in production)

SECRET_KEY='ThisIsATempSecretForLocalEnvs.ReplaceInProd.'

FAST_API_ACCESS_SECRET_TOKEN='ThisIsATempAccessTokenForLocalEnvs.ReplaceInProd'

# AWS Credentials

AWS_ACCESS_KEY_ID='...'

AWS_SECRET_ACCESS_KEY='...'

AWS_DEFAULT_REGION='...'

# Optional Keys

# ANTHROPIC_API_KEY='...'

# OPENAI_API_KEY='<YOUR OPENAI API KEY>'

# Vector Store

bucket='test-vector-bucket'

index='test-vector-index'

# Optional: Custom embedding model

# embedding_model='...'

To use Bedrock for both embedding and LLM, set LLM_MODEL_ID to a string starting with bedrock: and ensure your AWS credentials are configured.
If you prefer using models like Claude (Anthropic) or OpenAI, set the respective API key and adjust LLM_MODEL_ID accordingly (e.g., anthropic:... or openai:...).

Once configured, run:

just all

Then open your browser and navigate to

http://localhost:8501

You’ll find three tabs:

• Main – a chat interface to interact with your vector store

• Load File – to upload PDF documents and populate your vector index

• See Vectors – to explore and manage uploaded embeddings

Initially, the vector store will be empty, so let’s add some data.
Download the prepared NYC CPC dataset from:
https://hipposys-public-assets.s3.us-east-1.amazonaws.com/newsletter-assets/NYC+Planning.zip

Extract the PDF files into the data/ folder. Then, go to the Load File tab and upload them.

After processing is complete, check the See Vectors tab—you’ll see the loaded data! You can even manage the embeddings: delete individual rows, entire documents, or wipe all vectors.

Now, switch to the Main tab and try a query like:

What are the changing conditions of the Special Hudson Yards District?

You should receive a highly relevant response—your setup is working!

How It Works

The core logic of our vector integration lives in app/vector/__init__.py.

Connecting to the S3 Vector Store

We use the boto3 SDK to interact with AWS’s new S3 vector store:

self.s3vectors = boto3.client('s3vectors', region_name='us-east-1')

This client handles all interactions—storing, searching, listing, and deleting vectors in the configured S3 bucket.

Searching the Vector Bucket

The search method performs the following steps:

1. Embeds the input text using an embedding model (e.g., Amazon Titan).

2. Sends the resulting vector to the S3 bucket for similarity search.

3. Returns the top matches with metadata and distance.

def search(self, input_text):
    request = json.dumps({"inputText": input_text})
    response = self.bedrock.invoke_model(modelId=self.embedding_model, body=request)
    embedding = json.loads(response["body"].read())["embedding"]

    query = self.s3vectors.query_vectors(
        vectorBucketName=self.bucket,
        indexName=self.index,
        queryVector={"float32": embedding},
        topK=3,
        returnDistance=True,
        returnMetadata=True
    )
    return query["vectors"]

Listing and Deleting Vectors

You can also view or clean up your vector store using the following methods:

def list_vectors(self):
    query = self.s3vectors.list_vectors(
        vectorBucketName=self.bucket,
        indexName=self.index,
        returnMetadata=True
    )
    return [{**vector['metadata'], 'key': vector['key']} for vector in query['vectors']]

def delete_vectors(self, vector_keys: list):
    self.s3vectors.delete_vectors(
        vectorBucketName=self.bucket,
        indexName=self.index,
        keys=vector_keys
    )

Uploading Data to the Vector Store

The upload_file function (also in the same module) handles:

• Loading text from files

• Generating embeddings

• Storing them into the S3 vector bucket

This process is triggered when you upload documents via the Load File tab in the UI.

LangChain Integration

To integrate this logic into your LangChain agent, we wrap the search method as a LangChain Tool inside app/server/chat.py:

retriever = VectorDB()
tool = Tool(
    name="Data_Retriever",
    func=retriever.search,
    description="Searches S3 vector bucket for similar embeddings and returns matching results",
)
tools = [tool]

This enables your LangChain agent to retrieve semantically relevant data from your S3 vector bucket in response to user queries.

Conclusion

AWS S3 Vector Buckets offer a simple and scalable alternative to traditional vector databases—no clusters, no heavy maintenance, and no additional services to run. Combined with LangChain, you can now build RAG pipelines that are lightweight, cost-effective, and easy to set up.

Our demo showed how to embed and store documents, query them semantically, and retrieve meaningful answers—all using S3 as your vector store. Whether you're working with Amazon Bedrock, Anthropic, or OpenAI models, this setup provides a flexible and developer-friendly foundation for building intelligent applications.

Introduction

“English-to-SQL” is a hot topic right now—and for good reason. It’s the dream of every data analyst, data engineer, and data-savvy team: imagine asking a question in plain English and having an LLM not only understand it, but also run the exact SQL query needed to get the answer from your database or data lake.

There are many ways to implement English-to-SQL, and each approach comes with different tradeoffs in complexity, accuracy, and contextual awareness. What we’re showcasing here is just one practical and powerful path—focused on giving the LLM rich, structured context using dbt and LangChain.

Our dbt-to-English project laid the foundation by giving LLMs access to your dbt metadata, real SQL object names, and custom prompts. Now, we’ve taken it further: by integrating LangChain’s QuerySQLDatabaseTool, your LLM-powered assistant can not only understand your data environment, but actually connect to your database and run live SQL queries based on natural language inputs.

If you haven’t read our previous blog post about dbt-to-English, we recommend starting there—it sets the stage for what’s next.

Let’s explore how this new feature works in practice.

Getting Started: Connecting to the Database and Generating Fictional Sample Data

Let’s begin with a simple example to understand how the system works.

1. Configure Your Environment
Update your private.env file by adding a new key from the private.env-template file, for example:

DATABASE_URI=...

You can either use your own database or connect to the provided local one:

DATABASE_URI=postgresql://postgres:postgres@postgres:5432/postgres

2. Launch the Environment
Run the dbt-to-English environment (GitHub repo) using Docker:

docker compose up --build

3. Run dbt to generate source tables and transformation models.

 cd DbtExampleProject
 dbt build

4. Enable Database Connection in UI
In the app interface, select the catalog and manifest files. Then proceed to the System Prompt section, where a new checkbox labeled "Connect to the Database?" appears below the prompt text. Check this box.

Now add the following instruction to your prompt:

8. Execute a query to retrieve only the first 5 rows (no more) from the table, using PostgreSQL dialect. You must use your retrieval tool for this step!
   After analyzing the structure of the results (including column names, data types, and format patterns), generate entirely fictional placeholder data that illustrates the table schema only.
   Do not include any actual database values, even if they appear to be examples.

5. Run the Script
Let’s use the stg_customers model for this test. The system connects to the database via the provided DATABASE_URI, runs a query to fetch the first 5 rows, analyzes the schema, and produces fake—but structurally accurate—data.

6. Verify Query Execution
Inspect the PostgreSQL logs to confirm that the query was executed. The logs show that the tool did connect and access the table, which confirms the integration is functioning correctly.

7. Compare with Actual Data
To verify the accuracy of the AI's schema understanding, run the same query manually:

SELECT * FROM dbt.stg_customers sc LIMIT 5;

As expected, the tool generated entirely fictional data that mirrors the structure of the actual records. This proves that the placeholder data is schema-faithful and safe for demonstration or documentation purposes. Data analysts, product managers, or other non-technical users can now explore data structures confidently, without any risk of exposing real production data.

Advanced Validation Use Case

Now, let’s explore a more advanced and practical use case—identifying outdated dbt descriptions.

1. Add a Description with Known Issues

We’ll begin by editing the schema.yml file to add a description for the order_count_per_customer model. Intentionally include two incorrect statements and one correct one. For example:

“The following table is always empty but consists of four columns. Its data is generated using a SQL GROUP BY clause.”

2. Refresh Metadata

Restart the docker compose process to ensure the catalog and manifest files are regenerated with the updated model description:

docker compose down

docker compose up --build

Then, reselect the updated catalog and manifest files in the interface.

3. Update the Prompt for Comparison

Modify the prompt to instruct the LLM to perform a structured validation:

Compare the actual behavior of the current model with its dbt description (documentation).
If needed, use the QuerySQLDatabaseTool with the PostgreSQL dialect to validate the implementation.
Return your findings in a table with two columns:

• Description Item – a specific claim or element from the dbt model's description

• Status – either Correct if the behavior matches the description, or Incorrect if it does not

The table must include only the items that can be explicitly verified as correct or incorrect.
Avoid listing any assumptions, inferred logic, or undocumented behaviors.

Make sure to check the "Connect to the Database" checkbox before running the model.

4. Run the LLM

Execute the process.

As observed, the output table is highly accurate. The LLM correctly interpreted the dbt description, connected to the database, and validated each point:

• It verified that the table is not empty.

• It confirmed the presence of four columns.

• It validated the use of a SQL GROUP BY clause.

This confirms that the LLM can reliably detect outdated or incorrect documentation by cross-referencing dbt metadata with actual model behavior—providing a valuable tool for maintaining accurate, trustworthy data documentation.

How It Works Under the Hood

Let’s break down how this functionality is implemented from a technical perspective.

The core logic resides in the following file within our repository:
fastapi/server/tr_dbt_to_english.py

To enable database interaction, we utilize the QuerySQLDatabaseTool from the langchain_community package.

This tool is initialized with a SQLDatabase object, which can be constructed using the SQLDatabase.from_uri() method.

In addition to setting up the database connection, we define the LLM Chat instance—this serves as the "brain" behind the operation.

Once the QuerySQLDatabaseTool is ready, we configure a ReAct agent (LangChain tutorial). This agent integrates the LLM and the tools it can use (such as querying the database).

Finally, we use self.react_agent.stream to stream the model’s response, allowing it to dynamically interact with the database and return results in real time.

Conclusion: From Metadata to Meaningful Queries

What began as a way to help LLMs understand dbt metadata has now evolved into a practical, end-to-end system for transforming natural language into real, executable SQL—accurately and safely. By combining the structure of dbt, the flexibility of LangChain, and the power of live database connectivity, we've created a workflow that not only interprets intent but validates it against real data sources. Whether you're verifying outdated documentation, previewing schema with fictional data, or simply trying to bridge the gap between humans and databases, this integration shows what's possible when metadata, modeling, and AI work together. And best of all—it’s reproducible, extensible, and ready for your data team to explore.

Subscribe now

Introduction

DBT is a powerful tool for transforming data and managing analytics engineering workflows. But let’s face it—while it’s great for developers, it can be tough for analysts, product managers, or even other engineers to understand exactly what’s happening inside a specific model. The SQL can get dense, the logic isn't always obvious, and while DBT Docs and DBT Cloud’s Explore are helpful, they fall short when it comes to things like query-level lineage or automatic explanations of what a model is actually doing.

That’s why we built DBT to English—a tool that gives you LLM-powered insights into your DBT project. Whether you're using Bedrock or Anthropic, you can plug in your preferred LLM and get clean, contextual, human-friendly interpretations of your models, SQL logic, metadata, and lineage. Plus, it’s fully customizable: tweak the system prompt to get documentation that fits your team’s voice and style.

In this blog post, we’ll walk you through how to get started with the tool and show you how it can bring clarity and accessibility to even the most complex DBT projects.

Running Locally

To get started, clone the repository: https://github.com/hipposys-ltd/dbt-to-english

Then, copy the environment template file and create your local .env configuration:

cp private.env-template private.env

In your .env file, set the LLM_MODEL variable depending on the provider you're using:

• For Anthropic models, start the value with Anthropic: and provide your ANTHROPIC_API_KEY.

• For Amazon Bedrock, start with Bedrock: and include your AWS credentials and connection settings.

Once configured, build and start the app with Docker:

docker compose up --build

After it’s running, open your browser and go to http://localhost:8501/ to access the Streamlit interface.

DBT to English Translation

When you open the UI, you'll see a form that asks for two files: catalog.json and manifest.json.
If you don’t have your own DBT project, you can use sample files from our repository:
DbtExampleProject/target/catalog.json and DbtExampleProject/target/manifest.json.

Once you've selected the files, click on “Parse JSON Files.”
This will reveal two additional fields:

1. A field to select the nodes (models) you want to parse or generate documentation for.

2. A System Prompt field that defines the instruction sent to the LLM.

   

Choose a few models and leave the System Prompt as-is for now, then click “Run LLM.”

You’ll receive a detailed, plain-English explanation of your selected model.

• First, you'll see a natural language description—no SQL syntax or complex calculations.

  

• Next, an interactive graph will be generated. Unlike dbt Docs or dbt Cloud, this graph includes not just model-level lineage but also internal relationships like CTEs and calculations.

  

• Following that, SQL Logic & Calculations and Columns Table will show how each column was derived, including its source tables.

  

  

• Finally, you'll see a Dependencies Table with brief explanations for each dependency.

Pretty cool, right?

Want to change the output?

Perfect! That’s exactly what the System Prompt is for. Let’s go back to it and take a closer look.

The response you received was directly shaped by the System Prompt. Feel free to experiment—change the structure, add your own instructions or formatting, rearrange the output, etc.

Important: If you modify the diagram-related instructions, please keep the example section as-is.
Our UI relies on that example to recognize the diagram portion in the LLM response and render it as an interactive graph.

Conclusion

"DBT to English" is our way of making DBT projects more accessible, understandable, and human-friendly—no matter your technical background. By leveraging the power of large language models, we’ve opened the door for teams to explore data models, understand complex SQL logic, and visualize lineage at a much more granular level—all through a clean, conversational interface.

But this is just the beginning.

We enthusiastically invite the community to contribute to this open-source initiative—whether you're interested in adding new features, supporting more LLM providers, improving prompts, or enhancing the UI. Your feedback and contributions will help shape the future of how we understand and interact with modern data stacks.

Visit our GitHub repository at https://github.com/hipposys-ltd/dbt-to-english to join the project, submit pull requests, or share your ideas. Let’s make DBT more approachable!

Introduction

Unless you’ve been living under a rock, you’ve probably heard of MCP by now. In a nutshell, it’s a way to add extra functionality to AI systems like Claude Desktop, by giving them new “tools” that they can use. The way to do this is to create something called an MCP server, which provides the AI system with a list of new functionality.

At Hipposys, we built a custom MCP server for Apache Airflow that can transform how teams interact with their orchestration platform. Our implementation enables you to query pipeline statuses, troubleshoot failures, retrieve DAG information, and trigger different DAGs depending on their statuses.

This article will guide you through setting up our MCP server locally, allowing you to build powerful AI assistants that can manage your Airflow environment with simple language commands—no more complex UI navigation or code modifications required.

What makes this especially powerful is that it opens up Airflow insights to a wider range of users. Product managers, customer-facing teams, and other stakeholders—who may not have access to Airflow or feel comfortable using it—can now get clear answers about which jobs ran, failed, or succeeded. This enables better collaboration across teams and reduces the dependency on engineering for operational insights.

Check out our repository at https://github.com/hipposys-ltd/airflow-mcp to get started with the implementation.

Running Airflow MCP locally

Let's start with running MCP in Claude Desktop. You can connect to an existing Airflow instance if you have permissions for it. If you don't have a running Airflow environment or just want to test things out, we’ve provided a demo Airflow env in the Airflow MCP repository - you can clone the repo https://github.com/hipposys-ltd/airflow-mcp and run just airflow.

Installing Just on Mac

You can install Just on macOS using Homebrew:

brew install just

Just is a command runner that executes commands defined in a justfile. It's similar to Make but with a simpler syntax and additional features, making it ideal for automating project tasks like our Airflow setup.

Important: If you're using your own Airflow cluster, make sure the REST API is enabled with Basic Auth.

You can enable it by setting the following environment variable in your Airflow configuration:

AIRFLOW__API__AUTH_BACKEND=airflow.api.auth.backend.basic_auth

More details can be found in the Airflow API Authentication documentation.

Now, open Claude Desktop, go to Settings, select the Developer tab and edit the MCP config. Paste the following configuration. Once done, restart Claude Desktop:

{
    "mcpServers": {
        "airflow_mcp": {
            "args": [
                "run",
                "-i",
                "--rm",
                "-e",
                "airflow_api_url",
                "-e",
                "airflow_username",
                "-e",
                "airflow_password",
                "hipposysai/airflow-mcp:latest"
            ],
            "command": "docker",
            "env": {
                "airflow_api_url": "http://host.docker.internal:8088/api/v1",
                "airflow_password": "airflow",
                "airflow_username": "airflow"
            }
        }
    }
}

This MCP setting requires Docker to be installed as it will use our image (hipposysai/airflow-mcp) to run the Airflow MCP server.

Change airflow_api_url, airflow_username and airflow_password if you have your own Airflow cluster; otherwise leave them as shown in the example.

Once configured, open Claude Desktop and ask "What DAGs do we have in our Airflow cluster?"

Claude will connect to your Airflow instance through the MCP server and list all available DAGs.

Expanded Airflow MCP Functionality

Let's explore more complex operations with Airflow MCP. We'll identify all DAGs that failed during their most recent execution and trigger them automatically, then monitor their status.

“Identify all DAGs with failed status in their most recent execution and trigger a new run for each one”

The analysis reveals that the transform_forecast_attendance DAG failed and was subsequently rerun. Checking the Airflow UI confirms it was successfully triggered and is now executing.

When we query the status of this new run, we discover it failed again.

“Did it succeed?”

This recurring failure prompted us to investigate which operators are being used by the DAG.

Interestingly, it employs only a single Python operator.
To understand the issue better, we checked its historical performance and found that this DAG has never completed successfully.

“Has this DAG ever succeeded?”

Such troubleshooting becomes significantly more streamlined with Airflow MCP. Note that throughout this entire process—retrieving Airflow metadata, identifying failed DAGs, triggering reruns, and analyzing DAG components—we accomplished everything directly through the MCP interface without having to access the Airflow UI.

Conclusion

The Airflow MCP represents a significant advancement in how data teams interact with their workflow orchestration. By bridging natural language processing with Airflow's powerful capabilities, we've eliminated countless hours spent navigating complex interfaces or writing custom scripts. But this is just the beginning of what's possible. The current functionality can be substantially extended. We enthusiastically invite the community to contribute to this open-source initiative—whether you're interested in adding new features, improving documentation, or enhancing compatibility with different LLM providers. Visit our GitHub repository at https://github.com/hipposys-ltd/airflow-mcp to join the project, submit pull requests, or share your ideas. Together, we can transform how teams manage and interact with their data pipelines, making Airflow more accessible and powerful than ever before.

In our next blog post, we'll dive into how to integrate Airflow MCP with the LangChain library and leverage it with various LLMs to unlock even more flexible and intelligent workflow orchestration.

Introduction

If you've read our article How to Create a RAG Chatbot and tried running the repository RAG-base, you might have encountered a common challenge—lack of access to an LLM service such as Anthropic’s Claude or OpenAI’s ChatGPT, or not having an embedding model set up. As a result, running the project locally or working with any RAG solution, not just ours, becomes difficult.

To address this, we've introduced several updates that make it easier than ever to get started—no LLM subscription or complex setup required!

Run Models Locally?

You might be wondering—how is it even possible to run models locally?

There are various tools available for running AI models on your local machine. In this tutorial, we’ll be using a tool called Ollama, which allows you to install and run models available in the Ollama model library. However, to make things easier, we'll use a pre-configured Docker image of Ollama that comes with pre-installed models (both embedding and LLM), so you won’t need to install anything manually. It's worth noting that many other tools can run models locally, each with its own advantages and limitations.

Can I Run ChatGPT, Claude, or Amazon Titan Locally?
Unfortunately, no. The weights of these models are proprietary and not publicly available, meaning they cannot be run locally. Instead, we rely on “open-weight” models, such as Llama. You can find Llama models in the Ollama library or on Hugging Face.

Great! So I Can Run an Open-Weight Model Locally and Get the Same Performance as ChatGPT?
Not quite. One of the key factors that affect a model's accuracy is the number of parameters it has. While the exact number of parameters in ChatGPT-4 is not publicly disclosed, various sources estimate it to be around 1.8 trillion parameters.

For comparison, I tried running llama3.1, which has 8 billion parameters—about 225 times fewer than ChatGPT-4. The result? It felt significantly less accurate, and my MacBook Pro, with 16 GB RAM and 6 Apple M2 Pro CPU cores allocated to Docker, struggled to handle it. The fans made scary noises, and the model ran extremely slowly (note: I was using the CPU, not a GPU).

It's important to choose a model that matches your hardware capabilities. Additionally, when we pay for LLM services such as ChatGPT or AWS Bedrock, we're not just paying for the model itself but also for the infrastructure—powerful hardware, optimized backend services, and user-friendly interfaces that enhance the overall experience.

Implementing Local Model Support

To use a local LLM or embedding model, specify a model from the Ollama model library under the LLM_MODEL_ID and EMBEDDING_MODEL keys in the configuration file, following the ollama keyword.

Important: Ensure that the selected LLM supports tools for compatibility with the existing architecture. If you modify the default values, don’t forget to update the LLM and EMBEDDING_MODEL arguments in docker-compose.local-models.yml.

Once you've made the necessary changes, run the following command to start the Docker container with Ollama models and the open-webui interface for testing:

docker compose -f docker-compose.yml -f docker-compose.milvus.yml -f docker-compose.local-models.yml up --build

After the container is up and running, you can access the UI at localhost:8082 to experiment with your models.

If you find that your local models lack accuracy, consider replacing at least the embedding model with a local one. For example, mxbai-embed-large, which has only 300 million parameters.

Conclusion

In conclusion, running RAG models locally is now more accessible than ever, thanks to tools like Ollama and pre-configured Docker images. While proprietary models like ChatGPT or Claude cannot be run locally, open-weight alternatives such as Llama provide a viable solution for experimentation and development. However, it’s important to keep in mind that the performance of these local models may vary based on hardware capabilities and the number of parameters in the chosen model. By carefully selecting appropriate models and configurations, users can create an efficient and functional local setup without relying on costly LLM services.

Introduction

Have you ever encountered a broken dashboard in Superset, Tableau, Power BI, Streamlit, or any other visualization tool—only to realize that someone deleted a model your dashboard relied on?

What if I told you that you could prevent such issues by writing your transformation tables in dbt, storing your dbt logic in Git, and adding automated tests using Trino + dbt?

In this post, I’ll show you how to implement this safeguard for Superset dashboards, using a real example from our public GitHub repository: superset-dbt-trino-guard. 🚀

Linking Dashboard Metadata to dbt Models

How can we connect dashboard metadata with our dbt metadata?

Most visualization tools store their metadata in a relational database. What if we could create a dbt model that captures our dbt metadata and then compare what our dashboards expect to receive with what actually exists?

In this tutorial, we use a Postgres database to store dbt objects, Superset with Postgres as its application database, and Trino + dbt to validate the connections.

Exploring the GitHub Repository

To begin, clone the repository from GitHub:

🔗 superset-dbt-trino-guard

Ensure that Docker is installed on your system.

Setup Instructions

Start the environment by running the following command: docker-compose up

Launch the relevant applications by running: source all_the_apps.sh

This script will automatically open all necessary tabs in Google Chrome for you. Here's what each tab is for:

Tab 1: DBeaver Web Interface

• This is the web version of DBeaver, which we’ll use to interact with MariaDB, PostgreSQL, and Trino.

• Click on Settings -> Login, and use TestTest123 as both the username and password.

  

• Once logged in, you’ll see three connection folders on the left:

• Superset: Connect using superset as both the username and password.

• PostgreSQL: Connect using superset as both the username and password.

• Trino: Connect using tutorial as the username and leave the password blank.

• On the right-hand side, you’ll see folders containing SQL scripts that we’ll use shortly. If you don’t see them, try clicking the "Tools" menu at the top and selecting "Show Scripts".

  

  

Tab 2: Jupyter Notebook Interface

• Inside the work folder, you’ll see:

  • A dbt folder containing all the required dbt code for both transformation and tests.

  • Notebooks with Python scripts that help you run the dbt projects and test metadata.

Tab 3: Superset

Access Apache Superset for data visualization and dashboarding.

Connect using superset_admin as the username and superset as the password.

Tab 4: Trino UI

• In this tab, you can access the Trino UI by logging in with the username tutorial (leave the password blank).

• Here, you’ll find metrics, metadata, and other information about the Trino query engine.

  

  
  

Setting Up the Superset Dashboard and Data Pipeline

To build a Superset Dashboard, we need to complete three key steps:

1. Create the source table

2. Run dbt transformations

3. Set up the Superset dashboard

Step 1: Creating the Source Table

Open the DBeaver tab, navigate to the postgres database, and execute the DDL/DML queries to create the public.customers table.

Step 2: Running dbt Transformations

Next, switch to the Jupyter Notebook tab and open the edw_dbt notebook. Run the transformation script to create the customers_202301 table.

• The SQL code for this transformation is located in the customers_202301.sql file.

• This script extracts all records from public.customers that were created in January 2023.

Step 3: Setting Up the Superset Dashboard

1. In the Superset tab, create a new connection to the PostgreSQL database.

   

2. Navigate to the Datasets tab and add a new dataset for the customers_202301 table.

   

3. Create a chart using this dataset (for simplicity, we’ll use a table chart).

   

4. Finally, create a dashboard and add the chart to it.

   

To verify the changes, return to DBeaver, open the Superset queries, and check the updates in the slices, dashboard_slices, and tables tables.

Now your Superset Dashboard is set up and ready to visualize data! 🚀

Building a dbt Metadata Model

Now, we want to create a dbt model that stores metadata about our dbt project.

In the PostgresDbt project, next to the customers_202301 model, we will create a new model called metadata. This model will:

• Define the DDL for the metadata table (without inserting any data initially).

  

• Use the insert_metadata macro as a post_hook to automatically insert metadata after the model runs.

  

  

The post-hook extracts metadata from the graph context variable, inserting details about all dbt models, including:

• Schema

• Model name

• Alias

• Group

• And more...

Running the Metadata Model

1. Open the Jupyter Notebook tab.

2. In the edw_dbt notebook, run the script to build the metadata model.

3. Once the model is created, switch back to DBeaver and check the contents of the dbt.metadata table.

The metadata table should now contain information about both metadata and customers_202301 models, providing a structured overview of your dbt project.

Comparing Superset and dbt Metadata with Trino

Now, let's open the TrinoDbt project and look at the test named superset_test. While it may seem complex at first, it's quite straightforward.

Step 1: Retrieving Superset Metadata

• The first step is to extract metadata from Superset using the superset_sources CTE.

• This CTE retrieves metadata about tables used across all Superset dashboards.

• It may look complicated because it includes logic to parse datasets that are not simple tables but SQL queries instead.

• Important Note: If you're using a different data visualization tool, this is the only part of the test you'll need to modify.

  

Step 2: Comparing Metadata

The test then extracts all objects from the dbt metadata table.

Step 3: Handling Schema Differences

• Since we want to run this test before merging changes to production, we avoid using the production schema (dbt).

• Instead, dbt assigns a custom schema for each PR, such as dbt_pr_236.

• To ensure a proper comparison, we replace the dbt prefix with the custom schema before running the test.

• If you have multiple environments, you can run your tests in a test instance of the database using the production schema, eliminating the need to replace prefixes.

• It verifies that all objects retrieved from Superset (via superset_sources CTE) exist in the dbt metadata table.

  

Running the Test

1. Open Jupyter Notebook and navigate to test_superset.

2. Run the test, which first builds the metadata model within a custom schema.

3. Switch to DBeaver and verify that all objects now have custom schemas.

4. The script then executes the Trino test to compare metadata.

Verifying the Test

• Now, try deleting the customers_202301 model and rerunning the test.

• The test should fail, indicating that the model is missing.

  

• Success! We caught an issue before deploying to production, without affecting the production schema.

Additionally, I created a GitHub workflow test to automate this validation:
📁 .github/workflows/tests.yml

Conclusion

By integrating dbt metadata validation with Trino, we can proactively prevent broken dashboards in Superset. This approach ensures that all required transformation models exist before deploying changes, reducing downtime and improving data reliability.

Although this tutorial focused on Superset, the same methodology applies to any visualization tool that stores metadata in a relational database, such as Tableau, Looker, or any other BI tool that uses a relational database as its metastore.

Furthermore, by incorporating this validation into a CI/CD pipeline with GitHub Actions, we can automate metadata checks, catching issues before they reach production. This enables teams to confidently develop and maintain their data pipelines while ensuring that dashboards remain accurate and fully functional.

Introduction

Recently, Trino introduced built-in AI functions ( docs ), similar to what Databricks offers. While this is a great step forward, there are some notable limitations.

First, Trino's AI functions don’t yet support AWS Bedrock—currently, they only work with OpenAI, Anthropic, and local models via Ollama. This means that if you want to use Claude while keeping your data within your VPC, you're out of luck.

Second, the implementation is somewhat rigid. The built-in functions rely on predefined prompts that are neither visible nor customizable, making it impossible to tweak prompts or debug responses. Moreover, there's no official way to create fully custom AI functions with your own prompts and parameters.

To overcome these limitations, we built a demo that integrates Redshift, AWS Lambda, and Bedrock to create custom AI functions. In this tutorial, we’ll not only replicate some of Trino's AI functions but also extend them with custom implementations tailored to specific use cases.

Lambda Functions in Redshift

Let's start by creating a simple Lambda function that can be used within Amazon Redshift.

Step 1: Set Up IAM Permissions

First, we need to create an IAM Role that allows Redshift to invoke Lambda functions.

1. Navigate to AWS IAM → Roles → Create Role.

2. Select Redshift - Customizable as the trusted service.
   

   

3. Attach an appropriate policy. For this tutorial, I created a policy that allows invoking all Lambda functions, but in a real-world scenario, you should restrict access to specific Lambda functions for better security.
   

   

Next, associate the IAM role with your Redshift cluster:

1. Open your Redshift cluster settings.

2. Choose Associate IAM Role and select the role you just created.

Now, your Redshift cluster can invoke Lambda functions!

Step 2: Create a Lambda Function

Let’s create a simple Lambda function called test_redshift_function. This function will take two input columns and return their product.

Step 3: Invoking Lambda from Redshift

To call a Lambda function from Redshift, we use external functions.

When defining an external function in Redshift, you need to:

• Specify the IAM role with Lambda invocation permissions.

• Link it to the corresponding Lambda function.

• Define the expected input parameters.

Once the function is created, you can invoke it directly from Redshift queries!

Enhance Grammar with AI Functions

Trino introduced the ai_fix_grammar function, which takes a column of text strings and returns grammatically corrected sentences. Let's recreate a similar function ourselves using an AWS Lambda function with an x86_64 architecture.

We'll leverage the langchain library to call an AWS Bedrock model and use the following prompt to correct sentences with AI:

"You will receive a sentence from a human. If the grammar is incorrect, correct it. If it is correct, leave it unchanged. Return only the sentence and nothing else. If it is not a sentence, leave it as is."

Implementing the Function

Below is the complete code for our function:

While setting this up, we may encounter three common errors:

Error 1: Missing Library

Since we’re using langchain, we need to install it. The quickest way to package dependencies for Lambda is to install them locally and upload them as a ZIP file:

1. Install dependencies:
   pip install -r requirements.txt -t . --platform manylinux2014_x86_64 --only-binary=:all: (The --platform manylinux2014_x86_64 flag ensures compatibility with our Lambda architecture.)

2. Create a ZIP package:
   zip -r {lambda_name}.zip .

3. Deploy the function:
   aws lambda update-function-code --function-name {lambda_name} --zip-file fileb://{lambda_name}.zip

Error 2: Timeout Issues

LLM calls can take longer than typical Lambda executions. Since the default timeout is relatively short, we should increase it:

1. Navigate to Configuration → General Configuration

2. Click Edit

3. Increase the Timeout value
   

   

   
   

Error 3: Insufficient Permissions

To allow our Lambda function to invoke AWS Bedrock, we need to update its role:

1. Go to IAM Roles

2. Attach a policy that grants Bedrock invocation permissions

   

   
   

Final Steps

Once the Lambda function is deployed, we just need to create an external function as we did before—and it's ready to use! Some sentences will remain unchanged if they are already correct, while others will be automatically corrected by the AI.

AI Function with a Custom Prompt

Now, let's take it a step further and allow our analysts to customize the LLM’s behavior by passing a prompt as a parameter. Here’s the code for our Lambda function:

For this example, I created a "feedback" table. Let’s say we want to use the LLM to determine whether each feedback entry is positive or negative. To achieve this, we pass our custom prompt:
"Determine whether the feedback is positive or negative, and return only one word as the result."

Next, we provide the "feedback" column as input. The function processes each entry, and as expected, all feedback is correctly classified.

Conclusion

By integrating Amazon Redshift, AWS Lambda, and Bedrock, we’ve demonstrated how to create flexible, custom AI functions tailored to specific use cases. This approach not only allows analysts to modify prompts and debug outputs, but it also enables the addition of new AI-powered functions without waiting for official support from query engines like Trino.

Most importantly, using AWS Bedrock within a VPC ensures that no data ever leaves your secure environment, addressing privacy and compliance concerns. Whether you need grammar correction, sentiment analysis, or any other AI-powered transformation, this architecture provides a scalable, customizable, and secure foundation for AI-driven analytics within Redshift.