# LLM-Powered Data Engineering - Transcript

Hey, hi all. This is Prudhvi. I work as a data scientist at Gramener, a data science company. I usually work with LLMs - meaning I try to incorporate LLMs wherever it's needed, and wherever it's not needed, I'll realize, "okay, this isn't needed here." I wanted to share some experiences with you all so that you can incorporate LLMs where it actually helps. I believe all of you use LLMs on a daily basis, so I wanted to share some of that so you can also use them in your daily life to make it easier - not replacing humans, but working along with you, so you can do in one day what used to take three.

So, why do we need LLMs in data engineering. I'm focusing on one particular topic here: data engineering, because somewhere along the way I felt that the engineering team doing data pipelines, ETL pipelines, ingestion pipelines, quality pipelines - we can actually automate and accelerate this instead of doing it manually every time. Modern data systems are large and constantly evolving. Models are being released very frequently, and they're fast and accurate - why not use that at the enterprise level itself, with some guardrails, which we'll get to.

If you look at a data engineering pipeline, there's a lot going on - quality pipelines, ingestion pipelines, the ETL pipelines (extract, transform, load) where you transform a database from one format to another, which usually takes a lot of time and needs ongoing maintenance. If something breaks in between - say there's a schema change, or a value becomes nullable because the data keeps updating - the pipeline doesn't break instantly. It breaks after some time, and then we go debugging, debugging, debugging, and eventually realize where it broke. Instead, we can automate parts of this and reduce that manual effort.

Usually in data engineering, a lot of human time goes into writing schemas - schemas being the data types of your dataset, whatever database you have. We write a schema that tells you what column is of what type. We write dbt tests - dbt is a data build tool, a Python framework used to check the quality of your data. So instead of a data analyst going through every row, we write rules and test them against the database, to see how good our data quality is and how well we're maintaining it - and debugging mismatches obviously takes even more time.

Now, we have LLMs that are good at reasoning, good at understanding context. We don't want LLMs coming into our pipelines and guessing what's happening - building context is what matters, so we build that context and then use it in our pipelines, leveraging the fact that they're good at reasoning and understanding context.

Schemas, when I say it that way, are actually living artifacts - not static contracts you write once and then ingest forever. They change as your data changes; your data is alive, so your schema keeps updating too. To build context for an LLM, you don't need to hand over the whole database - you don't need to pull your entire database into ChatGPT or Claude to explain what's happening. Instead, you give it just the schema - column names, rows, and data types. Give that context to whichever LLM you choose - OpenAI, Anthropic, whatever - and say "do this," and you'll get a better, more accurate result.

## Live demo: Schema Forge

So let me show a live demo of something I built that I think works well as an accelerator - it's called Schema Forge. I send the schema of an uploaded dataset to an LLM and ask it to build three things: one, descriptions of the dataset - what each row and each table is doing; two, relationships and entity diagrams, which is usually something the engineering team spends real time creating by hand - joins, relationship modelling, all of that; and three, with all that context, generate dbt rules so I can check the quality of the data I have. It's a web app right now, but this could just as easily be an API integrated wherever your enterprise pipelines live.

The data flow is: we take the data, pass it to Schema Forge, and it infers the schema, detects relationships and modelling, detects data types, and then generates dbt rules that we validate against.

Let me jump into the live demo. Before I show it, let me show the dataset I've been using, so you get context on what's happening. I converted it to Excel so it's easier to display. This is a pharma-trials dataset I created - it has six tables, one per sheet: patients data, doctors' data, lab results, adverse events related to the trials. There's a lot of information in it. Now, looking at this as a third person who didn't build it, I don't know what "PT type" is. I don't know what "ETH" is. I don't know what "BM score" is. That's typical of how datasets and databases actually look in the wild - whoever built it has the documentation, but a third person looking at it cold doesn't. And the LLM doesn't know either, because we haven't given it that context yet.

So what I'll do is upload this to Schema Forge and see what happens. I've configured the LLM already - in this case I'm using GPT-4.1 mini, which is a bit outdated, but fine for this. The moment I upload it, it takes the schema of the dataset along with some sample rows - I send ten random rows per sheet, not just the header, because I want the LLM to be as accurate as possible, and there's no personal or client information in this synthetic dataset anyway. Better context in, better results out.

Take the patients table we just saw - it detects "PT type" as a string and calls it a patient type code, category A, B, C, whatever the hospital may have categorized it as. It flags "ETH score" as an ethnicity flag - probably an international hospital with African, Asian, Latin American categories, that sort of thing. It does the same across all six tables. And by giving it the schema and sample rows across the whole database at once, it also recognises relationships and modelling - which column is a primary key (patient ID, for instance, tagged as PII, personal identification information, which gets treated as sensitive and not surfaced everywhere), which are foreign keys, and so on.

Now I have a pretty good understanding of the dataset. It generates the schema for every table, plus constraints per column - for example, patient ID needs to be unique, not null, a positive integer, since each ID maps to one patient. It also traces foreign key references - clinician site ID connecting to the clinicians table, for instance - and it picks up a lot of these patterns across the database on its own.

Since I have all that, why not generate the diagrams and relationships too? Normally the engineering team sits down and works this out by hand, but here I get the diagram in one shot, because the LLM understood the relationships between tables just from the foreign and primary keys it detected - joining all the tables correctly, as far as I can tell, and surfacing relationships too. For example, for the prescriptions and patients tables, it suggests joining them with an inner join for a particular use case - giving me additional information I can act on to pull more insight out of the data.

Now to the quality and ingestion side. I'll generate the dbt rules - dbt being the Python framework for testing data quality. While generating, it has the schema and relationship context, and produces three things: the SQL, the actual code to run against the dataset; the YAML, explaining what each test does; and the tests themselves, mapped to both. So I have SQL I can run directly, and YAML that explains what's being tested. It's generating now - let it run, might take a moment.

While that's generating, let me show ingestion, which is really ETL. We each have our data in a particular format - I prefer SQLite, someone else prefers Postgres, someone else prefers DuckDB. Ingestion is converting from one to another, and you can do that two ways: ETL, using Python (a non-SQL language) to convert between formats, or ELT. Here I'll select a source format - say, SQL database - and a target, say Parquet, and generate. If I have a concern, like not wanting PII fields such as patient name carried into the converted file, I can just say so, and it generates working Python scripts to do the conversion - I've tested these locally, they work.

So now we have: schema ready, relationships ready, ER diagram ready, dbt rules ready, ingestion scripts ready. That cuts out something like 80% of the time an engineering team would spend building this kind of pipeline over a month. For every project: build the context, decide what you need generated, let the LLM generate it. LLMs are very good at generating code, not at guessing - so we lean on that strength specifically.

By now it should have finished generating the dbt rules - yes, it has. For the patients table, take the patient ID test: it should be a unique identifier per patient, not null, greater than zero - the same constraints we saw in the column descriptions. For date of birth, it enforces a particular date-format regular expression. It goes through every column this way, generating tests that check quality row by row - nullables, miscoded values, all the things you'd otherwise catch by hand.

And you can layer a chatbot on top of all this by embedding the generated context - I can ask it to add a new rule, and it does, understanding what's already been generated. That's the human-in-the-loop pattern: we use LLMs to accelerate the process and the pipeline, and have all of this ready in about two minutes.

I'm presenting this through an engineering lens, but the same accelerator works for a data scientist use case too - finding hidden patterns, visualising the data, whatever the need is. The core idea is: give LLMs good context, and let them do what they're good at. It's not replacing humans - we're using LLMs as a second brain, and accelerating something that would otherwise take ten days down to a couple of minutes. There's still human-in-the-loop, still guardrails, still things to check - but the initial scaffolding is ready fast, which speeds up everything downstream.

That's pretty much the live demo - let me switch back to the slides. So, the data flow: we uploaded the dataset, retrieved the schema, identified the relationships, generated visualizations and rules - zero to dbt rules in about two minutes, PII detected along the way because it already has the rules and data types in context. I haven't shown the working chatbot version, but it's a natural-language-to-SQL chatbot - ask it to generate a new rule, and it writes the SQL for that rule. And you can run this anywhere - it's not a packaged product, but you can run it in a browser, integrate it into your backend, wire it into your CI/CD pipelines - everything here is deterministic and reviewable, so you can edit anything, and nothing is hidden from you.

What stays the same through all of this: ETL is still owned by Python, which we already use traditionally, so your Python pipelines stay put - dbt just owns the execution. Schema Forge generates the rules, dbt runs them. Download it locally, install the Python package and dbt, and it takes care of running the generated rules. It's not something we think of as replacing a human - everyone's worried "AI is taking my job," but it isn't. We're accelerating our own job with AI - it's a second brain we use.

With that, I want to wrap up - it's open source, the full codebase is out there, you're welcome to connect, and I'm open to collaborating on other open-source projects too. Scan the QR code for GitHub and LinkedIn. Happy to take questions now.