LLM-Powered Data Engineering

Automating Schemas, Ingestion and Quality

Prudhvi Krovvidi · Hyderabad Python Meetup

Why LLMs in data engineering?

Systems are growing
Change is constant
Humans translate intent

Data breaks late.

Schemas drift silently.

Where we spend human time

Writing schemas
Updating dbt tests
Debugging mismatches

LLMs are good at

reasoning, understanding context, not guessing

Schemas are

living artifacts

What is SchemaForge?

Web-based schema reasoning engine
Upload CSV/Excel, get dbt rules
Interactive ER diagrams
Chat with your data

The data flow

Raw Data → Schema Inference → Relationship Detection → dbt Rules → Validation

Let me walk you through how this actually works in practice. Step 1: Raw Data - You start with CSV or Excel files. The kind of data you already have. Step 2: Schema Inference - The LLM analyzes structure, types, and semantics. Not just "this is a string" but "this is an email address, it's PII." Step 3: Relationship Detection - It infers foreign keys, identifies primary keys, understands how tables connect. This is where it goes beyond simple type detection. Step 4: dbt Rules Generation - It translates that understanding into executable dbt tests. Uniqueness checks, not-null constraints, relationship tests. Step 5: Validation - You review and deploy. The LLM suggests, you decide. The key insight: this isn't just automation. It's reasoning about your data structure, then codifying that reasoning into tests. This flow fits naturally between your ETL and your data quality checks. It doesn't replace either, it connects them.

Demo walkthrough

Upload CSV/Excel file
Watch real-time schema inference
Explore interactive ER diagram
Chat: "Find potential data quality issues"
Export complete dbt project

I'll switch to the live demo now. Watch for five specific things: First, I'll upload a CSV file. Notice it processes multi-sheet Excel files too. Second, watch the schema inference happen in real-time. You'll see the LLM response streaming in, not waiting for completion. This is important for large files. Third, the ER diagram. I can drag tables around, see relationships, understand foreign keys visually. This is built with GoJS and it's fully interactive. Fourth, the chat interface. I'll ask it "Find potential data quality issues in this dataset." Watch how it analyzes the data and suggests specific dbt tests. Fifth, I'll export the complete dbt project. This isn't just YAML. It's a full project with setup scripts, profiles, models, and tests. You can run "bash setup_dbt.sh" and have a working dbt environment in seconds. DEMO NARRATION DURING EACH STEP: Upload: "Here's a customer transactions file with multiple sheets." Schema: "Notice it's not just detecting types. It's understanding semantics. 'email' is flagged as PII automatically." ER Diagram: "See these relationships? It inferred that customer_id is a foreign key without being told." Chat: "Let me ask it a question... and watch the response stream in real-time." Export: "One click, and I have a production-ready dbt project with automated setup." If the demo slows down: The key insight is that this bridges the gap between raw data and production-ready data quality checks. No manual translation needed.

Real-world impact

Zero to dbt project in 2 minutes
Automatic PII detection
Visual schema exploration
Natural language to SQL

Let me give you concrete examples of how this changes your workflow: Scenario 1: New data source arrives. Traditionally, you'd spend an hour writing schema definitions, another hour creating dbt tests. With SchemaForge, upload the file, get a complete dbt project in 2 minutes. Not an exaggeration. Two minutes. Scenario 2: Compliance audit asks "where's our PII?" Instead of manually reviewing every column, upload your schemas. It automatically flags email, phone, SSN, credit card patterns. Privacy by default. Scenario 3: You inherit a complex database with 50 tables. Instead of reading documentation that's probably outdated, upload the schemas and see an interactive ER diagram. Drag tables around, understand relationships visually. Scenario 4: Business asks "can we add a uniqueness check on customer_email?" Instead of writing SQL, ask the chat: "Add uniqueness test for customer_email." It generates the dbt YAML immediately. This isn't about replacing your expertise. It's about eliminating the boring translation work so you can focus on the interesting problems.

Run it anywhere

Browser with OpenAI API
Laptop with Ollama (no internet)
CI/CD with custom LLM endpoint
Pre-commit hooks for validation

One of the most interesting architectural decisions: extreme flexibility in deployment. Option 1: Use it in your browser with OpenAI API. Standard approach, works great. Option 2: Run Ollama locally on your laptop. No API keys, no internet required, no data leaving your machine. Perfect for sensitive data or air-gapped environments. This is huge for enterprises with strict data policies. Option 3: Point it at your company's internal LLM endpoint. Many companies are running their own models. SchemaForge works with any OpenAI-compatible API. Option 4: Integrate into CI/CD. Generate dbt tests automatically when schemas change. Everything is deterministic and reviewable in Git. The pattern here is: you choose your deployment model based on your constraints. The tool adapts to your environment, not the other way around. This is what makes it production-ready for real companies with real compliance requirements.

What stays the same

Python owns ETL
dbt owns execution
Humans own decisions

Try it yourself

prudhvi1709.github.io/schemaforge

Scan the QR · Connect · Contribute

This is completely open source. MIT licensed. You can try it right now at prudhvi1709.github.io/schemaforge. No signup, no installation. Just open it and upload a CSV. The GitHub repo has the full source code. It's a great reference if you're building similar tools. Even if you don't use SchemaForge directly, the pattern is reusable: LLMs for schema reasoning, streaming for UX, client-side for privacy. Happy to take questions, especially around: - How this fits into existing pipelines - Trade-offs between local and cloud LLMs - Where this approach may not fit - Technical implementation details - Privacy and compliance considerations PERSONAL REMINDERS: Slow down. Pause before the demo. Let the demo breathe. Don't over-explain AI internals. Always anchor back to Python and dbt. Emphasize the real-world scenarios. Show enthusiasm but stay grounded.