Data Engineer in the Age of the AI Agents
DataHub Consulting, Experts in Analytics, Business Intelligence, and ComplianceRead it in 20 minutes
1200
627
Read it in 20 minutes
AI agents can now build a working business intelligence solution: the lakehouse, the pipelines, the semantic model and the Power BI reports. That single fact is changing how organisations think about data engineering, and it’s changing the data landscape faster than most of us expected.

AI adoption is touching every industry, and companies like Anthropic, OpenAI and Microsoft are racing to put capable agents into the hands of everyday teams.
Over the last year I have seen a real change in mindset from the organisations that Datahub work with. The conversation used to start with , “how many data engineers do we need, and for how long?”. Now it often starts with “We want to adopt AI in our data framework, how can we do this?”
In this article I look at where the data engineer has come from, what AI agents can genuinely do today on a Microsoft Fabric project, and what I believe the role will look like in the next few years.
Lets first ask the question and understand the role of the data engineer. Since AI is reshaping data engineering, it felt only fair to ask two AI assistants to define the role. Here is what they said.
A Data Engineer is an IT professional responsible for designing, building, maintaining, and optimising the systems and infrastructure that collect, store, process, and deliver data so that organisations can use it effectively. In simple terms, a Data Engineer makes data available, reliable, accurate, and accessible for people and systems that need it.
Its list of key responsibilities:
A Data Engineer designs, builds and maintains the systems that collect, store, move and transform data, so it arrives reliable, well structured and ready for analysts, data scientists and business users. Put simply, data engineers build the infrastructure, and others use the data it delivers.
Its core responsibilities:
I built my first data warehouse in 2007, and it set the direction of my whole career. Since then, I have worked with Microsoft Gold Partners around the world, first as a Senior Consultant and then as a Solutions Architect, before founding Datahub Consulting in 2018. The tools have changed completely in that time. The core job, turning messy source data into something the business can trust, has not.

Figure: four eras of data engineering, and how the engineer’s time has shifted along the way.
On-premises BI (2007 to 2015). When I started, everything lived in the client’s own data centre. We orchestrated the loads with SSIS, built the reports in SSRS, and if the business wanted to slice and dice the numbers, we built multidimensional cubes and wrote MDX. Projects were long and waterfall, and every ETL package was built by hand.
Early cloud adoption (2016 to 2022). In 2016, Azure SQL Data Warehouse moved the warehouse into the cloud, and in 2021 Azure Synapse Analytics brought the warehouse, Spark and pipelines together in one place. Data Factory took over orchestration from SSIS, and we built in Visual Studio with CI/CD pipelines in Azure DevOps. If I’m honest, a lot of the early work was still the same on-premises patterns lifted and shifted onto someone else’s hardware. It was during this period, in 2018, that I set up Datahub Consulting.
The lakehouse (2023 onwards). Microsoft Fabric and Databricks changed how we build. Development moved into notebooks written in T-SQL, Python and PySpark, built in Visual Studio Code and managed through Azure DevOps, and pipelines became metadata-driven rather than built one at a time. This is when data engineering started to look a lot more like software engineering.
AI-assisted (2026 and beyond). The platforms haven’t changed, and we still work in Visual Studio Code and Azure DevOps, but an AI agent now writes much of the code. The engineer specifies what’s needed, the agent builds it, and the engineer validates the result. For the first time, the biggest change is to how we spend our day rather than to the toolset.
That’s why I added the bar along the bottom of the timeline. For most of my career, the tools changed but the time went on the same thing: writing code. Now that time is moving towards specifying, reviewing and validating.
Each shift asked data engineers to learn a new set of tools. The latest shift is different: it asks us to change what we spend our time doing.
In 2026, AI moved from “technology of the future” to an everyday tool across every industry, and business intelligence was no exception. I will be honest: I got this one wrong. I expected AI to assist data engineers by writing a bit of SQL here and a Python function there. I did not expect it to build the solution from start to finish.

As the owner of Datahub Consulting and a data engineer with nearly two decades of experience. In some cases, I’m now talking to organisations that no longer want to engage data engineers the way they used to. They want Claude to build the BI solution, with a data engineer to manage the process.
Other example of organisations adopting an AI first approach:
Am I happy with an AI first approach? I wasn’t at first but quickly realised that we must adopt AI as a co-builder. This has allowed Datahub Consulting to deliver our accelerated solutions. We will always insist on completing the project whilst maintaining control, governance, and security. As data professionals we will not compromise on this.
The real question is not whether AI can build a BI solution. It is how much of the work an organisation is comfortable handing over. From what we see, most sit somewhere on this scale:
What holds organisations back is rarely the technology. It is trust and control. Who has approved the agent’s access to production data? Is personal data leaving the tenant? Who is accountable when a KPI is wrong? For regulated sectors such as finance, healthcare and aviation, these are data protection questions as much as engineering ones, and they need answers before the agent is switched on.
Asa well as a Data & AI Practice, Datahub have a Risk & Compliance division where we advise organisations on global data protection laws. We have recently implemented MS fabric solutions to include safeguards to protection personal information where employee personal information was included. These safeguards also ensured compliance with UK and EU data protection laws. Personally, I would question if the AI agent would pick up on this if not prompted to include specific data protection fundamental framework.
Take one of the most common data engineering projects we deliver: a Microsoft Fabric medallion architecture, a semantic model and a suite of Power BI reports. A few years ago, that was a three-to-six-month engagement for a small team. With an AI agent such as Claude Code, most of the build can now be generated from a well-written specification.
The reason this works is that almost everything in a modern Fabric solution is now code. Notebooks, pipelines and lakehouse definitions can live in a Git repository. Power BI projects (.pbip files) store the semantic model as readable text definitions and the report as structured files rather than a single binary .pbix. Anything that is code, an agent can read, write and change.
Here is what that looks like layer by layer:

There are different degrees of support that an AI Agent can give. From a basic level where you ask it to create code to do x, y, z.
To then allow the AI agent the ability to develop the solution as a co-builder. Personally, I would only allow an AI agent to work where there is full CI/CD, in its own branch.
With CI/CD you can branch from main so that any development will be done in isolation in the branch. Then I would expect a human data engineer to review, check, validate, and then initiate the merge back into main.
Datahub have tested using an AI agent on our own internal demo MS Fabric capacity using .md files so that the agent is used as a co-builder.
If you’ve ever opened a GitHub repository, you’ll have seen a README.md file. Markdown files are just plain text with a bit of formatting: headings, bullet points and the odd table. There’s nothing clever about them. But they have quietly become the most important files on an AI-assisted project, because they’re how you tell the agent how to behave.
Claude Code, for example, reads a file called CLAUDE.md from the root of the project every time it starts work. Other agents look for an AGENTS.md file that does the same job. Whatever you write in that file becomes the agent’s standing orders, and from there it can point to other .md files: your naming conventions, your coding standards, your security rules and your table schemas.
I think of it like onboarding a new consultant. On their first day you’d give them our standards, walk them through the client’s environment and tell them what they mustn’t touch. The .md files are that induction pack. The difference is that you write it once and the agent reads it every single time, without skipping the boring bits.

Figure: the .md files feed the agent, the guardrails limit what it can do, and a consultant signs off everything it produces.
There’s no single right way to organise them, but this is how a Fabric project could be set up:
| File | What it tells the agent |
| CLAUDE.md | The rulebook. How we work on this project, which files to read before starting, and the hard rules: never deploy to production, never put credentials in code, and stop and ask if a requirement isn’t clear. |
| solution-design.md | The design agreed in the discovery workshop: the source systems, entities, business rules and KPIs. |
| naming-conventions.md | How everything is named, from lakehouses and notebooks to tables, columns and DAX measures. |
| coding-standards.md | Our approach to PySpark and T-SQL: logging, error handling, incremental loads, comments, and notebooks that can be safely re-run. |
| schemas/*.md | A file per table or layer: columns, data types, keys, slowly changing dimension types and source-to-target mappings. |
| security.md | Workspace roles, row-level security, where secrets live (Azure Key Vault, never in code), and which columns hold personal data and how they’re masked or classified. |
| data-protection.md | How the agent handles personal data under UK GDPR: data minimisation, pseudonymisation, sensitivity labels, retention, and when to stop and flag a new source to the consultant. |
| testing.md | What “done” means: reconciliation rules, row count checks, tolerances and the tests that must pass before anything is handed over. |
| templates/*.md | The structure for the technical design, data dictionary and runbook that the agent fills in at the end. |
To give you a feel for it, here’s a cut-down version of a naming conventions file:
That’s all it is. Without a file like this, an agent will happily make up its own names, and they won’t always be the same from one notebook to the next. With it, every object it creates follows our standard, which means anyone at Datahub can pick up the project and find their way around.

The files I spend the most time on are the guardrails, because this is where engineering meets compliance. A few lines in CLAUDE.md, security.md and data-protection.md decide whether the agent ever goes near personal data, whether it can deploy anything on its own, and what it does when it isn’t sure. Ours say, in plain English, that the agent works in a development workspace only, never sees real customer data, never stores a password or connection string in code, and stops to ask a person rather than guess.
One word of warning. An instruction in a .md file guides the agent, but it doesn’t enforce anything by itself. So we back the files up with proper controls: workspace permissions, a service account with limited access, and agent settings that block the commands it shouldn’t run. The .md file says what the rules are. The permissions make sure the rules can’t be broken. For a regulated client, that combination matters as much as any line of PySpark.
Because of the work we do in our Risk & Compliance division, data protection gets its own file rather than a couple of lines in the security one. The agent reads data-protection.md before it touches any source, and it covers how personal data must be handled in every pipeline, notebook and report it builds. Here’s a shortened version:

Two lines in that file matter more than the rest. The first is that the agent never decides a lawful basis or a retention period itself. Those are judgements for a person, and often for the client’s DPO. The second is that if the agent finds personal data it wasn’t expecting, it stops and tells us. That might be the trigger for a DPIA, and I’d much rather find that out in development than after go-live.
It also means compliance is built in from the first pipeline, not bolted on at the end. Every personal data column is labelled, pseudonymised and documented as it’s built, so the data dictionary that comes out the other end is something a DPO can actually use.
Once the files are in place, the process becomes very repeatable:

Figure: the agent loops between build, test and fix on its own; people decide at the plan and at hand-over.
For me, the documentation step is the real win. On traditional projects, documentation is the thing that gets squeezed when the deadline gets close, and it’s often out of date by go-live. Here it’s produced from what was actually built, so it matches the solution.
The other benefit is reuse. Most of these files don’t change from one client to the next. Our naming conventions, coding standards and testing approach are the same whether we’re working with an airline or a retailer. Only the design, the schemas and the security details change. So every new project starts with a set of Datahub standards the agent already follows, and that’s a big part of how we deliver in weeks rather than months.
The agent is only as good as its instructions and its guardrails. At Datahub, every build follows the same human-led flow: a consultant runs a discovery workshop, the requirements go to Claude, Claude produces a solution document, and the consultant approves or amends it before any build begins. The consultant then reviews the build, runs the tests and signs off the numbers with the business.

Datahub’s AI-assisted build flow · 5 steps, 3 human gates
Claude drafts and builds; a named consultant sets the requirements, approves the design and signs off the result. If the design needs changes, it goes back to Claude before any build starts.
That last step is the one no agent can do on its own. Claude can write a perfect DAX measure for “revenue”, but only someone who understands the business knows whether it should include refunds, intercompany sales or VAT.
The data engineer is not disappearing, but the job is moving from writing code to directing, checking and owning what an AI agent produces. Think of the agent as a very fast junior colleague: tireless and capable, but in need of clear direction and a careful reviewer.
| The traditional data engineer | The AI-era data engineer | |
| Main output | Hand-written pipelines, notebooks and DAX | Clear specifications, approved designs and validated results |
| Time spent | Mostly building | Mostly requirements, review, testing and stakeholder work |
| Key skills | T-SQL, Python, PySpark, SSIS, DAX | All of those, plus prompting, code review, data modelling and testing |
| Delivery timescale | Three to six months | Six to eight weeks |
| Biggest value | Knowing how to build it | Knowing what to build, and whether it is right |
The skills that will matter most:
Technical knowledge does not become less valuable. You cannot review a PySpark notebook you do not understand. What changes is that the knowledge is spent on reviewing and validating the work rather than typing it.
In a couple of years, the data engineer’s role will look very different. As a data professional i’m all for working smarter, not harder, but I will admit that the speed of AI’s arrival in business intelligence does surprise me.
For permanent data engineers, I think the role will change considerably, and those who embrace AI will thrive. Those who refuse to adopt it risk being left behind. They will have to learn to spend less time writing code and more time working alongside an AI agent as a co-worker.
I worry more for data engineering contractors. The six-month contract to deliver a BI solution is fading fast. Organisations now expect delivery in six to eight weeks, and that’s only possible with AI in the loop. I expect demand for contract data engineers to fall, while demand grows for business analysts and data analysts who can gather requirements, liaise with stakeholders, design test cases, and manage deployments.
If you are an organisation or a data engineer looking to adopt AI into your data solutions, think about what you want from the agent and what you are happy letting it do. Think about: AI as assistant, AI as co-builder, or AI as builder. This will be an early design decision.
Also, check with the IT security team at the organisation. More and more organisations are creating and adopting an AI usage policy. This policy may dictate your project constraints with the use of AI.
At Datahub we recognised early that AI would reshape both the data landscape and an organisations need for faster delivery of business intelligence solutions. We have built accelerators that deliver solutions in weeks rather than months, and that has meant integrating AI agents into how we work.
Our view is simple: AI should support our teams, not replace their judgement. Without the right guardrails, AI will deliver the wrong answer faster than ever before, and with total confidence. That is why every AI-built solution we deliver has a named consultant who approves the design, controls the agent’s access to data, tests the outputs, and signs off the numbers. Engineering and compliance, under one roof.
What are your thoughts? Is your organisation already handing BI builds to AI agents, or still deciding how far to go? I would love to hear where you sit on the scale.
The first step does not need to be complicated.
It would not cost you anything to start a conversation with our CEO, a highly experienced data professional and Microsoft Certified Trainer.
Contact Our Team
No pressure. No obligation. Just a professional conversation focused on helping your organisation understand your data landscape and what you want to achieve.
Contact Datahub Consulting today to begin the conversation and take the next step towards stronger data platform.
Contact us: Contact us | DataHub Consulting
Datahub Consulting Website: Data Consultancy Services | Datahub Consulting
We do not employ salespeople; our team are all experienced technical specialists that can talk you through any of our services.
Contact us