DataHub Consulting, Experts in Analytics, Business Intelligence, and Compliance 1200 627

Written by

Date

17 November 2026

Category

AI agents can now build a working business intelligence solution: the lakehouse, the pipelines, the semantic model and the Power BI reports. That single fact is changing how organisations think about data engineering, and it’s changing the data landscape faster than most of us expected.

AI adoption is touching every industry, and companies like Anthropic, OpenAI and Microsoft are racing to put capable agents into the hands of everyday teams.

Over the last year I have seen a real change in mindset from the organisations that Datahub work with. The conversation used to start with , “how many data engineers do we need, and for how long?”. Now it often starts with “We want to adopt AI in our data framework, how can we do this?”

In this article I look at where the data engineer has come from, what AI agents can genuinely do today on a Microsoft Fabric project, and what I believe the role will look like in the next few years.

What is a data engineer?

Lets first ask the question and understand the role of the data engineer. Since AI is reshaping data engineering, it felt only fair to ask two AI assistants to define the role. Here is what they said.

ChatGPT’s answer

A Data Engineer is an IT professional responsible for designing, building, maintaining, and optimising the systems and infrastructure that collect, store, process, and deliver data so that organisations can use it effectively. In simple terms, a Data Engineer makes data available, reliable, accurate, and accessible for people and systems that need it.

Its list of key responsibilities:

  • Data collection: building systems that gather data from databases, applications, APIs and external systems.
  • Data pipelines: designing automated ETL/ELT processes to extract, transform and load data into central storage.
  • Data storage: building and managing data warehouses, data lakes and databases.
  • Data transformation: cleaning, structuring, validating and standardising data.
  • Data quality: checks to make sure data is accurate, consistent, complete and trustworthy.
  • Performance optimisation: improving the speed, scale and efficiency of data processing.
  • Security and governance: storing, accessing and managing data securely and in line with regulation.

Claude’s answer

A Data Engineer designs, builds and maintains the systems that collect, store, move and transform data, so it arrives reliable, well structured and ready for analysts, data scientists and business users. Put simply, data engineers build the infrastructure, and others use the data it delivers.

Its core responsibilities:

  • Ingestion: pulling data from source systems (databases, APIs, files, streams) into a central platform.
  • Transformation: cleaning, modelling and reshaping raw data, for example into medallion layers or star schemas.
  • Pipelines and orchestration: automated, scheduled and monitored workflows for batch and real-time loads.
  • Storage and architecture: designing lakehouses, warehouses and data lakes for performance, scale and cost.
  • Data quality and reliability: validation, error handling, lineage and observability.
  • Governance and security: access control, data classification and compliance with rules like GDPR.

Nearly two decades of data engineering

I built my first data warehouse in 2007, and it set the direction of my whole career. Since then, I have worked with Microsoft Gold Partners around the world, first as a Senior Consultant and then as a Solutions Architect, before founding Datahub Consulting in 2018. The tools have changed completely in that time. The core job, turning messy source data into something the business can trust, has not.

Figure: four eras of data engineering, and how the engineer’s time has shifted along the way.

On-premises BI (2007 to 2015). When I started, everything lived in the client’s own data centre. We orchestrated the loads with SSIS, built the reports in SSRS, and if the business wanted to slice and dice the numbers, we built multidimensional cubes and wrote MDX. Projects were long and waterfall, and every ETL package was built by hand.

Early cloud adoption (2016 to 2022). In 2016, Azure SQL Data Warehouse moved the warehouse into the cloud, and in 2021 Azure Synapse Analytics brought the warehouse, Spark and pipelines together in one place. Data Factory took over orchestration from SSIS, and we built in Visual Studio with CI/CD pipelines in Azure DevOps. If I’m honest, a lot of the early work was still the same on-premises patterns lifted and shifted onto someone else’s hardware. It was during this period, in 2018, that I set up Datahub Consulting.

The lakehouse (2023 onwards). Microsoft Fabric and Databricks changed how we build. Development moved into notebooks written in T-SQL, Python and PySpark, built in Visual Studio Code and managed through Azure DevOps, and pipelines became metadata-driven rather than built one at a time. This is when data engineering started to look a lot more like software engineering.

AI-assisted (2026 and beyond). The platforms haven’t changed, and we still work in Visual Studio Code and Azure DevOps, but an AI agent now writes much of the code. The engineer specifies what’s needed, the agent builds it, and the engineer validates the result. For the first time, the biggest change is to how we spend our day rather than to the toolset.

That’s why I added the bar along the bottom of the timeline. For most of my career, the tools changed but the time went on the same thing: writing code. Now that time is moving towards specifying, reviewing and validating.

Each shift asked data engineers to learn a new set of tools. The latest shift is different: it asks us to change what we spend our time doing.

The arrival of AI agents

In 2026, AI moved from “technology of the future” to an everyday tool across every industry, and business intelligence was no exception. I will be honest: I got this one wrong. I expected AI to assist data engineers by writing a bit of SQL here and a Python function there. I did not expect it to build the solution from start to finish.

As the owner of Datahub Consulting and a data engineer with nearly two decades of experience. In some cases, I’m now talking to organisations that no longer want to engage data engineers the way they used to. They want Claude to build the BI solution, with a data engineer to manage the process.

Other example of organisations adopting an AI first approach:

  • I’ve spoken with a number of organisations where a Business Analyst (BA) and a Data Analyst have created a Fabric solution without a data engineer. The business analyst gained the requirements from the stakeholders, and the data analyst gave the agent the reporting requirements. All this was then given to Claude Code to come up with the goods. How this adhered to data engineering bast practice, data modelling concepts, and full testing, I don’t know.
  • I’ve also spoken with contractors where contract job descriptions are now including knowledge and experience of integrating Claude AI into the build.

Am I happy with an AI first approach? I wasn’t at first but quickly realised that we must adopt AI as a co-builder. This has allowed Datahub Consulting to deliver our accelerated solutions. We will always insist on completing the project whilst maintaining control, governance, and security. As data professionals we will not compromise on this.

How far will organisations go?

The real question is not whether AI can build a BI solution. It is how much of the work an organisation is comfortable handing over. From what we see, most sit somewhere on this scale:

  1. AI as assistant. Engineers write the solution and use AI for snippets, documentation and debugging. Low risk, modest gains.
  2. AI as co-builder. The agent drafts pipelines, notebooks and DAX; an engineer reviews every change before it is merged. This is where most organisations are today.
  3. AI as builder. The agent builds end to end from an approved design, inside a controlled workspace, and engineers act as the quality gate before anything reaches production.

What holds organisations back is rarely the technology. It is trust and control. Who has approved the agent’s access to production data? Is personal data leaving the tenant? Who is accountable when a KPI is wrong? For regulated sectors such as finance, healthcare and aviation, these are data protection questions as much as engineering ones, and they need answers before the agent is switched on.

Would an organisation trust AI with personal information?

Asa well as a Data & AI Practice, Datahub have a Risk & Compliance division where we advise organisations on global data protection laws. We have recently implemented MS fabric solutions to include safeguards to protection personal information where employee personal information was included. These safeguards also ensured compliance with UK and EU data protection laws. Personally, I would question if the AI agent would pick up on this if not prompted to include specific data protection fundamental framework.

How Claude can build a Microsoft Fabric medallion solution

Take one of the most common data engineering projects we deliver: a Microsoft Fabric medallion architecture, a semantic model and a suite of Power BI reports. A few years ago, that was a three-to-six-month engagement for a small team. With an AI agent such as Claude Code, most of the build can now be generated from a well-written specification.

The reason this works is that almost everything in a modern Fabric solution is now code. Notebooks, pipelines and lakehouse definitions can live in a Git repository. Power BI projects (.pbip files) store the semantic model as readable text definitions and the report as structured files rather than a single binary .pbix. Anything that is code, an agent can read, write and change.

Here is what that looks like layer by layer:

  • Bronze (raw). Claude creates the lakehouse, configures source connections and writes metadata-driven pipelines that land data from SQL Server, APIs or files, including incremental loads driven by a configuration table.
  • Silver (cleansed). It writes PySpark or T-SQL notebooks to deduplicate, type, standardise and validate the data, with logging and error handling built in.
  • Gold (modelled). It builds the Kimball-style fact and dimension tables, surrogate keys and slowly changing dimensions defined in the design.
  • Semantic model. Using the .pbip project files, it defines the tables, relationships, hierarchies and DAX measures, and can connect to an existing semantic model to extend it.
  • Power BI reports. It creates the report pages and visuals and writes the DAX behind them, based on the instructions the user provides.
  • Testing. It drafts test plans and reconciliation queries that compare row counts and totals between source, silver and gold.

The .md files that AI agents use

There are different degrees of support that an AI Agent can give. From a basic level where you ask it to create code to do x, y, z.

To then allow the AI agent the ability to develop the solution as a co-builder. Personally, I would only allow an AI agent to work where there is full CI/CD, in its own branch.

With CI/CD you can branch from main so that any development will be done in isolation in the branch. Then I would expect a human data engineer to review, check, validate, and then initiate the merge back into main.

Datahub have tested using an AI agent on our own internal demo MS Fabric capacity using .md files so that the agent is used as a co-builder.

If you’ve ever opened a GitHub repository, you’ll have seen a README.md file. Markdown files are just plain text with a bit of formatting: headings, bullet points and the odd table. There’s nothing clever about them. But they have quietly become the most important files on an AI-assisted project, because they’re how you tell the agent how to behave.

Claude Code, for example, reads a file called CLAUDE.md from the root of the project every time it starts work. Other agents look for an AGENTS.md file that does the same job. Whatever you write in that file becomes the agent’s standing orders, and from there it can point to other .md files: your naming conventions, your coding standards, your security rules and your table schemas.

I think of it like onboarding a new consultant. On their first day you’d give them our standards, walk them through the client’s environment and tell them what they mustn’t touch. The .md files are that induction pack. The difference is that you write it once and the agent reads it every single time, without skipping the boring bits.

Figure: the .md files feed the agent, the guardrails limit what it can do, and a consultant signs off everything it produces.

What goes in the files

There’s no single right way to organise them, but this is how a Fabric project could be set up:

FileWhat it tells the agent
CLAUDE.mdThe rulebook. How we work on this project, which files to read before starting, and the hard rules: never deploy to production, never put credentials in code, and stop and ask if a requirement isn’t clear.
solution-design.mdThe design agreed in the discovery workshop: the source systems, entities, business rules and KPIs.
naming-conventions.mdHow everything is named, from lakehouses and notebooks to tables, columns and DAX measures.
coding-standards.mdOur approach to PySpark and T-SQL: logging, error handling, incremental loads, comments, and notebooks that can be safely re-run.
schemas/*.mdA file per table or layer: columns, data types, keys, slowly changing dimension types and source-to-target mappings.
security.mdWorkspace roles, row-level security, where secrets live (Azure Key Vault, never in code), and which columns hold personal data and how they’re masked or classified.
data-protection.mdHow the agent handles personal data under UK GDPR: data minimisation, pseudonymisation, sensitivity labels, retention, and when to stop and flag a new source to the consultant.
testing.mdWhat “done” means: reconciliation rules, row count checks, tolerances and the tests that must pass before anything is handed over.
templates/*.mdThe structure for the technical design, data dictionary and runbook that the agent fills in at the end.

To give you a feel for it, here’s a cut-down version of a naming conventions file:

That’s all it is. Without a file like this, an agent will happily make up its own names, and they won’t always be the same from one notebook to the next. With it, every object it creates follows our standard, which means anyone at Datahub can pick up the project and find their way around.

Guardrails are the important bit

The files I spend the most time on are the guardrails, because this is where engineering meets compliance. A few lines in CLAUDE.md, security.md and data-protection.md decide whether the agent ever goes near personal data, whether it can deploy anything on its own, and what it does when it isn’t sure. Ours say, in plain English, that the agent works in a development workspace only, never sees real customer data, never stores a password or connection string in code, and stops to ask a person rather than guess.

One word of warning. An instruction in a .md file guides the agent, but it doesn’t enforce anything by itself. So we back the files up with proper controls: workspace permissions, a service account with limited access, and agent settings that block the commands it shouldn’t run. The .md file says what the rules are. The permissions make sure the rules can’t be broken. For a regulated client, that combination matters as much as any line of PySpark.

A file just for data protection

Because of the work we do in our Risk & Compliance division, data protection gets its own file rather than a couple of lines in the security one. The agent reads data-protection.md before it touches any source, and it covers how personal data must be handled in every pipeline, notebook and report it builds. Here’s a shortened version:

Two lines in that file matter more than the rest. The first is that the agent never decides a lawful basis or a retention period itself. Those are judgements for a person, and often for the client’s DPO. The second is that if the agent finds personal data it wasn’t expecting, it stops and tells us. That might be the trigger for a DPIA, and I’d much rather find that out in development than after go-live.

It also means compliance is built in from the first pipeline, not bolted on at the end. Every personal data column is labelled, pseudonymised and documented as it’s built, so the data dictionary that comes out the other end is something a DPO can actually use.

How the agent uses them to build, test and document

Once the files are in place, the process becomes very repeatable:

  • Read. The agent reads CLAUDE.md, then the design, standards and schema files it points to.
  • Plan. It writes a short build plan listing what it’s going to create. A consultant checks the plan before any code is written.
  • Build. It creates the lakehouses, pipelines, notebooks and semantic model, following the naming and coding standards.
  • Test. It runs the tests set out in testing.md. If a reconciliation fails, it reads the error, fixes its own code and runs the tests again, and it keeps going until they pass.
  • Document. Once the tests pass, it fills in the documentation templates: the technical design, the data dictionary and the runbook.
  • Hand over. Everything comes back to a consultant for review before it goes anywhere near production.

Figure: the agent loops between build, test and fix on its own; people decide at the plan and at hand-over.

For me, the documentation step is the real win. On traditional projects, documentation is the thing that gets squeezed when the deadline gets close, and it’s often out of date by go-live. Here it’s produced from what was actually built, so it matches the solution.

The other benefit is reuse. Most of these files don’t change from one client to the next. Our naming conventions, coding standards and testing approach are the same whether we’re working with an airline or a retailer. Only the design, the schemas and the security details change. So every new project starts with a set of Datahub standards the agent already follows, and that’s a big part of how we deliver in weeks rather than months.

Where the engineer still matters

The agent is only as good as its instructions and its guardrails. At Datahub, every build follows the same human-led flow: a consultant runs a discovery workshop, the requirements go to Claude, Claude produces a solution document, and the consultant approves or amends it before any build begins. The consultant then reviews the build, runs the tests and signs off the numbers with the business.

Datahub’s AI-assisted build flow · 5 steps, 3 human gates

Claude drafts and builds; a named consultant sets the requirements, approves the design and signs off the result. If the design needs changes, it goes back to Claude before any build starts.

That last step is the one no agent can do on its own. Claude can write a perfect DAX measure for “revenue”, but only someone who understands the business knows whether it should include refunds, intercompany sales or VAT.

The future of data engineering

The data engineer is not disappearing, but the job is moving from writing code to directing, checking and owning what an AI agent produces. Think of the agent as a very fast junior colleague: tireless and capable, but in need of clear direction and a careful reviewer.

 The traditional data engineerThe AI-era data engineer
Main outputHand-written pipelines, notebooks and DAXClear specifications, approved designs and validated results
Time spentMostly buildingMostly requirements, review, testing and stakeholder work
Key skillsT-SQL, Python, PySpark, SSIS, DAXAll of those, plus prompting, code review, data modelling and testing
Delivery timescaleThree to six monthsSix to eight weeks
Biggest valueKnowing how to build itKnowing what to build, and whether it is right

The skills that will matter most:

  • Requirements and business analysis. Turning a stakeholder’s question into a precise brief an agent can act on.
  • Data modelling. The agent follows the design, so the design has to be right. Kimball and medallion principles matter more, not less.
  • Review and testing. Reading generated code critically, designing test cases and reconciling numbers back to source.
  • Governance and security. Setting the guardrails: what the agent can access, which environments it can deploy to, and how personal data is protected.
  • Deployment and DevOps. Managing Git, environments and release processes so AI-built changes reach production safely.
  • Knowledge of AI Integration. Data engineers will need to learn how to integrate AI agents to meet the requirements and safeguards of the organisation.

Technical knowledge does not become less valuable. You cannot review a PySpark notebook you do not understand. What changes is that the knowledge is spent on reviewing and validating the work rather than typing it.

Conclusion

In a couple of years, the data engineer’s role will look very different. As a data professional i’m all for working smarter, not harder, but I will admit that the speed of AI’s arrival in business intelligence does surprise me.

For permanent data engineers, I think the role will change considerably, and those who embrace AI will thrive. Those who refuse to adopt it risk being left behind. They will have to learn to spend less time writing code and more time working alongside an AI agent as a co-worker.

I worry more for data engineering contractors. The six-month contract to deliver a BI solution is fading fast. Organisations now expect delivery in six to eight weeks, and that’s only possible with AI in the loop. I expect demand for contract data engineers to fall, while demand grows for business analysts and data analysts who can gather requirements, liaise with stakeholders, design test cases, and manage deployments.

If you are an organisation or a data engineer looking to adopt AI into your data solutions, think about what you want from the agent and what you are happy letting it do. Think about: AI as assistant, AI as co-builder,  or AI as builder. This will be an early design decision.

Ensure compliance with IT policies

Also, check with the IT security team at the organisation. More and more organisations are creating and adopting an AI usage policy. This policy may dictate your project constraints with the use of AI.

Datahub’s adoption of AI

At Datahub we recognised early that AI would reshape both the data landscape and an organisations need for faster delivery of business intelligence solutions. We have built accelerators that deliver solutions in weeks rather than months, and that has meant integrating AI agents into how we work.

Our view is simple: AI should support our teams, not replace their judgement. Without the right guardrails, AI will deliver the wrong answer faster than ever before, and with total confidence. That is why every AI-built solution we deliver has a named consultant who approves the design, controls the agent’s access to data, tests the outputs, and signs off the numbers. Engineering and compliance, under one roof.

What are your thoughts? Is your organisation already handing BI builds to AI agents, or still deciding how far to go? I would love to hear where you sit on the scale.

Get in Touch

The first step does not need to be complicated.

It would not cost you anything to start a conversation with our CEO, a highly experienced data professional and Microsoft Certified Trainer.

Contact Our Team

No pressure. No obligation. Just a professional conversation focused on helping your organisation understand your data landscape and what you want to achieve.

Contact Datahub Consulting today to begin the conversation and take the next step towards stronger data platform.

Contact us: Contact us | DataHub Consulting
Datahub Consulting Website: Data Consultancy Services | Datahub Consulting

Find out how we can help

We do not employ salespeople; our team are all experienced technical specialists that can talk you through any of our services.

Contact us