Data Engineering in the AI Era: Building Trusted Data and Engineering with AI

Key takeaways

• AI-generated code is an implementation candidate, not evidence of business correctness. Define expected behaviour first, then review and test.
• Data for AI needs more than a successful upload: preparation, retrieval and freshness must be evaluated together.
• Preparation checks decide whether content is ready to publish; retrieval controls decide who may receive it. Enforce access before information reaches the model.

AI changes data engineering in two ways: it helps engineers build pipelines, and it creates new requirements for the information those pipelines supply.

An assistant can draft a transformation. An AI application can retrieve a passage from a policy document. Neither capability removes the need to define business rules, validate results and enforce access.

The engineering challenge is to make implementation faster without weakening accountability.

Engineering with AI

Microsoft Fabric’s notebook Copilot, documented as preview, supports code generation, explanation, refactoring and troubleshooting. It can use notebook context, including attached lakehouses and loaded DataFrames, to make its assistance more relevant.

Consider a supplier-transactions pipeline. Copilot might draft code to standardise dates, remove duplicates and join reference data. The engineer must still decide:

• Which fields identify a genuine duplicate?
• Should unmatched transactions be rejected or retained?
• Does the join unexpectedly increase transaction counts?
• Do totals reconcile before and after processing?

Generated code is an implementation candidate — not evidence of business correctness. Define the expected behaviour first, then review and test the result.

From code assistance to delegated tasks

Fabric’s data engineering agent (Project Osmos) extends assistance to broader tasks. Microsoft documents capabilities including inspecting data, creating and testing Spark notebooks, writing Delta tables and validating results. It remains a preview capability and is not intended for production use.

Delegation makes task definition more important. “Clean the customer data” leaves too many decisions open. A stronger instruction is:

Prepare the customer table using the approved customer identifier. Retain the latest record by source timestamp, quarantine records without an identifier, and reconcile accepted plus rejected records to the input count.

This defines the required outcome and acceptance evidence. Engineers must also resolve edge cases, such as conflicting records with identical timestamps, rather than leaving the agent to invent a rule.

Engineering data for AI

A document-based assistant needs more than a successful file upload. Its pipeline may need to extract text, preserve structure, segment documents, attach metadata and publish searchable content. Microsoft’s retrieval-augmented generation (RAG) guidance treats these as design choices to evaluate together.

Figure 1. Engineering data for an AI application: preparation and publishing are separated from request-time retrieval, with quarantine handling and an evaluation feedback loop. An illustrative lifecycle.

For an HR policy assistant, the pipeline must distinguish current policies from superseded versions, retain effective dates and preserve references to the originals.

Otherwise, retrieval can return a perfectly accurate passage from a policy that no longer applies.

The diagram also separates two responsibilities: preparation checks determine whether content is ready to publish; retrieval controls determine whether a particular user may receive it. Passing the first does not satisfy the second.

Measure quality against the intended questions

Completeness, uniqueness, validity and reconciliation remain essential. AI applications also need evidence that the prepared content supports useful retrieval.

For a document assistant, test whether:

• Extraction preserves important tables, headings and qualifications.
• Each passage retains enough context to be interpreted correctly.
• Updated or withdrawn documents are reflected in the serving layer.
• Retrieved passages retain correct source references.
• Representative questions return relevant evidence.

Suppose a supplier assistant retrieves a payment term but misses an exception in the next paragraph. The defect may lie in document segmentation — not the language model.

Evaluate preparation and retrieval together before changing the model or prompt.

Enforce access before information reaches the model

Copying data into a search index creates another place where access must be controlled. Microsoft’s secure RAG guidance emphasises retrieving grounding information within the appropriate authorisation boundary.

A workforce assistant might combine general policies with restricted compensation information. Its retrieval layer must use the relevant identity and permission controls before supplying content to the model.

Permission metadata must remain accurate as access changes. A prompt saying “do not reveal confidential information” is not a substitute for enforcing that boundary.

Operate the data behind the answer

An AI response can deteriorate without a model change. A refresh can fail, extraction can omit a section, or an index can retain obsolete content.

Microsoft’s GenAIOps guidance extends operational management beyond the model to contextual data, prompts and orchestration.

Connect engineering signals to application consequences:

Retain enough traceability to connect a poor answer to its retrieved sources, processing run and serving-layer version.

What changes for the engineer?

SQL, Python, Spark and data modelling remain foundational. They enable engineers to inspect generated code, challenge assumptions and diagnose failures.

The additional skills are retrieval design, evaluation, metadata management and controlled delegation — not prompting alone.

Start with one bounded use case. Establish a baseline, define acceptance criteria and measure both delivery effort and operational quality. Expand only where the evidence supports it.

AI can accelerate implementation. Data engineers remain responsible for the quality, meaning and accessibility of the information that reaches the application.

Documentation reviewed on 22 October 2026. Confirm current availability, preview restrictions and security requirements before implementation. Examples and the flowchart are illustrative.