Disclaimer: AI helped with the English translation and syntax checks. The ideas, structure, text, and occasional weirdness are mine, for better or worse.
Today feels like a good day to publish some of my scribbles about working with Microsoft Fabric and the ecosystem built around it, all designed to make our lives āeasier.ā
I genuinely like GitHub Copilot.
Especially when working with Microsoft Fabric and Data Engineering.
It helps me focus on architecture, business logic, data models, SQL, notebooks, pipelines, and all those wonderfully specific problems that usually appear shortly after somebody says: āThis should be quite simple.ā
Before Copilot, snippets in DataGrip and Redgate SQL Toolbelt were my heroes. Like, for real.
I built templates with variables and placeholders for recurring SQL patterns. I collected useful fragments, copied them into the next project, and carefully replaced everything that belonged to the previous customer. They felt incredibly efficient.
Today, I look back and wonder how I managed to live with such antiquated technology.
Then, somewhere deep inside, my subconscious starts knocking. It reminds me that I had to walk through the valley of tears called learning. I had to write things myself, get them wrong, understand why they were wrong, improve them, and eventually build those snippets because I understood exactly which repetitive work I wanted to avoid.
Without that journey, I would not be where I am today.
And somewhere further back, another thought quietly asks:
What about future Data Engineers? Will they understand what the machine produces for them? Will they understand the difference between SQL that runs and SQL that belongs in a maintainable data platform?
Will they recognize a good data model, a bad data model, and a beautifully generated solution to the wrong problem?
Will they understand why Fabric code works in one workspace and fails in another?
Or will understanding slowly become optional as long as the notebook completes, the pipeline turns green, and the tests remain sufficiently optimistic?
I do not have a final answer.
For now, Copilot is very good at taking repetitive work away from me. It can help with SQL patterns, notebook structures, documentation, metadata-driven development, variable libraries, naming conventions, and the small adjustments that used to consume a surprising amount of time.
This gives me more time to think about the actual platform.
Where should business logic live?
Which part belongs in Silver and which belongs in Gold?
Which objects should be reusable?
Which dependencies have we created?
What will happen when the solution moves into another workspace, tenant, or environment?
And, perhaps most importantly:
We use repository context, documentation, custom instructions, and the other mechanisms available to help the model understand what we are building.
Errors still happen.
That is understandable. It is an LLM, not an Oracle (no pun intended).
Although both have historically produced confident answers based on incomplete context.
My issue starts somewhere else. I want to understand what Copilot is doing. When "Auto" selects a model, I want to see which model was selected. I want to understand why it was selected and what that decision means for quality, latency, and cost.
Right now, Auto looks like a harmless and convenient option in a dropdown. Behind that dropdown, a technical and economic decision is being made on my behalf.
That decision should be visible while I am working.
Show me the selected model.
Show me why it fits the task.
Show me the expected cost category.
Show me when a small and efficient coding model is enough.
Tell me when the task really needs something larger.
We already have models such as MAI-Code-1.1-Flash. It is fast, surprisingly capable, and affordable enough for many everyday development tasks.
That is exactly the kind of model I want Fabric developers to understand. Every task has a different shape. Generating a familiar SQL pattern is different from reviewing the architecture of a metadata-driven ingestion framework. Explaining a notebook cell is different from analysing dependencies across workspaces.
Creating documentation from an existing structure is different from designing the structure itself.
The same model does not have to be the best economic and technical answer to every one of these tasks.
Developers should understand this without studying model documentation, request multipliers, context windows, editor modes, policy settings, and the financial personality of every option in the model picker.
The way Copilot is used matters as well.
Inline suggestions, chat, agents, editor windows, programming languages, and task types create different situations. A specific model may be the better choice. A particular editor context may matter. Auto may make a perfectly good decision.
The interface rarely tells me enough about it. This feels familiar. I had a similar discussion when Capacity Unit licensing appeared in Microsoft Fabric. The technology was powerful. The possibilities were exciting. The concept made sense. Understanding which workload consumed capacity, when it consumed capacity, and how an architectural decision affected the bill was considerably less exciting.
A notebook, pipeline, Warehouse query, semantic model, or enthusiastic experiment could suddenly turn into an educational experience for the finance department.
And here we are again.
As an Xbox player who is now rather pleased to have bought one on day one after seeing last weekās prices, another question comes to mind:
While we are all asking for more efficiency, are we repeating the old āmore ironā mistake from the server-farm era?
Demand increased, so we added more hardware.
More servers.
More memory.
More capacity.
At some point, infrastructure growth became the answer to problems that might also have deserved better architecture.
Fabric should already have taught us something here.
Scaling capacity is useful.
Understanding workload behaviour is useful.
Designing an efficient platform is even more useful.
The same principle should apply to AI models.
Efficient model selection should lead to efficient use of compute and memory. And by memory, I mean RAM, before my subconscious starts knocking again.
Using the right model for the task should result in better prices, more available performance, or simply more useful work from the same infrastructure.
A large reasoning model for every task may feel powerful. Adding another server to the rack did too, though.
Increasing a Fabric capacity can create a similar feeling. The problem disappears.
At least until somebody looks at the utilisation, the electricity bill, the Azure invoice, or the next capacity size.
Auto could help us avoid repeating the same pattern with AI. It could select a model that solves the problem well while using the resources the task actually requires. A small model for familiar and repetitive coding tasks. A stronger model for complex reasoning, architecture, and unfamiliar problems. More expensive machinery where it creates more value. Less expensive machinery where it already does the job well.
I simply want to see that happening. Because transparency would help us learn.
It would help developers understand which models work well for SQL, notebooks, Fabric artefacts, documentation, reviews, and architectural reasoning.
It would help teams create sensible guidelines.
And it would help us improve those guidelines when the technology changes again, which it certainly will, probably shortly after we finish documenting it.
GitHub Copilot can make Fabric and Data Engineering work significantly more efficient. Many things work extremely well. I enjoy working with it, and I want my engineers to use it intelligently and confidently.
Microsoft has built something I genuinely like. It could be something I love. Somehow, Microsoft keeps making that unnecessarily difficult.
Innovation belongs at the forefront. As a developer and architect, I enjoy trying things before most other people do. Exploring new approaches is part of my work and, quite often, part of the fun.
But that willingness is not universal.
Many developers simply want reliable tools that help them build reliable data platforms.
End users want the same.
They did not volunteer to become beta testers, model-routing experts, Capacity Unit analysts, or part-time researchers into token economics. And unfortunately, GA does not always mean gold. More options are usually better. Intelligent automation can be even better.
Transparency makes both understandable, trustworthy, and efficient.
Without it, Auto becomes another workload wandering around the Fabric capacity while everyone is being assured that the platform is working as designed.
Somewhere inside Microsoft, I assume, there may even be a semantic model explaining it.
Auto may select it.