Comment by ssteinau on "A Repo That Mirrors Your Architecture - Source Control & Structure in Fabric"

CommentA Repo That Mirrors Your Architecture - Source Control & Structure in Fabric

ssteinau

Hi Peer,

this post, the previous posts, and the future ones are great. You put a lot of thought into this and I agree to most of it (as if that were the benchmark, right?).

One problem I have here is the layout of the solution folder in Git, because it does not work well with what you promised in the previous post: Think, start small (and scale up later).

A layer in your terminology is a combination of Fabric workspace and a concept of what goes into that workspace - let's call that a "responsibility".

So there is a workspace "Store" and it has the "store" responsibility of holding all storage related fabric items like lakehouses etc. We might as well call the workspace "Workspace1", as responsibility is now a separate thing associated to the workspace. Where am I going with this?

In the minimal reference architecture, we have Workspaces "Engineering" and "Analysis" that hold multiple responsibilities:

Engineering holds responsibilities "ingest" and "prepare", and Analysis holds "model" and "present".

We can see that because in the advanced architectures, the Engineering and Analysis layers get split into multiple layers that have the same name as the responsibilities.

The problem with that is: Fabric only allows git sync with one folder, not many. So with the folder layout in Git as in this post, the Engineering workspace can either link to "prepare" or "ingest, not both. And that defeats the purpose of the Engineering layer. In consequence, any other than the Enterprise architectures fall flat, in the strict sense.

The solution appears obvious: With Engineering being the combination of "prepare" and "ingest", let's combine the Git folders as well, so only an 'engineering' folder exists in Git. This is fine, as long as you never scale up. Because if you do, the burden of peeling out and separating "ingest" from "preparation" makes a scale up a migration. It becomes costly.

I think the better solution to this problem is to represent combined layers like Engineering and Analysis in Git by introducing nested folders.

So the folder structure becomes:

FabricOps/

├── .azure-pipelines/ # Azure DevOps pipeline definitions

├── .github/ # GitHub Actions workflows

├── automation/ # Scripts, environment definitions, IaC

├── documentation/ # Docs and design notes

└── solution/ # The platform layers themself

├── core/

├── engineering

    ├── ingest/

    ├── prepare/

├── store/

├── analysis

    ├── model/

    ├── present/

└── orchestrate/

With this structure, a scale up becomes a workspace retarget (from engineering to engineering/ingest, plus another workspace targeting engineering/prepare), or a git mv + a workspace retarget if you want to get rid of the engineering and analysis folders in Git to produce the flat structure of the post. Git mv and workspace retarget (= choosing another Git sync folder for the workspace) are two very simple operations, definitely not a migration. And this makes scale up also reversible, using the same means. You can scale down if you want to.

This isn't foolproof of course. The folders "ingest/prepare" and "model/present" appears as folders withing the Fabric workspace, which incentivizes a user to honor their intention. If the user doesn't and just puts items in the root folder, in the wrong folder or simply another folder, then we're back to doing a migration. But from the technical side, I think this is clean, we just cannot solve every problem with technology. The user has just to play along sometimes.

Best regards, Sebastian

Full disclosure: I am working on something very similar to FabricOps of Peer and might have been influenced to use some of his ideas. Isn't plagiarism the nicest form of recognition? Anyway, this is me giving back.

Sebastian, you're right. And thank you for the care that went into this. A flat tree only works at the top of the ladder, which contradicts what I promised in Episode 2. I've adopted nested folders and updated the post, with your name on it.

One thing I changed while adopting it. Your tree has orchestrate and core at the top level, but at the Minimum tier those live in Engineering, and a workspace can only bind one folder. So I shaped the tree after the three-layer floor instead: store, engineering (ingest, prepare, orchestrate, core) and analytics (model, present). That way a tier is only a choice of which folders get a workspace, and nothing ever moves.

On the "user has to play along" part, I think you're being harder on the idea than it deserves. Whether an item is in its responsibility folder is just a path in the repo, so CI can check it on every commit. Someone can still put a notebook in the wrong place, but they find out the same day instead of on the day they scale up, and fixing it is a drag and a commit, not a migration. I've added that to the post as well.

On "definitely not a migration" I agreed with one condition. It's only a rebind if the references were dynamic to begin with. Notebooks that resolve lakehouses by name, and pipelines and models whose references are resolved at deploy time, don't notice the split at all. Static IDs baked into definitions do, and those are the ones people have. So the folder layout makes scaling cheap, and designing for automation is what makes it free. Which is a decent summary of where this series is going.

Good luck with your own version. Borrow whatever's useful. This is exactly the kind of feedback that makes the ideas better.

Search Hashnode