Peer Grønnerup

Sebastian, you're right. And thank you for the care that went into this. A flat tree only works at the top of the ladder, which contradicts what I promised in Episode 2. I've adopted nested folders and updated the post, with your name on it. One thing I changed while adopting it. Your tree has orchestrate and core at the top level, but at the Minimum tier those live in Engineering, and a workspace can only bind one folder. So I shaped the tree after the three-layer floor instead: store, engineering (ingest, prepare, orchestrate, core) and analytics (model, present). That way a tier is only a choice of which folders get a workspace, and nothing ever moves. On the "user has to play along" part, I think you're being harder on the idea than it deserves. Whether an item is in its responsibility folder is just a path in the repo, so CI can check it on every commit. Someone can still put a notebook in the wrong place, but they find out the same day instead of on the day they scale up, and fixing it is a drag and a commit, not a migration. I've added that to the post as well. On "definitely not a migration" I agreed with one condition. It's only a rebind if the references were dynamic to begin with. Notebooks that resolve lakehouses by name, and pipelines and models whose references are resolved at deploy time, don't notice the split at all. Static IDs baked into definitions do, and those are the ones people have. So the folder layout makes scaling cheap, and designing for automation is what makes it free. Which is a decent summary of where this series is going. Good luck with your own version. Borrow whatever's useful. This is exactly the kind of feedback that makes the ideas better.

How are you refering to your child activities, i.e. when using the Invoke Pipeline og Invoke Notebook activities? Be sure to design your pipelines with automation in mind and use dynamic references when possible. Also read my blog post on dynamically configuring pipelines here: https://peerinsights.hashnode.dev/automating-fabric-dynamically-configuring-microsoft-fabric-data-pipelines and the post on designing for automation here: https://peerinsights.hashnode.dev/designing-for-automation-in-microsoft-fabric

Hi Raeed, thanks for reading and liking the post. Silver and Gold lakehouses are related to data engineering workspaces. I also refer to these as Prepare or Transform workspaces. The Prepare/Transform stage handles the movement of data from Bronze to Silver and from Silver to Gold. The data transformantion is imlemented using Notebooks. Data in the Gold layer is then further on consumed by Serve or Insights workspaces containing semantic models. I have been a bit busy lately but I will prioritize enhancing my repo to also include metadata store, dynamic ingestion pipelines and notebooks etc. Also stay tune for an update to include automatically connecting and syncronizing with Azure DevOps Repos. I will write an additional post for this enhancement within the next couple of days.

Luiza Oancea Also one way to build job security ;) But you are right - It is a complicated to area and not always as logical as one would expect.

Thanks for your kind words, Raeed - glad you found the blog post valuable! To successfully approve the endpoint request, the identity being used needs to have at least Contributor or Owner permissions on the resource. Alternatively, a custom role with the necessary permissions could also work. While I haven’t tested it myself, the Azure AI Enterprise Network Connection Approver role might also have the required privileges - worth exploring if you're looking for a more scoped access option.