Azure Data Factory CI/CD with GitHub and Azure PipelinesWithout source control for Azure Data Factory (ADF), you only have the option to publish your pipeline. And, it has to validate. Now with source control, we can save intermediate work, use branches, and publish when we are ready. The next step is CI/...6 min read·bobblackburn.comAug 19, 2024
CI/CD with GitHub Database Projects and Azure PipelinesWith the high pace of development, data architects/engineers have to be able to keep up with our DevOps counterparts. We also want the many benefits of CI/CD for databases such as fewer errors, stable builds, better release management, and less manua...6 min read·bobblackburn.comMay 19, 2024
Databricks Zip File ValidationIn many IoT systems, you will receive feeds that are in a zip format. Sometimes they are empty. We want to validate they contain a file before we start processing them because it involves moving the file around different file systems, running a shell...2 min read·bobblackburn.comApr 29, 2024
3 Steps to Run PowerShell in Azure Data FactoryAzure Data Factory has many capabilities. But no tool is the best at everything. Sometimes you have an existing script that needs to be automated or PowerShell is the best programming option for the task at hand. Currently, ADF does not have a PowerS...3 min read·bobblackburn.comApr 29, 2024
SQL Server Indexing: 5 Steps to Better PerformanceCustomers don’t want to wait more than a few seconds for an application to process and move to the next step. And we keep getting squeezed on our batch window to do all our daily processing. We cannot just throw more hardware at the problem. It may b...6 min read·bobblackburn.comApr 29, 2024