This article explains how to rename multiple files with ADF during copy from source to sink.

In this article, we will rename multiple files while copying them between different containers in our Azure storage account.

Use Case:
To be able to rename multiple .csv files using Azure Data Factory while copying…

This article explains how to send parameters to Databricks from ADF and receive output from Databricks in ADF.

Quite often as a Data Engineer, I need to use Databricks as part of my Azure Data Factory Data Pipeline. This involves configuring the pipeline’s ability to send parameters to Databricks and…

This article explains how to call or trigger Azure Data Factory from Logic Apps and pass parameters to it.

As a Data Engineer, I am very comfortable in calling or triggering Logic Apps from Azure Data Factory. So, it was interesting for me to find out that in a scenario…

This article explains how to copy data from SharePoint to SQL in a delta load manner

data pipeline; power automate; sharepoint to sql

This article will guide you through steps needed to implement delta logic to copy data from SharePoint to SQL with Power Automate. Before we start, I assume that you, my dear reader, know how to build a simple version of this data pipeline. If not, don’t worry. I got you…

This article explains how to store dynamic filenames with ADF’s Mapping Data Flow.

As a Data Engineer, I find capturing filenames is as important of an information as ensuring data quality. Numerous projects that I worked upon needed me to store the filenames in a separate attribute just so I can have additional information about which file supplied a particular set of data.

This article demonstrates how ADF can be used to pause and resume a Dedicated SQL pool (formerly SQL DW).

What do you do when your team does not need its dedicated SQL Pool? Do you keep it running? If yes, I will show you an interesting feature in Azure where…

This article explains how to use switch statement in ADF pipeline instead of multiple if-else activities.

As a Data Engineer, I have had many use cases wherein I needed to use multiple if-else conditions as part of my data pipeline. What I found was that ADF doesn’t support nested if-else…

This article explains how to handle dynamic column names using adf mapping data flow

As a Data Engineer, working with source file datasets often comes with a painful reality- changing column names. So, how can you ensure that the mapping between source and destination stays intact even if source column names keep changing? The answer lies in Mapping Data flow.

Let’s say you have…

This article explains how to use Logic Apps to copy mail attachments to Azure blob storage.

Azure Logic Apps is a powerful tool that you can make use of to copy email attachment to Azure Blob Storage. This articles shows the steps needed to achieve the same.

Task:
You receive daily emails (outlook account) with excel spreadsheet as an attachment. You are required to copy the spreadsheet…

This article explains how to run a python script in Azure Automation.

Let’s say you work in a team of 10 Data Engineers. Each engineer has developed a pipeline for a specific project that involves bringing data into temporary containers in Blob Storage. The containers are created at runtime. When the pipelines execute, the Project Manager needs a list of all such…

Tech Zero

I work as a Data Engineer which means on any day I'm involved in creating, maintaining and optimizing data infrastructure and data stores for data pipelines.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store