PinnedAzure Logic Apps: How To Trigger One Logic App From Another Logic AppThis article describes how azure logic app can be triggered from another logic app using HTTP POST Method. — Overview In this article, we will go over the steps needed to trigger a logic app from another logic app. A high-level architecture looks like below: The parent logic app uses HTTP POST to send a request to the child logic app. …Call Logic App From Othe4 min read
PinnedHow To Use Fuzzy Matching In Azure Data FactoryThis article explains how to use Fuzzy Matching in Azure Data Factory Data Flow. One of the most amazing features Microsoft could have unveiled has been released in its Data Flow capacity — Fuzzy Match. To give you a quick background of what a fuzzy match is, consider the below…Adf Fuzzy Matching4 min read
PinnedHow To Use Script Activity in Azure Data FactoryThis article demonstrates how to use script activity in ADF to execute SQL commands. — We’ll cover the following topics- ✅ Exploration of components of ADF script activity; ✅ Example of Use Cases on how to use script activity; 📢 a Tip Let’s explore the Script activity in ADF and its components. Right away, we can see that there are two radio buttons — Query…Adf Script Activity5 min read
PinnedAzure Data Factory — How To Upsert Data During CopyThis article explains how to upsert data using only copy activity in ADF. — Azure Data Factory recently released an extremely helpful feature to upsert data using copy activity. In this article, let’s explore how to utilize this feature in your next data pipeline. I will copy data from a sample csv spreadsheet that looks like below into my Azure Synapse Data Warehouse.Adf Copy Upsert4 min read
PinnedHow To Use Switch Statement in Azure Data Factory PipelineThis article explains how to use switch statement in ADF pipeline instead of multiple if-else activities. As a Data Engineer, I have had many use cases wherein I needed to use multiple if-else conditions as part of my data pipeline. What I found was that ADF doesn’t support nested if-else…Azure Data Factory3 min read
Jul 29Got Technical Skills As a Data Engineer? Good, Let’s Talk About Non-Technical SkillsData Engineering is a technical field. Yes, we all know that. You are expected to be well-versed with Data Modeling on RDBMS or No-SQL DBs, Advanced SQL, Python/Scala programming, Data Warehousing Modeling and Concepts and Cloud Technologies To Build Pipelines With. But, is that all there is? …Data Engineer Soft Skills4 min read
Jul 19What Are The Qualities Of a Smart Azure Data Factory Pipeline?This article describes my experiences and opinions about useful qualities that need to be included in a ADF pipeline. — Over the past many years of building data pipelines in Azure Data Factory (ADF from hereon) and improvising the pipelines built by ex-employees, I have witnessed a series of commonalities that separates pipelines built by experienced data engineers(DE from hereon) from those built by junior DEs.Adf Pipeline Features5 min read
Jul 18Azure Data Factory: Store SQL Execution Logs During Script ActivityThis article describes how to store SQL execution logs while using Script activity in ADF. In a previous article of mine, I introduced the Script activity in ADF. As a continuation to that, we can also store the logs of our SQL scripts that we execute with the help of…Adf Script Activity4 min read
Apr 22How to Connect Azure Data Lake To SnowflakeThis article explains how to establish a connection string between Azure Data lake to Snowflake Data warehouse. — Over the past many months, I have been working with Snowflake Data Warehouse. This has definitely been an amazing learning experience and an opportunity for me to see what an absolute beast Snowflake computing power can be. So, I figured why not share the same learning here. In this article…Adlsg 2 To Snowflake5 min read
Mar 7Azure Data Factory — Prevent Copy Failures With Retry FeatureThis article explains how to use the retry feature in ADF copy activity. When using Azure Data Factory’s Copy activity, you can encounter multiple failures. However, if any/none of those failures are due to the underlying logic but are triggered due to a connection reset, Java Native Interface error etc…Adf Copy Retry2 min read
