Member-only story

How To Successfully Fail a Data Pipeline in ADF

Tech Zero
4 min readNov 15, 2021

--

This article demonstrates how to use the Fail activity in Azure Data Factory to intentionally fail a data pipeline.

In this article, we will build a data pipeline that uses fail activity in ADF to terminate a data pipeline and log a custom error message.

Use Case:
To be able to identify whether an excel file exists in storage container or not. If the file doesn’t exist, fail the data pipeline with an error message.

Requirement:
As a Data engineer, I need to build a data pipeline that checks whether a file has been uploaded everyday in a folder inside my storage account container or not. If the pipeline does not find a file with the name Azure File 1.xlsx among the list of files, it should fail the pipeline and log a custom error message.

👉In my storage account container — dev I have created a folder called temp. Inside that folder, I have placed two files — Azure File 1.xlsx and Azure File 2.xlsx.

ADLSG2 container
(ADLSG2 container)

Step 1: Create a Customized Binary Dataset

As we are working with excel files, let’s create a new generic binary dataset in the following way —

New dataset > Azure Data Lake Storage Gen2 > Binary > Linked Service to Connect to ADLSG2. Let’s parameterize the connection property of our new…

--

--

Tech Zero
Tech Zero

Written by Tech Zero

Product Manager, Data & Governance | Azure, Databricks and Snowflake stack | Here to share my knowledge with everyone

No responses yet