Member-only story
This article demonstrates how to use the Fail activity in Azure Data Factory to intentionally fail a data pipeline.
In this article, we will build a data pipeline that uses fail activity in ADF to terminate a data pipeline and log a custom error message.
Use Case:
To be able to identify whether an excel file exists in storage container or not. If the file doesn’t exist, fail the data pipeline with an error message.Requirement:
As a Data engineer, I need to build a data pipeline that checks whether a file has been uploaded everyday in a folder inside my storage account container or not. If the pipeline does not find a file with the name Azure File 1.xlsx among the list of files, it should fail the pipeline and log a custom error message.
👉In my storage account container — dev I have created a folder called temp. Inside that folder, I have placed two files — Azure File 1.xlsx and Azure File 2.xlsx.
Step 1: Create a Customized Binary Dataset
As we are working with excel files, let’s create a new generic binary dataset in the following way —
New dataset > Azure Data Lake Storage Gen2 > Binary > Linked Service to Connect to ADLSG2. Let’s parameterize the connection property of our new…