How To Successfully Fail a Data Pipeline in ADF

Use Case:
To be able to identify whether an excel file exists in storage container or not. If the file doesn’t exist, fail the data pipeline with an error message.

Requirement:
As a Data engineer, I need to build a data pipeline that checks whether a file has been uploaded everyday in a folder inside my storage account container or not. If the pipeline does not find a file with the name Azure File 1.xlsx among the list of files, it should fail the pipeline and log a custom error message.

ADLSG2 container
(ADLSG2 container)

Step 1: Create a Customized Binary Dataset

ADF parameterized binary dataset
(ADF parameterized binary dataset)

Step 2: Bring Get Metadata1 Activity to ADF Canvas

ADF Get metadata activity
(ADF Get metadata activity)
ADF Get metadata activity output
(ADF Get metadata activity output)

Step 3: Drag a ForEach Activity To Connect With Get Metadata1 Activity

(ADF create a new variable)
ADF for each activity on Get Metadata activity
(ADF for each activity on Get Metadata activity)
ADF append variable
(ADF append variable)

Step 4: Check If File Exists And Fail Pipeline If File Not Found

ADF if condition
(ADF if condition)
ADF fail activity
(ADF fail activity)
ADF how to fail a data pipeline
(ADF how to fail a data pipeline)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store