Azure Data Factory: Store SQL Execution Logs During Script Activity

This article describes how to store SQL execution logs while using Script activity in ADF.

Image credit — https://www.checkatrade.com/blog/ideas/log-store-ideas/

In a previous article of mine, I introduced the Script activity in ADF. As a continuation to that, we can also store the logs of our SQL scripts that we execute with the help of Script activity.

Let’s assume we need to use the script activity to perform the following operations and log their execution state:

✔️Create a table
✔️ Insert values into that table
✔️ Extract count from that table
✔️ Drop the table

These are the options available in Script activity for logs —

Activity output — Use this option when you need to see the output of the script activity within the ADF runtime environment.
One thing to note here is that output size in this mode is restricted to 2MB or 5000 rows.

External storage — Use this option when you want to store the logs in your Azure Storage Account and you don’t want to run into any output size constraint. When you click this radio button, ADF will ask about your Azure storage container where the logs need to be stored.