Member-only story
ADF Pipeline — Want The Most Recent File To Copy? Here’s How.
This article helps to explain how to use ADF to copy latest files from a source to destination.
When you use ADF’s copy activity to copy files from one location to another, you can make use of Filter by last modified property to filter for most recent files. Here’s how to do so.
Test Set Up
I have a folder in my ADLSG2 (Azure Data Lake Gen 2) container with multiple files, with two files in particular — Test File 1.csv and Test File 2.csv — being the most recent file.
In ADF canvas, I will drag and drop the copy data activity and connect my source to the above folder. The sink location (also known as destination) will be another folder in the same ADLSG2 container.
Use Case 1️⃣: Get All Files Starting From The Last 24 Hours To Present
In the source dataset configuration, I will use the Filter by last modified property and set the start time (UTC) setting to the last 24 hours. To do so, click the text box next to the property and then Add dynamic content.
