Azure Data Factory: Step-by-Step Process to Read Multiple Pages of An API & Export to Different CSV Files

This article shows how we can use ADF REST connector to read the multiple pages in an API response and export to CSV files.

If you have a scenario where the API you are working with sends data over multiple pages, ADF REST connector can help loop over those pages and copy their data to CSV files.

In my last article, I showed two methods to read an API response using ADF and export the response to a CSV file.

ADF — Here Are 2 Ways to Extract Data From API and Export to CSV

For this use case, I will use the open source API —

https://api.artic.edu/api/v1/artworks

As we can see, the API headers contains information about the total number of pages that are present with the API and the URL for the next page of API response.

You can check how the data looks on the next page of this API by using a filter in the URL to go to the second page as shown below —

https://api.artic.edu/api/v1/artworks?page=2

Since the total number of pages is 10,124, I don’t need to read data from all those…