Azure Data Factory Pagination Rules

Today we are going to talk about Pagination Rules section in Copy Data Activity. Pagination rules comes handy when dealing with JSON data, specially when trying to extract JSON data from an API.

As the data from the API normally will be extracted in big chunks of data, the extraction of the JSON data will be splitted into different pages. The way to configure everything related to how much data you want per page or guide azure data factory when to stop searching for more data is explained below.

Pagination Section

Find the pagination section within the Copy Data Activity block.

Example

We will perform the two most basic examples with two paremeters:

Suppose that we have JSON data comming in 5 pages within “results”:[]. 500 records in total - 100 records per page. Generally, the number of records per page will be specified in the API endpoint of the Linked Service you will create.

EndCondition

In this parameter, we specify in Azure Data Factory to stop reading once “results”:[] will not have more data.

Suppose that 5 pages are being received within “results”:[] as shown in the screenshot:

But once “results”:[] will be empty in the 6th page(no more data to extract), we instruct the activity to stop reading. This will avoid running the pipeline in an infinite loop as the activity knows when to stop.

AbsoluteUrl

In this parameter we are specifying where to look for the next page. For this example we receive 5 pages. But how the activity knows where to look for the next page? In our case it comes under “links” :{[“next”]}, but it depends on the API. You need to check on the JSON content extracted from the API used.

For a practical demonstration of what was said check the video materials below.

Youtube Materials