How to get list of all table names from table storage using Web activity in ADF

Hemanath Goud Kandula 1

In Azure Table Storage we have more than 1000 tables and to get the table name I am using the Rest API of Table Storage and SAS Url token with Web activity in ADF. I cannot get all table names in response and in response, we are having `x-ms-continuation-NextTableNamevalue at the end. From the official documentation page of Azure, I found that in web activity we can only get 1000 table names, and if we need to get the remaining table names we need to use `x-ms-continuation-NextTableNamevalue. I don't know how to use continuation-token to get the remaining tables. Can someone help me how I can able to get all table names at once in the web activity?

phemanth 6,885 Reputation points Microsoft Vendor

2024-02-12T14:50:34.1+00:00
@Hemanath Goud Kandula
Thanks for reaching out to Microsoft Q&A.

The current Endpoint Pipeline executes a Query By Factory that supports pagination through the Request Body, where the continuation token is included as part of the request body rather than in Query Parameters or headers. This resulted in repeated data for the first page, as the Continuation token was not honored when passed as headers.

Regrettably, pagination for this specific endpoint (Pipeline Runs - Query By Factory) cannot be achieved through headers, query parameters, or absolute URI, which are the only supported forms of pagination for the copy activity. Given this limitation, a potential workaround could involve the following steps:

Initialize a variable, ContinuationToken, as blank.

Execute the Copy activity with the ContinuationToken value using merge behavior.

Perform a Web activity.

Set the variable ContinuationToken with the continuation token obtained from the Web activity.

Repeat steps 2-4 using the Until activity until the ContinuationToken is returned as blank.

The ultimate objective is to carry out the copy activity iteratively with different request bodies to obtain subsequent pages of data until the ContinuationToken is no longer provided. This outline serves as a demonstration, and adjustments can be made to optimize the logic based on specific requirements or preferences.

For more information about continuation tokens, see

Query Timeout and Pagination PipeLine runs - Query By Factory

Hope this helps. Do let us know if you any further queries.
Hemanath Goud Kandula 1 Reputation point

2024-02-12T15:55:01.6433333+00:00

Hi @phemanth What will be the expression value for the Until activity and source and sink parameters for the Copy activity in the ADF pipeline? can you demonstrate the example in gif or images?
phemanth 6,885 Reputation points Microsoft Vendor

2024-02-13T11:40:17.26+00:00

@Hemanath Goud Kandula
please go through this link thoroughly: https://www.appsloveworld.com/azure-data-factory/8/an-example-of-continuationtoken-with-pipeline-runs-query-by-factory
Hemanath Goud Kandula 1 Reputation point

2024-02-13T14:17:39.4866667+00:00

Hi @phemanth Can you show me these in the ADF pipeline because I am not using any coding language to implement the scenario?
phemanth 6,885 Reputation points Microsoft Vendor

2024-02-14T09:08:14.7466667+00:00

@Hemanath Goud Kandula
you can use the Web Activity to call the REST API of Azure Table Storage. However, handling continuation tokens directly in ADF without using any code might be challenging. Here’s a workaround using ADF’s Until activity to loop through the pages:

Web Activity: Make the initial API call without a continuation token. This will retrieve the first page of results (up to 1000 tables).

Set Variable Activity: Extract the continuation token from the response of the Web Activity and store it in a variable.

Until Activity: Create an Until activity that continues as long as the continuation token variable is not null. Inside this Until activity:

Web Activity: Make the API call again, this time including the continuation token stored in the variable.

Append Variable Activity: Append the new tables to a list variable.

Set Variable Activity: Update the continuation token variable with the new continuation token from the response. After the Until activity completes, you will have a list of all table names in the list variable.

1 answer

Amira Bedhiafi 17,201 Reputation points

2024-02-10T16:07:20.45+00:00

In Azure Table Storage, when querying a service to list tables (or other entities), the service may paginate the results if there are a large number of items. This is exactly what you're experiencing with the 1000 table limit and the use of continuation tokens (x-ms-continuation-NextTableName). Unfortunately, there's no direct way to get all table names in a single request if you have more than 1000 tables. However, you can achieve this by making multiple requests, using the continuation token from each response to get the next set of tables. https://stackoverflow.com/questions/39965591/get-list-of-all-tables-in-microsoft-azure-tablestorage
Please sign in to rate this answer.

1 person found this answer helpful.
Hemanath Goud Kandula 1 Reputation point

2024-02-11T14:35:59.0633333+00:00

Hi @Amira Bedhiafi I understand your answer and I would like to achieve this in ADF pipeline in web activity. Is there a way to loop the response by multiple requests with continuation token and the response can be parse to for each loop for processing the tables.
Sign in to comment

Share via

How to get list of all table names from table storage using Web activity in ADF

1 answer