Put Blob REST API Content-Length error

Jez Walters 20

I'm trying to write the XML response obtained from a REST API GET request to a file in Azure Blob Storage, using Azure Data Factory.

I'd prefer to use the Copy activity to achieve this, but it doesn't support XML sinks. I've therefore been forced into using separate Web activities, to get the response and then write it to Blob storage.

I have 3 activities in my pipeline:

A Web activity, to issue my REST API GET request
A Set Variable activity, to assign the response obtained from step 1 to the variable called "Body"
A Web activity, to issue a REST API PUT request (using the "Body" variable from step 2)

This approach works fine for small responses but when I try doing the same for larger responses (around 1MByte), I get the following error:

Error calling the endpoint 'https://mystorageaccount.blob.core.windows.net'. Response status code: 'NA - Unknown'. More details: Exception message: 'NA - Unknown [ClientSideException] Bytes to be written to the stream exceed the Content-Length bytes size specified.'. Request didn't reach the server from the client. This could happen because of an underlying issue such as network connectivity, a DNS failure, a server certificate validation or a timeout

What am I doing wrong, or is there a better way of achieving my goal (but still using Azure Data Factory)?

Jez Walters 20

I've created the Web activity for the REST API PUT request to write to Blob storage (as described in https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob) as follows:

URL: https://mystorageaccount.blob.core.windows.net/mycontainer/mydirectory/myfile.xml
Method: PUT
Body: @variables("Body")
Authentication: System Assigned Managed Identity
Resource: https://storage.azure.com

I've set the PUT request Headers as follows:

Name	Value
x-ms-date	@formatDateTime(utcNow(), 'ddd, dd MMM yyyy HH:mm:ss G\MT')
x-ms-version	2024-05-04
Content-Length	@length(variables('Body'))
x-ms-blob-type	BlockBlob
Authorization	authentication-managed-identityContent-Typetext/xml

This issue may or may not relate to the response data itself, although I can't see anything particularly different for the larger responses - apart from their size.

Sumarigo-MSFT 44,251 Reputation points Microsoft Employee

2024-05-08T09:51:51.1333333+00:00

@Jez Walters It seems like you are encountering an issue with the size of the response data when trying to write it to Azure Blob Storage using Azure Data Factory. The error message you received indicates that the bytes to be written to the stream exceed the Content-Length bytes size specified.

One possible solution to this issue is to use the Azure Blob Storage connector in Azure Data Factory instead of the Web activity. The Blob Storage connector supports XML sinks and can write data directly to Azure Blob Storage without the need for a separate Web activity.

To use the Blob Storage connector, you can create a new pipeline with a Copy activity. In the Copy activity, you can specify the REST API GET request as the source and the Azure Blob Storage as the sink. You can then map the response data to the appropriate fields in the Blob Storage sink.

If you prefer to continue using the Web activity, you can try increasing the timeout value for the activity to allow more time for the data to be written to Azure Blob Storage. You can also try splitting the response data into smaller chunks and writing them to Azure Blob Storage in batches to avoid hitting the Content-Length limit.

I hope this helps! Let me know if you have any further questions.

6 answers

Jez Walters 20 Reputation points

2024-05-09T13:52:16.3366667+00:00
TestPipeline.txt

For anyone wanting to reproduce the problem, the above test pipeline (I've changed the extension changed from ".json" to ".txt") contains 3 activities:

A Web activity to issue a REST API GET request, to get makes of cars (using a publicly available anonymous endpoint)

A Set Variable activity to extract the XML response from step 1 (into a variable called "Body")

A Web activity to issue a REST API PUT request, to write the XML response to Blob storage (using the variable called "Body" from step 2)

For privacy/security reasons, I've amended the URL in the Web activity in step 3 (to use the storage account "mystorageaccount", and the container "mycontainer").
Please sign in to rate this answer.
Anand Prakash Yadav 6,940 Reputation points Microsoft Vendor

2024-05-14T11:37:25.6666667+00:00

Taking offline
Sign in to comment

Share via

Put Blob REST API Content-Length error

6 answers