Share via

How do I copy data from a public sharepoint URL to Microsoft Blob Storage?

Namrata Narain 0 Reputation points
2024-04-08T16:28:15.86+00:00

Hi everyone!

I am new to using Azure services. My co-author at Stanford has uploaded several files (in csv format) to a public link. This link starts as: office365stanford-my.sharepoint.com/personal

It has 40 files, ranging from a few 100 MB to 100 GB. I don't want to have to download the files to my local machine to transfer it to Blob. How can I do this directly?

I tried to set up Azure Data Factory but the pipleline is failing with the following error:

ErrorCode=HttpRequestFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with client error, status code 403 Forbidden, please check your activity settings. If you configured a baseUrl that includes path, please make sure it ends with '/'.

Request URL: https://office365stanford-my.sharepoint.com/personal/gxzheng_stanford_edu/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fgxzheng_stanford_edu%2FDocuments%2FCNpatents%2Ffull_text_en&ga=1/.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden.,Source=System,'

Note, I have already set up Microsoft Blob and I'm able to use AzCopy to transfer my local files into it.

Thanks for all your help!!

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,468 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,691 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,793 questions
0 comments

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Marcin Policht 12,940 Reputation points MVP
    2024-04-08T16:43:23.41+00:00

    You can use Copy Blob (programmatically) - more at https://stackoverflow.com/questions/45003502/save-an-image-from-url-to-azure-storage-directly?noredirect=1&lq=1 or use tools such as this one https://docs.fineuploader.com/branch/master/features/azure.html

    If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

    hth

    Marcin

    0 comments
  2. KarishmaTiwari-MSFT 18,612 Reputation points Microsoft Employee
    2024-04-25T07:07:02.74+00:00

    Here are some alternative methods to transfer your files directly:

    • Azure Logic Apps: You can create an Azure Logic App that automates the file transfer from SharePoint to Azure Blob Storage.
    • Power Automate: Microsoft Power Automate can also be used to copy files from SharePoint to Azure Blob Storage, though it may have limitations on the file size.
    • Server-to-Server Transfer: Utilize AzCopy’s server-to-server transfer capabilities to move files directly between storage servers.

    Additional Reading: https://learn.microsoft.com/en-us/answers/questions/1513111/how-to-copy-folders-and-files-from-a-sharepoint-si

    0 comments
  3. JohnyBenz 301 Reputation points
    2024-05-06T17:43:10.1166667+00:00

    Unfortunately, due to security restrictions, directly copying files from a public SharePoint link to Azure Blob Storage using tools like AzCopy or Azure Data Factory (ADF) isn't feasible. Public SharePoint links typically provide read-only access, and these tools require elevated permissions for data transfer. As an effective method, I would suggest using third-party cloud migration tools that offer functionalities to transfer data between SharePoint and Azure Blob Storage like Gs Richcopy360 and CloudFuze. However, these tools often require additional costs and thorough security evaluations before use. It's crucial to understand their data access practices. Goodluck

    0 comments