Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,698 questions
0 answers
How to Run a Python ETL Script in Azure Data Factory and Choose the Best Approach?
I'm currently working on an ETL process where I need to run a Python script within Azure Data Factory (ADF). The script involves data extraction, transformation, and loading (ETL) tasks. I’m aware that ADF supports various ways to execute custom code,…
asked 2024-05-20T11:30:14.8233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 4%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAI%3C/text%3E%3C/svg%3E)
Aadhil Imam
0
Reputation points
asked 2024-05-20T11:30:14.8233333+00:00
Aadhil Imam
0
Reputation points
2 answers
Need help in understanding times in ADF's tumbling window trigger
Hi Team, I'm confused by many times in tumbling window trigger. We have at least 4 times for tumbling window trigger. @trigger().outputs.windowStartTime @trigger().outputs.windowEndTime @trigger().scheduledTime @trigger().startTime Besides, there…
asked 2024-05-20T10:01:15.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 18%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZE%3C/text%3E%3C/svg%3E)
Zhengxiao (Edward) Wang
0
Reputation points Microsoft Employee
answered 2024-05-20T11:12:31.0366667+00:00
Amira Bedhiafi
16,071
Reputation points
2 answers
In ADF, All Linked Services not showing up in Git Mode from Live Mode
There are total 4 linked services created in my ADF and its on live mode. I make a git configuration and set the ADF to new branch along with providing publish branch. But only 2 linked services showed up leaving behind a LS with setup on azure sql db…
asked 2024-05-12T14:36:55.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 71%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Sudip Bala
20
Reputation points
commented 2024-05-20T10:53:48.4733333+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
0 answers
ADF Copy Activity failed with ORA-00928: missing SELECT keyword
ADF Copy activity is failing with following error. Its happening after copying certain amount of data. So don't think its issue with pipeline. Also same pipeline works in another environment(PROD/DEV) so mostly its data issue. How to identify what is…
asked 2024-05-13T04:46:02.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 96%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vivek, Anand
20
Reputation points
commented 2024-05-20T10:53:23.73+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
2 answers
Can't able to connect the VM installed postgresql from ADF
Hi Team, We are unable to access the PostgreSQL from the ADF linked server. it's a azure PostgreSQL flexible server, and we have enabled the "Allow public services" setting within Azure for this server. However, we are still unable to create a…
asked 2024-05-14T06:10:02.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 79%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Aravind
20
Reputation points
answered 2024-05-20T10:52:45.1733333+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
1 answer
Error when uploading CSV files to Azure blob storage using ManagedIdentity with C# and GraphQLAPI. "Total size of resourceFiles is using 147% of available space after processing."
Hello All, I have created a console app using C# where I try to fetch the data from GraphQLAPI and store the data in csv files. post that it would be uploaded to the Azure blob storage using ManagedIdentity. This complete code work fine in my local VS…
ASP.NET Core
ASP.NET Core
A set of technologies in the .NET Framework for building web applications and XML web services.
4,228 questions
ASP.NET
ASP.NET
A set of technologies in the .NET Framework for building web applications and XML web services.
3,303 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,698 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,360 questions
asked 2024-05-15T08:40:14.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 94%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Siddhesh Chavan
0
Reputation points
commented 2024-05-20T10:49:02.6166667+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
1 answer
how to retrieve the path to workspace_publish branch OR any branch in devops , use that path in azure powershell script to read file from that branch.
I want to retrieve path to the branch and folder inside it, basically i wan to read the jsons here and change the dev keyword with uat and save the file back to the sam folder in workspace_publish branch. I can change dev to uat, but am not able to…
asked 2024-05-15T12:58:55.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 24%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
Wasim
20
Reputation points
commented 2024-05-20T10:42:11.4466667+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
1 answer
CSV to XML conversion in databricks which have some blank values as well in csv
I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…
asked 2024-05-16T08:34:02.7366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 75%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Manoj
0
Reputation points
commented 2024-05-20T10:40:36.41+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
0 answers
can use a dataset for pipeline but can't use for dataflow in Azure Data Factory
Created a dataset for Azure SQL and it works fine in any pipelines of Azure Data Factory. However, it gives me the connection error from Data Flows - refer to the below. The Azure SQL is configured to be accessible from any Azure services. Spark…
asked 2024-05-20T05:53:34.5933333+00:00
Jerry Lee
0
Reputation points
commented 2024-05-20T10:17:55.1633333+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
0 answers
Using CRM as a source in the ADF Metadata-driven copy task results in Fetch XML syntax errors
I am using the Metadata-driven copy task in Azure Data Factory (ADF). The process works as expected when using SQL server as a source, but when I use CRM as a source, I get source query syntax errors when the pipeline executes. The problem seems to be…
asked 2024-05-20T06:54:23.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 88%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWB%3C/text%3E%3C/svg%3E)
Warren Briggs
20
Reputation points
commented 2024-05-20T10:15:35.94+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
0 answers
How can I get all files inside a drive irrespective of folder structure in ADF?
I want to copy files files from sharepoint drive which has lots of nested folders. Maximum hierarchy of folders is for 12 levels. Currently I'm using below endpoint in ADF's Web activity as it was mentioned in some of the articles that it provides every…
asked 2024-05-14T00:43:50.9066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 35%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMY%3C/text%3E%3C/svg%3E)
Mansi Yadav
0
Reputation points
edited a comment 2024-05-20T09:47:49.2033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,550
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
The Integration Runtime (Self-hosted) node has encountered an error during registration.
I am not able to register Self Hosted Integration Runtime on my Machine. This is the error I am getting : The Integration Runtime (Self-hosted) node has encountered an error during registration. Unable to determine the Integration Runtime (Self-hosted)…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 36%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE6%3C/text%3E%3C/svg%3E)
3 answers
How to copy the data from Amazon S3 to Azure SQL database incrementally.
Hello team, I wanted to wanted to copy data from Amazon S3(JSON format) to Azure SQL database. I wanted to learn the mechanism for incrementally updating the data from Amazon S3 to Azure SQL. Also, the environment has only SHIR deployed, so data flows…
asked 2024-05-01T20:06:01.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
commented 2024-05-20T09:00:54.8533333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
3 answers
Configuring Linked Service for Dev to Prod Movement in Azure Synapse Analytics
In Azure Synapse Analytics, I have a linked service of type Azure SQL Database Managed Instance, and I have two linked services, one for dev and one for prod. How can I move from dev to prod without changing the linked service manually? How can I make it…
asked 2024-04-15T07:21:19.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
105
Reputation points
answered 2024-05-20T08:54:48.05+00:00
Pinaki Ghatak
2,400
Reputation points Microsoft Employee
2 answers
Azure Data Factory - jobs not even running?
I have a persistent issue with ADF since Friday. Occasionally our pipelines will show "in progress" however the activities just wont move. Not even in a queued state, they just won't start at all. Right now we have a pipeline that has been…
asked 2024-04-15T10:04:23.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 61%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJD%3C/text%3E%3C/svg%3E)
Jordan Dwyer
96
Reputation points
answered 2024-05-20T08:51:15.2533333+00:00
Pinaki Ghatak
2,400
Reputation points Microsoft Employee
0 answers
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
How to use next link for pagination in adf (I have to use pagination for web activity)
Hi, I am trying to build up an adf pipeline to pull data from NetSuite. The result of the api call of NetSuite is like this in postman As we can see that this returns next link with the data and now we have to us this next link for pagination in adf. …
asked 2024-05-01T17:29:46.6833333+00:00
Amar Agnihotri
891
Reputation points
commented 2024-05-20T08:38:48.17+00:00
AnnuKumari-MSFT
31,561
Reputation points Microsoft Employee
4 answers
Authentication type missing from linked service in Azure Data Factory when
Hi, MS released a new connector for PostgresSQL in Linked Services of Azure Data Factory v2. We could create a new linked service in the development branch that worked ok. But when we deploy the linked service to the ADF (connected to Git), the…
asked 2024-04-15T12:18:05.4833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 66%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Ashish Ruparel
0
Reputation points
answered 2024-05-20T08:37:24.49+00:00
Pinaki Ghatak
2,400
Reputation points Microsoft Employee
1 answer
Azure Data Factory Oracle to Parquet datatype mismatch
Hi Team, I have build a dynamic pipeline to load the data from Oracle tables to datalake as Parquet files. Now, when I am trying to copy the data from source to sink through CopyActivity, then I am not able to convert the decimal datatype of the…
asked 2024-05-09T10:06:20.1066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vaibhav
105
Reputation points
commented 2024-05-20T08:28:38.4033333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
How to add Azure SQL Database as linked service to Azure Data Factory
I keep running into errors adding Azure SQL as a linked service to Data Factory. I'm new to Azure so please bear with me. Following all the online guides, I've done all of the following: On the SQL Server side: Under Microsoft Entra ID, enabled…
asked 2024-05-07T04:05:57.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 44%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
shellkyle
0
Reputation points
commented 2024-05-20T08:12:12.92+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee