How to compare two dates in ADF to see which is greater?
Hi there, I am using Azure Data Factory to compare two dates (one is stored in a string variable and one its utcnow) and I need to know when the utcnow is greater than the date in the stored string variable. Both are just dates no timestamps I looked at…
Data flow failing at Sink Step when pipeline is scheduled
I have Pipeline with Data flow debug where data is sinked into Data set. When run manually it runs fine and the parquet file is created. However when scheduled in Pipeline it gives following error. Pipeline runs fine only gives error on schedule…
Except Unique Assert Transformation in Dataflow
I am trying to use Asset transformation in dataflow to check the dups in source. I am not able to get the correct result when using parameters that will have single or multiple columns as primary keys (example shared below) when using below expression. …
Unable to create a cluster in Databricks getting quota exceeded error while everything is new
Azure Quota Exceeded Exception: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardDADSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location:…
Azure HTTP Linked service ADF Username only no password
I am trying to use Azure HTTP Linked services to connect an API which uses just a username and no password for basic auth. The password input is mandatory in Azure. Anything i can do?
Export JSON string but exclude key-value pairs with null/empty values
I'm attempting to output some data originally in csv format from ADF as a JSON string but I need to exclude any key-value pairs where the value is null or empty (e.g. "Report": []). First I am using an Aggregate transformation to convert 3…
Obvious behavior of dynamic mapping in ADF
I have a single copy activity that moves data from a source parquet file to an azure sql database. When I used auto mapping and tried to change the name of source or sink field it failed, however when I copy the json mapping generated and change one of…
Split column data into another column
Hi There, I want to extract data from a column , which has key-value pairs separated by commas, into respective columns using Azure Data Factory data flow. Example Data of Column containing key value…
ADF does not allow MI based auth for Azure table storage. Are there any alternatives for this?
I am trying to move away from key-based access for an azure table storage in ADF, but the only auth options I get are account key and SAS URI. Is there no way to use MI/SP for this? Blob storage has MI based auth but not table, why is this so?
How to migrate bigger repo to azure devops
Hello, I am migrating from GIT to Devops. My repo is quite huge and while doing git push I am getting error "HTTP 413 curl 22 the requested url retuned error: 413". I am using Https to do migration. SOmewhere I saw ssh can solve this issue.…
Default TImezone in the Dataflow
Hi Team, Is there anyway to change the default timezone in the datafactory? Currently, when I am creating a column(loadtime) with currentTimestamp() in derived column transformation in the dataflow, then I get the time in UTC. But, I want the default…
Issue with Copy Dataverse data into Azure SQL using Synapse Link Template
I am experiencing issues using ADF I am trying to use the template "Copy Dataverse data into Azure SQL using Synapse Link" to move f&o data from datalake to Azure…
Hubspot connector doesn't work for deal properties: 414 Request-URI Too Large
I am getting this error message when trying to copy data from Deal properties from Hubspot: Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000]…
All ADF dataflow stages disappear after publish
Hello, I made a source for my Dataflow and added several stages to the existing one. I published and Dataflow showed an error with a new source. I tried to delete it, but could not delete it. I published again and all my stages disappeared. It is not…
How to pass the correct audience when calling mssparkutils.credentials.getToken on Azure China (Mooncake) cloud?
I'm using Microsoft Spark Utilities (MSSparkUtils) with linked service to authenticate into Azure SQL using System Assigned Managed Identity (Synapse Workspace) on Azure China (Mooncake) cloud. However, when I call gettoken with the audience type…
How can I force a failure when testing an ADF Webhook?
I would like to try to force a failure when testing an ADF WebHook activity so it follows the failure path. Is there a way I can easily do this? I can try to swap the on success and on failure paths to test the path, but I'd really like to force an…
Delete the file from SharePoint location
Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…
How to add Auth Headers in REST Linked Service in Azure Data Factory (ADF) when using oAuth2 Client Credentials?
My API provider requires an API-Key additional header when using oAuth2 Client Credential to generate access token. In Azure DataFactory, when I create a REST linked service using OAuth2 Client Credential as authentication type, I also include Auth…
How to fix "Failure happened on 'Source' side A task was canceled" error
Data load from Salesforce object with 1million records fails with the below error in Azure Data Factory: Failure happened on 'Source' side. 'Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,' The data load…
ADF lost access to Dynamics CRM / Dataverse after MFA was turned on
Today my ADF pipelines started throwing errors: Operation on target Process order failed: ErrorCode=DynamicsFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to connect to Dynamics: Unable to Login to…