How to separate logs receiving on syslog port 514 to separate table during ingestion and avoid duplication.

Disha Bodade 65 Reputation points
2024-05-03T13:16:08.3933333+00:00

Hi Team,

I have centralized log forwarders setup which collects logs on 514 port from different application, I want to send those logs to separate table by filtering them at ingestion time.

Currently all logs are going to syslog using default DCR rule, also that DCR rule is not having any option to customize and add transformation scripts.

I don't want to install AMA agents on at respective application.

Please let me know any suggestions or solutions for it.

Thanks,

Disha

Microsoft Sentinel
Microsoft Sentinel
A scalable, cloud-native solution for security information event management and security orchestration automated response. Previously known as Azure Sentinel.
997 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Givary-MSFT 28,571 Reputation points Microsoft Employee
    2024-05-08T09:09:56.39+00:00

    @Disha Bodade Apologies for the delay in responding to this post, check if the below approach helps to achieve your requirement

    One way to achieve this is by creating a new custom log table in your Log Analytics workspace and then creating a new Data Collection Rule (DCR) to send the logs to this table. You can then modify the KQL transformation for your custom log to include additional filtering and apply the changes to your DCR.

    To create a new custom log table, you can follow the steps mentioned in the "Add a custom log table" section of the following document: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal

    Once you have created the custom log table, you can create a new DCR to send the logs to this table. You can follow the steps mentioned in the "Create a new data collection rule" section of the same document mentioned above.

    After you have created the DCR, you can modify the KQL transformation for your custom log to include additional filtering. You can follow the steps mentioned in the "Modify the KQL transformation for your custom log" section of the following document - https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview?tabs=portal

    Once you have modified the KQL transformation, you can apply the changes to your DCR. You can follow the steps mentioned in the "Apply changes to your data collection rule" section of the same document mentioned above.

    I hope this helps! Let me know if you have any further questions.


  2. Andrew Blumhardt 9,576 Reputation points Microsoft Employee
    2024-05-08T21:35:03.73+00:00

    I think you should ignore the instructions. No need to mess with the timestamps or raw data.

    What you need is to filter out the data from the target app. Something like this. Whatever you can find in the logs to identify the target application.

    source
    | where SyslogMessage contains "I0506"