Alert Logic Log Collector for Microsoft Azure Event Hubs

Microsoft Azure Event Hubs is a data streaming platform and event ingestion service that can receive and process millions of events per second. Alert Logic allows you to configure Event Hubs to collect your Azure logs and forward them to Alert Logic.

To grant Alert Logic permission to access Event Hubs for log collection, you must have:

  • A Microsoft Azure account with administrative privileges
  • An Alert Logic user account with administrative privileges

Alert Logic recommends that you create Azure deployments in the Alert Logic console for each Azure subscription you want to use to collect logs from Event Hubs.

Create an Alert Logic access key

Access keys store your customer information to allow integration of data from your Microsoft Azure account to Alert Logic integrations and APIs. You can create an access key through the Alert Logic console. For more information, see Create and Manage Alert Logic Access Keys .

Download and deploy the ARM template

The Azure Resource Manager (ARM) is the Azure API that allows you to use JSON-formatted ARM templates to deploy resources. You can deploy the ARM template configured to create a new Event Hub along with the log collector, or you can deploy the ARM template configured to subscribe the collector to an existing Event Hub.

Deploy the custom Azure Resource Manager template for a new Event Hub

Alert Logic provides a custom ARM template that you must configure to allow communication between Azure and Alert Logic.

To deploy the ARM template:

  1. Access the ARM template page in Azure.
    -or-
    Access the Premium plan ARM template page in Azure, which will deploy an Azure Functions Premium plan with virtual network integration and private endpoints for accessing the storage account of the Azure Functions app. Note that deploying a with the Premium plan template will incur additional costs in your Azure account. For more information, refer to the Azure Functions Premium plan overview.
  2. Provide the following template parameters:
    • Application Name—Type the name of the log source to appear in the Alert Logic console.
    • Alert Logic Access Key ID—Type the access key ID generated when you created the Alert Logic access key.
    • Alert Logic Secret Key—Type the secret key generated when you created the Alert Logic access key.
    • Alert Logic API endpoint—Do not change the default parameter value (api.global-services.global.alertlogic.com).
    • Alert Logic Data Residency—Do not change the default parameter value (default).
    • Event Hub Resource Group—Leave parameter value blank.
    • Event Hub Connection String—Leave parameter value blank.
    • Event Hub Namespace—Leave parameter value blank.
    • Event Hub Name—Type the name of the existing Event Hub.
      The value defaults to insight-operational-logs. Make sure you stream logs to the insight-operational-logs Event Hub Name (or the name you configured). This Event Hub is created automatically by Azure when a subscription Log Profile is integrated with Event Hub through the Azure Monitor service. For more information, follow the Stream the Azure Activity Log to Event Hubs guide.
    • Event Hub Max Throughput Units—(Optional) The maximum number of throughput units for the Event Hub.
    • Event Hub Auto Inflate Enabled—(Optional) Whether auto-inflate is enabled for the Event Hub.
    • Event Hub Partition Count—(Optional) The number of partitions for the Event Hub.
    • Event Hub Retention Days—(Optional) The number of days to retain data in the Event Hub.
    • Event Hub Consumer Group—Do not change the default parameter value ($Default).
    • Event Hub Filter JSON—(Optional) If you want to filter messages and use JSON format to define the filter, type a filter in JSON format (for example, {resultType":"Success"}). Only messages that contain the specified property are collected. If both JSON and regex filter values are provided, logs are collected based on both the values. If you do not want to filter messages, leave this blank.
    • Event Hub Filter Regex—(Optional) If you want to filter messages and use regex format to define the filter, type a filter in regex format (for example, /*.Policy or "Policy"). Only messages that contain the specified property are collected. If both JSON and regex filter values are provided, logs are collected based on both the values. If you do not want to filter messages, leave this blank.
    • Enable Application Insights—(Optional) Enable or disable Application Insights for monitoring invocation logs. Default value is No. For more information on monitoring Azure functions, refer to Microsoft's guide.
  3. Select I agree to the terms and conditions stated above.
  4. Click Purchase.

The new Event Hub you created with this ARM template uses the following event hub scaling parameters:

  • Maximum throughput units: 20
  • Auto-inflate: Enabled
  • Partitions: 32
  • Data retention period: 7 days
The wrong values configured for eventhub Maximum throughput units, Auto-inflate Partitions, and Data retention period causes data loss which can't be recovered. For more information, refer to the guides for Scaling with Event Hubs and Features and terminology in Azure Event Hubs.

If you want to use other hub scaling parameter values, you can edit the template or contact Alert Logic for assistance.

To use other hub scaling parameter values:

  1. On the Azure Custom deployment page, click Edit template, and then make the appropriate changes to the following lines:
    • "newEhubMaxThroughputUnits": xx,
    • "newEhubPartitionCount": xx,
    • "newEhubRetentionDays": xx
  2. Click Save.

Adjust firewall rules in Event Hub

You must configure the firewall configuration for Event Hub to receive messages.

To adjust the firewall:

  1. In the new Event Hub you created, on the left panel, under Settings, click Networking.
  2. In the Public Access tab, for Public network access, select Selected networks.
  3. In the Firewall section, add your required IP address.
  4. Select Yes to Allow trusted Microsoft services to bypass this firewall.
  5. Click Save.

Deploy the custom Azure Resource Manager template for an existing Event Hub

If you want to deploy the ARM template for an Event Hub that you already configured, you must provide additional parameter values when you deploy the Alert Logic custom ARM template.

To deploy the ARM template:

  1. Access the ARM template page in Azure.
    -or-
    Access the Premium plan ARM template page in Azure, which will deploy an Azure Functions Premium plan with virtual network integration and private endpoints for accessing the storage account of the Azure Functions app. Note that deploying a with the Premium plan template will incur additional costs in your Azure account. For more information, refer to the Azure Functions Premium plan overview.
  2. Provide the following required template parameters:
    • Application Name—Type the name of the log source to appear in the Alert Logic console.
    • Alert Logic Access Key ID—Type the access key ID generated when you created the Alert Logic access key.
    • Alert Logic Secret Key—Type the secret key generated when you created the Alert Logic access key.
    • Alert Logic API endpoint—Do not change the default parameter value (api.global-services.global.alertlogic.com).
    • Alert Logic Data Residency—Do not change the default parameter value (default).
    • Event Hub Resource Group—Type the resource group for the existing Event Hub.
    • Event Hub Connection String—Type the connection string for the existing Event Hub.
      • For information about using Azure PowerShell and Azure CLI to get the connection string, see the Microsoft document, Get an Event Hubs connection string.
    • Event Hub Namespace—Type the namespace for the existing Event Hub.
    • Event Hub Name—Type the name of the existing Event Hub.
    • Event Hub Max Throughput Units—(Optional) The maximum number of throughput units for the Event Hub.
    • Event Hub Auto Inflate Enabled—(Optional) Whether auto-inflate is enabled for the Event Hub.
    • Event Hub Partition Count—(Optional) The number of partitions for the Event Hub.
    • Event Hub Retention Days—(Optional) The number of days to retain data in the Event Hub.
    • Event Hub Consumer Group—Type the name of the consumer group of the existing Event Hub.
      If the Event Hub has other consumers, create a separate consumer group for the Alert Logic collector and use that name here. For more information, see the Microsoft document, Consumer Groups.
  1. Select I agree to the terms and conditions stated above.
  2. Click Purchase.

You ensure the firewall is configured for Event Hub to receive messages. To adjust the firewall, see Adjust firewall rules in Event Hub.

Verify the deployment

After you deploy the template, Alert Logic recommends you perform the following steps to verify the template deployed successfully:

  1. In the Alert Logic console, select the Azure deployment for the Azure subscription where you deployed this log collector.
  2. Click CONFIGURE LOG SOURCES, and then filter the list by Push (Office 365, EventHub) collection method.
  3. Verify that the new Event Hubs log source with the name you provided during deployment appears with the source status of "OK."

If the Event Hub log source does not appear, you can:

  • Wait approximately 30 minutes, and see if the new Event Hub log source appears.
  • Remove the Azure resource group that contains the log source, and deploy the ARM template again.
  • Contact Alert Logic Technical Support at:
    • US: (877) 484-8383 (Option 1)
    • UK: +44 (0) 203 011 5533 (Option 1)

Integrate with Azure Event Hubs

The following links contain instructions to help you integrate different Azure services with Event Hubs.