Alert Logic Log Collector for Microsoft Azure Event Hubs

Microsoft Azure Event Hubs is a data streaming platform and event ingestion service that can receive and process millions of events per second. Alert Logic allows you to configure Event Hubs to collect your Azure logs and forward them to Alert Logic.

To grant Alert Logic permission to access Event Hubs for log collection, you must have:

  • A Microsoft Azure account with administrative privileges
  • An Alert Logic user account with administrative privileges

Alert Logic recommends that you create Azure deployments in the Alert Logic console for each Azure subscription you want to use to collect logs from Event Hubs.

Create an Alert Logic access key

Access keys store your customer information to allow integration of data from your Microsoft Azure account to Alert Logic integrations and APIs. You can create an access key through the Alert Logic console. For more information, see Create and Manage Alert Logic Access Keys .

Download and deploy the ARM template

The Azure Resource Manager (ARM) is the Azure API that allows you to use JSON-formatted ARM templates to deploy resources. You can deploy the ARM template configured to create a new Event Hub along with the log collector, or you can deploy the ARM template configured to subscribe the collector to an existing Event Hub.

Deploy the custom Azure Resource Manager template for a new Event Hub

Alert Logic provides a custom ARM template that you must configure to allow communication between Azure and Alert Logic.

Two versions of the ARM template are offered for deploying the Event Hub collector: the standard ARM template and the Premium plan ARM template.

  • Standard ARM template: The standard ARM template uses the basic ‘Consumption’ hosting plan which does not support integration with virtual networks and private endpoints. Using this template therefore requires that the Azure Function App storage account has public network access enabled for all networks. The storage account in this case is reachable from the public internet, but remains inaccessible without proper authentication and authorization.
  • Premium plan ARM template: The Premium plan ARM template deploys the collector onto a Premium hosting plan which does support integration with virtual networks and private endpoints. Using this template therefore enables the Azure Function App storage account to restrict inbound connections to those originating from within a specific virtual network in your Azure subscription. The use of a Premium hosting plan will incur additional costs in your Azure account.

For more information about these plans, refer to the Azure Functions Premium plan overview.

To deploy the ARM template:

  1. Access the ARM template page in Azure, which deploys the standard 'Consumption' plan ARM template.
    -or-
    Access the Premium plan ARM template page in Azure, which will deploy an Azure Functions Premium plan, as described above.
  2. Provide the following template parameters:
    • Application Name—Type the name of the log source to appear in the Alert Logic console.
    • Alert Logic Access Key ID—Type the access key ID generated when you created the Alert Logic access key.
    • Alert Logic Secret Key—Type the secret key generated when you created the Alert Logic access key.
    • Alert Logic API endpoint—Do not change the default parameter value (api.global-services.global.alertlogic.com).
    • Alert Logic Data Residency—Do not change the default parameter value (default).
    • Event Hub Resource Group—Leave parameter value blank.
    • Event Hub Connection String—Leave parameter value blank.
    • Event Hub Namespace—Leave parameter value blank.
    • Event Hub Name—Type the name of the existing Event Hub.
      The value defaults to insight-operational-logs. Make sure you stream logs to the insight-operational-logs Event Hub Name (or the name you configured). This Event Hub is created automatically by Azure when a subscription Log Profile is integrated with Event Hub through the Azure Monitor service. For more information, follow the Stream the Azure Activity Log to Event Hubs guide.
    • Event Hub Max Throughput Units—(Optional) The maximum number of throughput units for the Event Hub.
    • Event Hub Auto Inflate Enabled—(Optional) Whether auto-inflate is enabled for the Event Hub.
    • Event Hub Partition Count—(Optional) The number of partitions for the Event Hub.
    • Event Hub Retention Days—(Optional) The number of days to retain data in the Event Hub.
    • Event Hub Consumer Group—Do not change the default parameter value ($Default).
    • Event Hub Filter JSON—(Optional) If you want to filter messages and use JSON format to define the filter, type a filter in JSON format (for example, {resultType":"Success"}). Only messages that contain the specified property are collected. If both JSON and regex filter values are provided, logs are collected based on both the values. If you do not want to filter messages, leave this blank.
    • Event Hub Filter Regex—(Optional) If you want to filter messages and use regex format to define the filter, type a filter in regex format (for example, /.*Policy or "Policy"). Only messages that contain the specified property are collected. If both JSON and regex filter values are provided, logs are collected based on both the values. If you do not want to filter messages, leave this blank.
    • Enable Application Insights—(Optional) Enable or disable Application Insights for monitoring invocation logs. Default value is No. For more information on monitoring Azure functions, refer to Microsoft's guide.
  3. If you are using the Premium plan ARM template page in Azure, provide the following additional required template parameters.
    • AppService Plan SKU Name—Select App service plan options for Elastic premium (EP1, EP2 or EP3) from the dropdown.
    • Vnet Name—Type the name of the virtual network for virtual network integration. The default value is [format('vnet-{0}', uniqueString(resourceGroup().id))].
    • Function Subnet Name—Type the name of the virtual network subnet to be associated with the Azure Function app. The default value is (al-function-subnet).
    • Private Endpoint Subnet Name—Type the name of the virtual network subnet used for allocating IP addresses for private endpoints. The default value is (al-privateendpoint-subnet).
    • Vnet Address Prefix—Type the IP address space used for the virtual network. The default value is (10.100.0.0/16).
    • Function Subnet Address Prefix—Type the IP address space used for the Azure Function integration subnet. The default value is (10.100.0.0/24).
    • Private Endpoint Subnet Address Prefix—Type the IP address space used for the private endpoints default value is (10.100.1.0/24).
  4. Select I agree to the terms and conditions stated above.
  5. Click Purchase.

The new Event Hub you created with this ARM template uses the following event hub scaling parameters:

  • Maximum throughput units: 20
  • Auto-inflate: Enabled
  • Partitions: 32
  • Data retention period: 7 days
The wrong values configured for eventhub Maximum throughput units, Auto-inflate Partitions, and Data retention period causes data loss which can't be recovered. For more information, refer to the guides for Scaling with Event Hubs and Features and terminology in Azure Event Hubs.

If you want to use other hub scaling parameter values, you can edit the template or contact Alert Logic for assistance.

To use other hub scaling parameter values:

  1. On the Azure Custom deployment page, click Edit template, and then make the appropriate changes to the following lines:
    • "newEhubMaxThroughputUnits": xx,
    • "newEhubPartitionCount": xx,
    • "newEhubRetentionDays": xx
  2. Click Save.

Adjust firewall rules in Event Hub

You must configure the firewall configuration for Event Hub to receive messages. For more information on adjusting firewall rules and IP addresses, refer to the Azure Event Hubs firewall rules documentation.

To adjust the firewall:

  1. In the new Event Hub you created, on the left panel, under Settings, click Networking.
  2. In the Public Access tab, you have the following options for Public network access:
    • If you select All Networks, traffic to the Event Hub namespace is not restricted to specific IP ranges, Virtual Networks (VNETs), or Private Endpoints, and the connection is secure. Skip to step 4.
    • If you select Selected Networks, you must add at least one IP firewall rule or specify a VNET in the Public network access settings. Proceed to step 3.
  3. If you selected Selected Networks, proceed to the Firewall section and add the required range of IP addresses that you want to allow to access the Event Hub.
  4. Select Yes to Allow trusted Microsoft services to bypass this firewall.
  5. Click Save.

Deploy the custom Azure Resource Manager template for an existing Event Hub

If you want to deploy the ARM template for an Event Hub that you already configured, you must provide additional parameter values when you deploy the Alert Logic custom ARM template.

Two versions of the ARM template are offered for deploying the Event Hub collector: the standard ARM template and the Premium plan ARM template.

  • Standard ARM template: The standard ARM template uses the basic ‘Consumption’ hosting plan which does not support integration with virtual networks and private endpoints. Using this template therefore requires that the Azure Function App storage account has public network access enabled for all networks. The storage account in this case is reachable from the public internet, but remains inaccessible without proper authentication and authorization.
  • Premium plan ARM template: The Premium plan ARM template deploys the collector onto a Premium hosting plan which does support integration with virtual networks and private endpoints. Using this template therefore enables the Azure Function App storage account to restrict inbound connections to those originating from within a specific virtual network in your Azure subscription. The use of a Premium hosting plan will incur additional costs in your Azure account.

For more information about these plans, refer to the Azure Functions Premium plan overview.

To deploy the ARM template:

  1. Access the ARM template page in Azure, which deploys the standard 'Consumption' plan ARM template.
    -or-
    Access the Premium plan ARM template page in Azure, which will deploy an Azure Functions Premium plan, as described above.
  2. Provide the following required template parameters:
    • Application Name—Type the name of the log source to appear in the Alert Logic console.
    • Alert Logic Access Key ID—Type the access key ID generated when you created the Alert Logic access key.
    • Alert Logic Secret Key—Type the secret key generated when you created the Alert Logic access key.
    • Alert Logic API endpoint—Do not change the default parameter value (api.global-services.global.alertlogic.com).
    • Alert Logic Data Residency—Do not change the default parameter value (default).
    • Event Hub Resource Group—Type the resource group for the existing Event Hub.
    • Event Hub Connection String—Type the connection string for the existing Event Hub.
      • For information about using Azure PowerShell and Azure CLI to get the connection string, see the Microsoft document, Get an Event Hubs connection string.
    • Event Hub Namespace—Type the namespace for the existing Event Hub.
    • Event Hub Name—Type the name of the existing Event Hub.
    • Event Hub Max Throughput Units—(Optional) The maximum number of throughput units for the Event Hub.
    • Event Hub Auto Inflate Enabled—(Optional) Whether auto-inflate is enabled for the Event Hub.
    • Event Hub Partition Count—(Optional) The number of partitions for the Event Hub.
    • Event Hub Retention Days—(Optional) The number of days to retain data in the Event Hub.
    • Event Hub Consumer Group—Type the name of the consumer group of the existing Event Hub.
      If the Event Hub has other consumers, create a separate consumer group for the Alert Logic collector and use that name here. For more information, see the Microsoft document, Consumer Groups.
  1. If you are using the Premium plan ARM template page in Azure, provide the following additional required template parameters.
    • AppService Plan SKU Name—Select App service plan options for Elastic premium (EP1, EP2 or EP3) from the dropdown.
    • Vnet Name—Type the name of the virtual network for virtual network integration. The default value is [format('vnet-{0}', uniqueString(resourceGroup().id))].
    • Function Subnet Name—Type the name of the virtual network subnet to be associated with the Azure Function app. The default value is (al-function-subnet).
    • Private Endpoint Subnet Name—Type the name of the virtual network subnet used for allocating IP addresses for private endpoints. The default value is (al-privateendpoint-subnet).
    • Vnet Address Prefix—Type the IP address space used for the virtual network. The default value is (10.100.0.0/16).
    • Function Subnet Address Prefix—Type the IP address space used for the Azure Function integration subnet. The default value is (10.100.0.0/24).
    • Private Endpoint Subnet Address Prefix—Type the IP address space used for the private endpoints default value is (10.100.1.0/24).
  2. Select I agree to the terms and conditions stated above.
  3. Click Purchase.

You ensure the firewall is configured for Event Hub to receive messages. To adjust the firewall, see Adjust firewall rules in Event Hub.

Verify the deployment

After you deploy the template, Alert Logic recommends you perform the following steps to verify the template deployed successfully:

  1. In the Alert Logic console, select the Azure deployment for the Azure subscription where you deployed this log collector.
  2. Click CONFIGURE LOG SOURCES, and then filter the list by Push (Office 365, EventHub) collection method.
  3. Verify that the new Event Hubs log source with the name you provided during deployment appears with the source status of "OK."

If the Event Hub log source does not appear, you can:

  • Wait approximately 30 minutes, and see if the new Event Hub log source appears.
  • Remove the Azure resource group that contains the log source, and deploy the ARM template again.
  • Contact Alert Logic Technical Support at:
    • US: (877) 484-8383 (Option 1)
    • UK: +44 (0) 203 011 5533 (Option 1)

Integrate with Azure Event Hubs

The following links contain instructions to help you integrate different Azure services with Event Hubs.