Log Management policies

Alert Logic allows you to create four types of Log Management policies. These policies dictate how Alert Logic collects log messages and updates your software.

  • A collection policy sets up rules for collection based on the type of log messages you want to collect. Alert Logic runs an existing collection policy in the corresponding collection source.
  • A correlation policy allows you to create a new log message when Alert Logic collects a configured number of log message types during a configured time frame. Then you can set these new messages to trigger alerts.

  • An updates policy allows you to schedule hosts to update to the latest version of the agent software at the agent's specified check-in.

Alert Logic automatically assigns either a Windows event log or Syslog source to each host in your environment. To edit a default collection source, you must create a new Windows event log or Syslog policy. Also, to collect flat file or S3 log messages, you must create a new collection policy, and then create the corresponding collection source.

To access the Log Management policies page, click CONFIGURATION, click Log Management, and then click Policies in the left navigation panel.

Collect Flat File policies

Flat file log messages (also known as text-based log messages) are a common log message format and can be collected, stored, and normalized similarly to Windows event log messages and Syslog messages. A flat file policy lets you collect flat files for Alert Logic to review.

Before you can create a flat file collection source, you must create a flat file collection policy. For more information, see Create and maintain flat file log collection sources.

After you create the flat file collection source, the collection source executes the flat file collection policy.

Configurations for collection policy

No industry standard exists to structure flat file log messages. As a result, log formats vary by computer device.

Enable collection and parse logs

  1. Define the rotation schema.
    The rotation schema (or pattern) is the order that the date appears within the log message. For example, a pattern may be MM.DD.YYYY (month, day, year) or DD.MM.YYYY (day, month, year).
  2. For Linux users, the Alert Logic console automatically detects standard Linux log rotate formats and also provides other common formats for selection during set up.

  3. Choose a single or multi-line log. By default, Alert Logic assumes a single-line flat file log message format. For multi-line file formats, you must:
    1. Define a fixed number of rows per log message. Use a known pattern that can be used at the beginning, middle, or end of the log message. Also, this pattern can be a Perl Compatible Regular Expression (PCRE).
    2. Pick the desired time stamp method.Three options exist to configure the time stamp for each flat file:
      • Choose the local time zone and settings of the log source.
      • Choose one of several predefined rules.
      • Create a custom time format.

Supported flat file rotation formats

Alert Logic supports gzip, bzip2, and zip compressed logs for the rotation schemes below:

  1. YYYYMMDD (IIS Native Method):
    • Typically rotated files are given the form: <name>YYYYMMDD.log

      For example: ex20091230.log

    • Newest log file is <name>.log

      For example: ex.log

    • Files ordered based on YYYYMMDD value
  2. YYYYMMDD (append method):
    • Rotated files are given the form: <name>.YYYYMMDD
    • Newest rotated log has the highest epoch time
  3. Epoch Timestamp
    • Rotated logs get epoch time appended, in the form: <name>.<epoch>

      For example:  access.log.1757392910

    • Newest rotated log is one with highest epoch time
  4. Incrementing Integer Method (logrotate)
    • Newest rotated log named <name>.1
    • Older logs increased in count:
      • syslog+
      • syslog.1 (newest)
      • syslog.2 (2nd newest)
      • syslog.3 (oldest)
  5. Other formats
YYMMMM-DD-YYMM_DD_YYYYDD_MM_YY
YYMMDDMM.DD.YYYYDD-MM-YYDD-MM-YYYY
YYMMDDhhMM.DD.YYDD.MM.YYYYDD_MM_YYYY
MM-DD-YYYYMM_DD_YYDD.MM.YYYYYY-MM-DD
YY-MM-DD   

Create a Flat File collection policy

The collection policy determines which flat file log messages to collect, how to separate log messages within a flat file, and how to read the time of each log message. Also, the collection policy can specify the flat file log message collection times.

To create a Flat File collection policy:

  1. Access the Log Management Policies page and click the Flat File tab.
  2. Click the Add icon ().
  3. In Flat File Policy Name, type a descriptive name.
  4. In Source File Path, type the path information.

    To use an agent for collection, specify the local file system path to the log files. Otherwise, specify the network share path to the log files.

  5. In File Name or Pattern, type the file name or date pattern of the flat file log messages. Log Manager can only collect flat file log messages that match the pattern.

    htaccess.* is a file name with a pattern. The * represents the time stamp of the flat file log. Log Manager accepts a variety of date formats.

  6. From File Name Rotation Scheme, select a file name rotation scheme. The format must match the format of your flat file log messages.

    The default Auto-Detect identifies many rotation schemes. Alert Logic recommends you specify the rotation scheme format of your flat file log messages. If you are unsure of the format, or if you do not find the specific format from the drop-down menu, select Auto-Detect.

  7. In Multi-line Handling, select a multi-line handling option:
    • If all of your flat file log messages contain a single line, keep the selection: File contains single line log messages.
    • If all of your flat file log messages do not contain a single line, select File contains log messages with multiple lines. Also, select and enter a configuration:
      • If the length of your log messages are consistent:

        Keep the selection: Each log message spans a fixed number of lines, and then in Number of lines, type the number of lines.

      • If the length of your log messages are not consistent:

        Select Each log message follows a known pattern, select the appropriate Pattern application, type the Pattern that takes place in the log message, and then if your pattern is a Perl Compatible Regular Expression (PCRE), select Regular expression.

        Pattern application options:

        • At the beginning of message: A line that matches the specified pattern marks the beginning of a new message; non-matching lines are lumped into the prior message.
        • In the middle of message: A line that does not match the specified pattern marks the beginning of a new message; matching lines are lumped into the prior message.
        • At the end of message: A line that matches the specified pattern marks the end of a message; non-matching lines prior to that are lumped into this message.
  8. In Timestamp Rule, select a timestamp rule option:
    • To use the timestamp from the collector, keep the selection: Set message time as collect time.
    • To use an existing timestamp, select Parse times from messages using a pre-defined timestamp format, and then select a format from Format a date string.
    • To use a custom timestamp, select Parse times from messages using a custom timestamp format, and then enter a format for the date string in the expanded configuration area. In the Format of date string field, type a format for the date string, and follow the on-screen instructions.
  9. In Host Credential, select or create a credential:

    If you use the Alert Logic agent for log collection, do not select or create host credentials.

    • To use an existing credential, keep the default selection: Use an existing credential, and then from Existing Credential, select a credential.
    • To create a new credential, select Create a new credential, and then enter new credentials. In the corresponding configuration fields, type a Credential Name, Host Username, and Host Password. In the Retype Password field, retype the host password.
  10. In Collection Schedule, select or create a collection schedule:
    • To select a collection schedule, keep the default selection: Use an existing schedule, and then from Existing schedule, select a schedule.
    • To create a new collection schedule, select Create a new schedule, and then enter and select the schedule options. Type a Schedule Name, select a Schedule Time Zone, and select Enabled to enable blackout periods. Also, you can add or remove extra blackout periods.
  11. Click Save.

Update a flat file collection policy

If you update, archive, or delete any collection, policies, or alert rule configurations, you could break interconnected configurations.

To update a Flat File collection policy:

  1. Access the Log Management Policies page and click the Flat File tab.
  2. In the list of flat file collection policies, click the pencil icon ( ) for the collection policy to edit.
  3. Make the necessary updates. For more information, see Create a Flat File collection policy.
  4. Click Save.

Delete a Flat File collection policy

If you update, archive, or delete any collection, policies, or alert rule configurations, you could break interconnected configurations.

To delete a Flat File collection policy:

  1. Access the Log Management Policies page and click the Flat File tab.
  2. Click the trash icon ( ) for the flat file collection policy to delete.
  3. Click Delete.

Collect Syslog policies

Syslog is a way for network devices to send event messages to a logging server – usually known as a syslog server. A Syslog policy lets you collect syslog files for Alert Logic to review.

You must create a collection policy before you can create a collection source.

If you update, archive, or delete any collection, policies, or alert rule configurations, you could break interconnected configurations.

Create a Syslog collection policy

To create a Syslog collection policy:

  1. Access the Log Management Policies page and click the Syslog tab.
  2. Click the Add icon ().
  3. In Syslog Policy Name, type a descriptive name.
  4. In Syslog Listen Port, type the port where the agent receives Syslog messages.
  5. In Local Syslog Cache Disk Limit (MB), type the amount of disk space you want to allow to store Syslog messages.
  6. In Collection Schedule Blackout Periods, select or create a collection schedule:

    • To select a collection schedule, keep the default selection: Use an existing schedule, and then from Choose a schedule, select a schedule.
    • To create a new collection schedule, select Create a new schedule, and then enter and select the schedule options. Type a Schedule Name, select a Schedule Time Zone, and select Blackout Periods to enable blackout periods. Also, you can add or remove extra blackout periods.

  7. Click Save.

Update a Syslog collection policy

To update a Syslog collection policy:

  1. Access the Log Management Policies page and click the Syslog tab.
  2. In the list of syslog collection policies, click the pencil icon ( ) for the syslog collection policy to edit.
  3. Make the necessary updates. For more information, see Create a Syslog collection policy.
  4. Click Save.

Delete a Syslog collection policy

To delete a Syslog collection policy:

  1. Access the Log Management Policies page and click the Syslog tab.
  2. Click the trash icon () for the syslog collection policy to delete.
  3. Click Delete.

Collect Windows Event Log policies

Windows Event Log files track significant events on a Windows server, such as user login or a program error. A Windows Event Log policy lets you collect event log files for Alert Logic to review.

You must create a Windows Event Log collection policy before you set up a Windows event log collection source.

If you update, archive, or delete any collection, policies, or alert rule configurations, you could break interconnected configurations.

Create a Windows event log collection policy

To create a Windows event log collection policy:

  1. Access the Log Management Policies page and click the Windows Event Log tab.
  2. Click the Add icon ().
  3. In Windows Event Log Policy Name, type a descriptive name.
  4.  In Host Credential, select or create a credential:

    If you use the Alert Logic agent for log collection, do not select or create host credentials.

    • To use an existing credential, keep the default selection: Use an existing credential, and then, from Choose a credential, select a credential.
    • To create a new credential, select Create a new credential and then enter new credentials. In the corresponding configuration fields, type a Credential Name, Host Username, and Host Password. In the Retype Password field, retype the host password.
  1. In Collection Schedule Blackout Periods, select or create a collection schedule:
    • To select a collection schedule, keep the default selection: Use an existing schedule. Next, from Choose a schedule, select a schedule.
    • To create a new collection schedule, select Create a new schedule, and then enter and select the schedule options. Type a Schedule Name, select a Schedule Time Zone, and select Blackout Periods to enable blackout periods. Also, you can add or remove extra blackout periods.
  2. Choose one of the following:
    • To collect all Windows event log streams, keep the default selection Collect All Available Event Log Streams selected.
    • To collect specific Windows event log streams, deselect Collect All Available Event Log Streams and select your desired streams under Alert and Collect on Selected Streams.
  3. Click Save.

Update a Windows event log collection policy

To update a Windows event log collection policy: 

  1. Access the Log Management Policies page and click the Windows event log tab.
  2. In the list of Windows event logs, click the pencil icon ( ) for the Windows event log to edit.
  3. Make the necessary updates. For more information, see Create a Syslog collection policy.
  4. Click Save.

Delete a Windows event log collection policy

To delete a Windows event log collection policy:

  1. Access the Log Management Policies page and click the Windows event log tab.
  2. Click the trash icon () for the Windows event log policy to delete.
  3. Click Delete to confirm.

Collect S3 policies

S3 collection policies set guidelines for collecting Amazon Simple Storage Service (S3) access logs, which provide details about a single access request, such as the requester, bucket name, request time, request action, response status, and error code, if any. A S3 policy lets you collect S3 logs for Alert Logic to review.

You must create an S3 collection policy before you can create an S3 collection source.

Though this feature appears to all users, this feature only works on AWS accounts.

If you update, archive, or delete any collection, policies, or alert rule configurations, you could break interconnected configurations.

Create a S3 collection policy

No default policy exists for the S3 collection policy. You must create a default policy for the S3 collection policy to use this feature.

To create a S3 collection policy:

  1. Access the Log Management Policies page and click the S3 tab.
  2. Click the Add icon ().
  3. In the Name field, enter a name for the new S3 policy.
  4. In Policy Template, select Customized.
  5. In Multiline Handling, select a multiline handling option:
    • If all of your flat file log messages contain a single line, select File contains single line log messages.
    • If all of your flat file log messages don't contain a single line, select File contains log messages with multiple lines. Also, select and enter a configuration:
      • If the lengths of your log messages are consistent, select Each log message spans a fixed number of lines and then type the number of lines in Number of lines,.
      • If the lengths of your log messages are not consistent, select Each log message follows a known pattern, select the appropriate Pattern application, type the Pattern that takes place in the log message, and then select Regular expression to use a Perl Compatible Regular Expression (PCRE).
  6. Select a Timestamp Rule option:

    • To use the timestamp from the collector, select Set message time as collect time.
    • To use an existing timestamp, select Parse times from messages using a pre-defined timestamp format, and then select a format from Format a date string.
    • To use a custom timestamp, select Parse times from messages using a custom timestamp format, and then enter a format for the date string in the expanded configuration area. In the Check Format field, type a format for the date string, and follow the on screen instructions.
  7. Click Save.

Update an S3 collection policy

Though this feature appears to all users, this feature only works on AWS accounts.

To update a S3 collection policy:

  1. Access the Log Management Policies page and click the S3 tab.
  2. In the list of S3 policies, click the pencil icon ( ) for the S3 collection policy to edit.
  3. Make necessary changes to:
    • The policy Name
    • The selected Policy Template. For for information about settings for a Customized policy template, see Create a S3 collection policy.
  4. Click Save.

Delete a S3 collection policy

To delete a S3 collection policy:

  1. Access the Log Management Policies page and click the S3 tab.
  2. Click the trash icon ( ) for the S3 collection policy to delete.
  3. Click Delete.

Related topics