Learning

Learning is a section in the Alert Logic Managed Web Application Firewall (WAF) management interface. To access the Learning section, click Websites on the left panel, under Services. On the Websites page, click the website you want to manage, and click Learning status to access the first section.

Learning includes the following features. Click on the link to go to the corresponding documentation to learn more:

Learning

This document includes the following sections. Click on the link to go to the corresponding section to learn more:

To go to the documentation for the previous section of Alert Logic Managed Web Application Firewall (WAF) management integration, see Deny and Error Handling . To go to the documentation for next subsection in the Learning section, see Learning Data.

To save configuration changes or edits you make to any features and options, you must click Save on the lower-right of the section or page where you are making changes. Click apply changes on the upper-left corner of the page, and then click OK. Your changes will not be stored if you do not properly save your changes.

The Learner builds a complete profile of the web site including static requests, web applications and input parameters bu analyzing incoming requests.

To avoid learning from worms, attacks and other unauthorized access the Learner employs a combination of heuristic attack classification, statistics and server responses.

When learning is enabled for the website the Learner keeps analyzing requests until no changes to the resulting policy are recorded. That is, for every 10,000 requests the Learner builds a trial policy, compares it to the former trial policy and records the number of changes. When a configurable number of trial policies in a row (default 30) has not resulted in a number of changes between each trial build exceeding a configurable threshold (default 0) a policy is built.

By default the Learner is configured to generate a short yet fine grained policy. This is achieved by identifying global characteristics of the web site and generating global patterns matching those characteristics. The global patterns typically account for the majority of the web systems content and applications leaving only the "real" web applications to be accounted for by specific web application policy entries.

Learning status

Learning progress indicators

The two bars in the top of the page indicates the current state of sampling and verification.

The Learner works in two stages when profiling the website.

Sampling progress

This is the process of collecting information about the website in terms of what paths/applications are used, what parameters do they take as input, what extensions are used for static content, etc.
Verification

The verification process 1) validates the data samples using statistical methods like analyzing spread in IP sources and time, number of requests, etc. and 2) verifies that the resulting policy covers the requests sampled.

As the Alert Logic Managed Web Application Firewall (WAF) Learner extracts characteristics like extensions, specific directories in paths and global parameters (parameter names a number of applications take as input - like print=1) and even patterns used in global parameters the verification process may start before the Data sampling progress has reached 100%.

Verification is calculated as the number of sample runs in a row with no policy changes relative to the required number configured in learner settings.

When Verification has reached 100% WAF will either build and commit a new policy or notify the administrator by email that verification has reached 100% and a new policy can be built and committed.

Policy resulting from current learn settings

This section shows a sample of the policy resulting from the Learner settings effective.

When the settings are changed the resulting policy sample is rebuilt using the new threshold values. This is done as a background job and depending on the load on the WAF node and the complexity of the sample data it may take anywhere from e few seconds to a minute or two to build the policy. If the new policy is not visible yet, wait a while and refresh the window.

Commit to WAF Button	Builds a policy which is accessible and editable in the Global Patterns and Web applications windows. When clicked the policy displayed in the table will be committed to the WAF engine, that is: made active for filtering requests. If policy verification has not reached a warning message (two actually) will be displayed asking to confirm the action. Remember: Alert Logic Managed Web Application Firewall (WAF) is a white-list based WAF. If the policy put into production does not match real life requests building the policy prematurely (not fully verified) is likely to result in false positives. If verification has not reached 100% it means that it not verified that the policy does not generate false posivites. Have patience and wait for verification to reach 100%.
Web applications Expandable: Click + to expand.	Learned web applications. Expand the item to get a list of applications learned. For each application is shown: URL path Methods learned Parameters. Parameters are shown as name, value pairs where the value is the name of the input validation class learned for that parameter. Note that only the applications private parameters are shown here. Parameters which the application have in common with other applications are included in the Global parameters list.
Global URL patterns Expandable: Click + to expand.	Global URL Path Policy built from learned applications. For each application group (see Applications learned) a regular expression is built which matches all samples in that specific group. Most CMS based web systems have a number of global parameters, like for instance `print=1`, which can be appended to most requests. Without the combination of Global URL Path Policy and Global Parameters Policy pages with static content that take global parameters, like `index.php?print=1`, would be learned as web applications and the URL paths would have to be added to the policy as web applications. This can potentially result in a huge policy which is never up to date because new content is added all the time. By making global policies that account for all the static content which is served dynamically only "real" web applications with a number of private parameters have to be mapped in detail. Thus the global patterns allows for building a condensed, yet fine grained, policy which also account for future standard content added to the web site.
Global parameters Expandable: Click + to expand.	Global Parameters Policy built from learned applications. Displayed in the format: `name = value` Depending on the Name grouping threshold value the name can either be a literal string or a regular expression matching a number of parameter names with name and value similarities. The value is displayed as a class name. When the policy is built the corresponding regular expression will be used.
Static content allowed extensions Expandable: Click + to expand.	Learned static path extensions which will be allowed.
Static content path allowed characters Expandable: Click + to expand.	Unique characters and character classes (like 'A' - all international word characters) learned from static path samples. Also the regular expression built to match requests for static content is shown. Note the last set of parantheses preceded by an escaped period `\.(\w+)`. This part will be matched with the list of allowed extensions to determine if the extension is alowed.

Policy history

When a new policy is generated and committed, either automatically or manually, it is added to the Policy history list.

Policy history	The policy number.
Type	Automatic or manual (requested by administrator user)
Report	Click link to see resulting policy and changes compared to the former (if any).
Sample run	The sample run number at which the policy was generated.
Web Apps	The number of entries in the `Web Application Policy`.
Global URLs	The number of entries in the `Global URL Policy`.
Global Parms	The number of entries in `Global Parameter Policy`.
Static	The number of entries in static file types `Static Content Policy`.

Sample run information

The Learner analyzes request samples in chunks of approximately 10,000 requests (or more if the system is very busy) . For each sample run an entry is added to the Sample run information table which shows total and delta values of summarizing the learning process.

Sample run	The sample run number.
Hits total	The total number of hits processed during the learning process.
URL paths	Total number of unique URL paths identified.
Parameters	Total number of unique parameter names identified. Uniqueness is determined by URL path. Two parameters with the same name but mapped as belonging to different URL paths are therefore identified as two unique parameters. When the policy is built WAF identifies parameters with similar names and input data as as global in scope and builds global patterns matching such parameters.
Changes	When the chunk of raw sample data has been processed WAF builds a policy based on the total sample population. This policy is compared to the policy built in the last sample run and changes are recorded. The number shown is the sum changes recorded to the Web Application Policy (`ACL`), Global URL Policy (`GURL)`, Global Parameter Policy (`GParm`) and the Static Content Policy (`EXT`). Click on the number shown to get a change report detailing the changes.
ACL	The number of changes to the `Web Application Policy` compared to the sample run before.
GURL	The number of changes to the `Global URL Policy` compared to the sample run before.
GParm	The number of changes to `Global Parameter Policy` compared to the sample run before.
Ext	The number of changes to the `Static Content Policy` compared to the sample run before.

The number of policy changes recorded is calculated with the Learner settings effective when the sample data is analyzed. Whereas the resulting policy (below) is recalculated when the Learner settings are changed this is not the case with the sample run policy builds. It is therefore possible that the two sections show different results. The next sample run is run using the new settings.

Lower button bar

The lower button bar contains the following buttons.

Re-analyze data

Button

To see the effect of deleting selected learning data in the resulting policy section click this button. Wait a few seconds and reload the page.

Reset learn data

Button

Use with caution!

When clicking this button and accepting the confirm pop-up window.

All learning data for that proxy will be deleted!

If learning is enabled the learning and data sampling process will start from scratch.