DATES IN WEB LOGS
Web log filenames usually contain the date the file was created. For example, IIS servers create a daily log and use the date in the filename, like this:
This log was created on January 10, 2014. The "140110" in the filename represents the 2 digit year, month, and day of the file creation date. Moving forward, the filename from January 11 will contain "141011", the filename from January 12 will contain "140112", and so on.
Therein lies the problem. If every new log file created has a different name, how do you configure Angelfish to automatically "see" each new file?
Date substitution statements automatically adapt to an ever-changing date, like this:
"YY" represents the current 2 digit year
"MM" represents the current 2 digit month
"DD" represents the current 2 digit day
During processing, Angelfish will replace any date substitution statements with the current year/month/day and look for files that match the pattern. If the current date is Jan. 10, 2014, the above example will match any file named "ex140110.log"
Date Substitution Offset
Datasources contain an offset setting which adjusts date substitution by a number of days. By default, the setting is "-1" which will substitute yesterday's date (i.e. the current day, minus one). If the current date is Jan. 10, 2014 and the offset is -1, "u_exYYMMDD.log" will match any file named "ex140109.log"
DATE SUBSTITUTION SYNTAX
Commonly-used date substitution syntax is below:
- YYYY The current 4 digit year
- YY The current 2 digit year
- MM The current 2 digit month
- DD The current 2 digit day
- HH The current 2 digit hour
- %d The day of the month as a decimal number (range 01 to 31).
- %H The hour as a decimal number using a 24-hour clock (range 00 to 23).
- %I The hour as a decimal number using a 12-hour clock (range 01 to 12).
- %m The month as a decimal number (range 01 to 12).
- %y The year as a decimal number without a century (range 00 to 99).
- %Y The year as a decimal number including the century.
Angelfish uses the strftime() function, which accepts additional abbreviations.
The regular expression .* (a.k.a. "dot star") roughly translates to "match everything". If you want to match all files in a directory, you can use /.* as your match pattern in the location field.
But what if you only want to match a certain list of files, like files from the year 2014? Simply append the wildcard to the match pattern, like this:
You can also combine wildcards and date substitution statements in the same field, like this:
Ddate substitution statements are most useful when combined with a wildcard, because they automatically compensate for errors. For example...if you use u_exYYMMDD.log for date substitution and your Angelfish server takes a nap over the weekend, you'll have to update each Datasource on Monday so the skipped days are processed.
A better approach is to use u_exYYMM.* - this statement will match any file from the current year and month. If your Angelfish server crashes on the 12th of the month and isn't revived until the 15th, Angelfish will automatically process the missing logs!
In addition to wildcards, there are other Regular Expressions you can use with date substitution.
Consider the following list of log files, which shows the first of the month from Jan 2013 - December 2013
If the current date is December 10, 2013:
- u_exYY.* matches all files from 2013 (all files listed)
- u_exYYMM.* matches all files from December 2013 (only u_ex131201.log)
- u_exYYMMDD.* matches all files from December 9, 2013 (no file exists)