Follow

Date Substitution and Wildcards in Datasources

DATES IN WEB LOGS

Web log filenames usually contain the date the file was created. For example, IIS servers create a daily log and use the date in the filename, like this:

u_ex140110.log

 

This log was created on January 10, 2014. The "140110" in the filename represents the 2 digit year, month, and day of the file creation date. Moving forward, the filename from January 11 will contain "141011", the filename from January 12 will contain "140112", and so on.

Therein lies the problem. If every new log file created has a different name, how do you configure Angelfish to automatically "see" each new file?


DATE SUBSTITUTION

Date substitution statements automatically adapt to an ever-changing date, like this:

u_exYYMMDD.log

 

"YY" represents the current 2 digit year
"MM" represents the current 2 digit month
"DD" represents the current 2 digit day

During processing, Angelfish will replace any date substitution statements with the current year/month/day and look for files that match the pattern. If the current date is Jan. 10, 2014, the above example will match any file named "ex140110.log"

Date Substitution Offset

Datasources contain an offset setting which adjusts date substitution by a number of days. By default, the setting is "-1" which will substitute yesterday's date (i.e. the current day, minus one). If the current date is Jan. 10, 2014 and the offset is -1, "u_exYYMMDD.log" will match any file named "ex140109.log"


DATE SUBSTITUTION SYNTAX

Commonly-used date substitution syntax is below:

  • YYYY   The current 4 digit year
  • YY   The current 2 digit year
  • MM   The current 2 digit month
  • DD   The current 2 digit day
  • HH   The current 2 digit hour
  • %d   The day of the month as a decimal number (range 01 to 31).
  • %H   The hour as a decimal number using a 24-hour clock (range 00 to 23).
  • %I   The hour as a decimal number using a 12-hour clock (range 01 to 12).
  • %m   The month as a decimal number (range 01 to 12).
  • %y   The year as a decimal number without a century (range 00 to 99).
  • %Y   The year as a decimal number including the century.

Angelfish uses the strftime() function, which accepts additional abbreviations.


WILDCARDS

The regular expression .* (a.k.a. "dot star") roughly translates to "match everything". If you want to match all files in a directory, you can use /.* as your match pattern in the location field.

But what if you only want to match a certain list of files, like files from the year 2014? Simply append the wildcard to the match pattern, like this:

 u_ex14.*

You can also combine wildcards and date substitution statements in the same field, like this:

u_exYYMM.*

BEST PRACTICES

Ddate substitution statements are most useful when combined with a wildcard, because they automatically compensate for errors. For example...if you use u_exYYMMDD.log for date substitution and your Angelfish server takes a nap over the weekend, you'll have to update each Datasource on Monday so the skipped days are processed.

A better approach is to use u_exYYMM.* - this statement will match any file from the current year and month. If your Angelfish server crashes on the 12th of the month and isn't revived until the 15th, Angelfish will automatically process the missing logs!

In addition to wildcards, there are other Regular Expressions you can use with date substitution.


EXAMPLES

Consider the following list of log files, which shows the first of the month from Jan 2013 - December 2013

u_ex130101.log
u_ex130201.log
u_ex130301.log
u_ex130401.log
u_ex130501.log
u_ex130601.log
u_ex130701.log
u_ex130801.log
u_ex130901.log
u_ex131001.log
u_ex131101.log
u_ex131201.log


If the current date is December 10, 2013:

  • u_exYY.* matches all files from 2013 (all files listed)
  • u_exYYMM.*  matches all files from December 2013 (only u_ex131201.log)
  • u_exYYMMDD.*  matches all files from December 9, 2013 (no file exists)

STILL HAVE QUESTIONS?

Please contact our support team - we're happy to help.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

0 Comments

Article is closed for comments.