Datasources Overview

Datasources Overview

Before Angelfish can process data, it needs know which log files you want it to read. Angelfish gets this information from a Datasource.

Datasources specify the location of log files, and each profile needs at least one Datasource, and more than one datasource can be specified for a profile. Logs can be stored locally on the server, or accessed remotely via a UNC path, SFTP, or FTP.

Datasource Fields

Datasource Name:  This one is up to you, frequently users keep the profile name and datasource name the same, or similar.

Profile Group:  If your environment has more than one profile group, match this field to the group the desired profile belongs to.

Log Format:  Log Format must match the log file format, or Angelfish won't process the log files correctly.
Help Article: Log Formats

Log Location Type:  Where are your logs stored?  Available locations can be local storage, or servers connected by FTP or UNC paths.

* Location:  The path to the log files to be read must be given so the file name pattern is defined.  (e.g. /log/path/access_log_YYYYMMDD.log)
Cannot put only the directory path here, it must have the file name pattern included as well, otherwise Angelfish will tell you it cannot find any logs.

Date Substitution Offset:  This tells Angelfish which logs to process, when you use the YYYYMMDD date substitution variables.  If -1 then you are going to process the prior days logs (normally used in daily processing).  If -2 you are processing logs from 2 days prior, and so on.

Important:

Delete After Processing:  This is rarely used, unless logs being read by Angelfish are a copy of logs stored elsewhere.  This feature attempts to delete the log file(s) that were processed, after processing is finished.
We recommended leaving this unchecked unless you are sure you have copies of the same log files in another location Angelfish is not pointing to.