NUsyslog 1.6.2+ NUsyslog is a replacement log rotation management tool that replaces the legacy date-rotate and newsyslog utilities. It's design is centered around giving maximum flexibility for log file management. Though it's customised for a specific environment, there is enough flexibility to make it useful for many needs. Command line / Cron Invocation NUsyslog is typically run out of cron on the hour. It reads its specified flags and a configuration file to determine how it acts and behaves. The format for invoking NUsyslog is as follows: nusyslog [-v | -vt] [-d] [-f[:logfile]] The various flag meanings: -v = verbose to logfile -vt = verbose to stdout -d = debug only, take no action (assumes -v) -f = force all logfiles to be rotate -f: = force rotate specific logfiles The '-v' flag writes debug output from the programs execution to the NUsyslog logfile (specified in the configuration file). The '-d' is identical, except that no actual work is done, only what the program WOULD do if invoked with the given arguments and configuration. '-vt' is identical to '-v' except that the output is sent to STDOUT instead of to the logfile. '-f' forces a logfile, which must be specified in the configuration file, to be rotate, no matter what conditionals are given in the configuration. With '-f', multiple logfiles can be specified by using a comma delimiter. Without any arguments, '-f' will force rotate all logfiles listed in the configuration file. Within crontab, it is normal to run NUsyslog on the hour. Configuration File The configuration file specifies the PID file for NUsyslog, compression program, run time logfile for NUsyslog, an optional delay variable, and ech of the logfile items. There are 11 base fields in the configuration per logfile entry. A few other global options are also available. The per logfile items specify how NUsyslog is to handle that logfile's rotation, archival, and conditions under which is rotated. The fields are seperated by whitespace. An example from a configuration file is listed below. #NUsyslog config file # 0 1 2 3 4 5 6 7 8 9 10 LOG local3.log *:* 644 7 1024 *:6,15 ZT - Lh EXEC:/ms/bin/doit LOG /var/log/syslog 0:0 640 6 0 *:* - - L SIGHUP:/var/run/syslog.pid LOG /tmp/log1 0:0 640 3 0 *:* - /home/mikeh hL EXEC:/bin/true A rough alignment of a field number is above the entries. All field entries are required. The basic descriptions for each field are below. 0 1 2 3 4 5 6 7 8 9 10 LOG FULLPATH UID:GID PERMS COUNT SIZE ROT_TIME FLAGS ARCHDIR FORMAT ACTION Summary of Log entry fields: 0 LOG keyword denoting that this is a logfile item 1 complete path to the log file, filename.*/? indicates expire only 2 uid:gid of rotated file, *=use existing for logfile 3 perms on rotated log file, should be set 4 number of days for timestamped files OR number of logs for numbered files 5 size indicator in MB, rotate if size >= to this, use '-' to ignore 6 days:hours rotate by time, both fields must match current time, *=all 7 flags: T=use timestamp instead of numbered files, Z=compress N=don't recreate on rotate, C=create if doesn't exist S=skip rotation+expire part, O=process even if open o=override expire-only 'keep 1' exemption 8 archive directory to move rotated files to, '-' to use exising directory 9 format, L=logfile, h=hostname, H=FQDN (ifset) 10 TYPE:file signal to send or script to exec, first field is signal or EXEC string, 2nd field is the pidfile or script name (signal example: SIGHUP) Field 0, LOG, is case-insensitive and denotes that this is a logfile item Field 1, specifies the complete logfile, including optional path (recommended) that is to be checked, rotated, and archived. Wildcard characters, * and ?, can be specified, but this restricts processing to expiration only as there is no exact file pattern that we can be certain to match on without errors. The wildcard characters can be specified in any format, 'mylog-*.??.log' for example. With wildcards, the most precise format specification as possible should be given to reduce the possibility of false matches. Compression and archival can still be used with wildcarded logfiles, but timestamp rotation or special archival formatting are ignored. Expiration is based on ASCII order designation, so wildcard logfiles need to increment their logfile names when rotated. On Solaris and Linux, an open file check is done to make sure the logfile is not in use before attempting to expire it. All matching logfiles except the "current" one (the highest in ASCII order) are archived. The following wildcard characters are: * = anything, including nothing ? = any single character % = any single digit @ = any single alphabetic character & = any single alpha-numeric character $ = any single white-space character It is recommended to avoid using '*' at the beginning or end of a logfile name. Field 2, specifies the UID and GID that the archived logfile will have. If * is specified for UID or GID, then the owner or group of the original logfile will be used instead. The format is UID:GID. Field 3, specifies the file permission modes in octal for owner, group, and world, respectively per digit. Field 4, depending on whether timestamps are used, denotes the number of logfiles that will be kept archived. For numbered files, this represents the maximum number of the archived logfile's extension. For expire only (wildcarded) logfiles, it is the maximum number of archival files that match the pattern in sorted order that will be kept around. For timestamped logfiles, it is the maximum number of days that will be archived as derived from the logfile's timestamp embedded in the filename. Field 5, specifies an optional rotate by size value. If not '-', then if the logfile reaches the specified size (in Megabytes) or greater, it will be rotated/archived no matter what is in the rotate by time field. Field 6, specifies the rotation time(s). The field format is DAYS:HOURS. Both days and hours can be comma delimited to specify multiple rotation times. By default, days match the day of the week and are in the range of 0 - 6. If a negative value is specified for a day, then it is assumed to be a day of the week. Allowed day of month range is -1 to -31. Be careful with ranges beyond -28 as not all months may match. Hours are specified in military time, 0 - 23. '*' can be used to match all days or hours. If the day AND time match, then the logfile will be rotated and archived. Field 7, denotes special flags that change the way NUsyslog handles the archiving of the file. Multiple flags are simply grouped together (eg. 'COZ'). Flag order is unimportant, but they ARE case-sensitive. The allowed flags are as follows: T = use timestamp instead of numbered files Z = compress N = don't recreate on rotate C = create if doesn't exist S = skip rotation+expire part O = process even if open - The flag, 'T', specifies that a timestamp should be added to the logfile during the archival process. The timestamp format is YYYYMMDDHHMM. This timestamp is then used in the expiration process. This option is ignored if wildcards are used in the logfile name. - The 'Z' flag indicates that the archived logfile should be compressed (see the ZIP option below). When expiring files, the specified zip extension will be assumed to be appended to the logfile name. - The 'N' flag tells NUsyslog not to recreate the original logfile after it has been rotated. Syslogd requires that the logfile exist, but some systems may not and will recreate it themselves. - The 'C' flag tells NUsyslog to create the logfile even if it did not exist before. - The 'S' flag tells NUsyslog to not actually do any rotation or archival/expiration, but if the logfile matches the given conditions, then the specified action (see field 10 below) will be carried out. - The 'O' flag tells NUsyslog to ignore whether or not the file is open and process it as if it were closed while doing an expire. Normally, if the archived logfile is still open (on Linux or Solaris), then it will not be processed. This flag overrides that behavior. NOTE that the T, C, N flags are ignored for wildcarded files. Field 8, specifies the archive directory, if any, that rotated/archived logfiles will be stored in. If '-' is specified, then the rotated/archived logfiles will be kept in the same directory as the original live logfile. Field 9, Specifies the logfile format for archived files. The three values allowed are 'L', 'h' and 'H'. 'L' specifies the base logfile name (without the path). 'h' specifies the hostname. 'H' specifies the fully qualified hostname. The minimum that must be specified is 'L'. The various includes are sperated by '.' in the archived filename. The order of the formatters specify the order in the archived logfile. For example, formatters 'Lh' for 'mylog.log' on host 'mybox' would produce an archive filename of 'mylog.log.mybox'. 'hL' would produce, 'mybox.mylog.log'. A timestamp (if the 'T' flag is specified) would be appended to that logfile for archival using '.' as a seperator. Field 10, specifies what post rotation, pre-archival, processing actions should be taken. A signal or script execution can be specified. To send a signal, the signal name (all caps as given in /usr/include/signal.h), for example, SIGHUP, followed by a colon and then a PIDfile that contains the PID of the process to send the signal to. If script execution is desired instead, then the keyword, 'EXEC', is specified, followed by a colon, then the full path name of the script. Optional arguments can be specified after the script name using ';' to seperate the arguments. Examples of each method: SIGHUP:/var/run/syslogd.pid EXEC:/etc/init.d/myservice;restart If multiple logfiles to be rotated call the same process or script, then it will be done only once. Other options Additional options besides 'LOG', allow specific definitions to be given. The 'ZIP' option allows one to specify which compression utility to use, what extension is added, and what nice value to use (as compression can be very CPU intensive). The format is: ZIP path_to_zip_program NICEVAL EXTENSION And example is: ZIP /usr/local/bin/gzip 15 .gz To specify the PID file for NUsyslog, use the PID keyword. An example is below. PID /ms/var/run/nusyslog.pid The PID file is required to keep two NUsyslog's from stomping on each other. Finally, the 'RUNLOG' keyword is used to specify where the output of the program can be saved (if '-v' or '-d' is specified). RUNLOG /var/tmp/nusyslog.log Additional verbosity can be included in logfile output if compiled with -DDEBUG (used with -v, -vt, or -d). An optional keyword, 'DELAY', can introduce a delay between rotation and archival. This can be useful to prevent a shared storage device from being overloaded by multiple archivals from many different systems simultaneously. The format is: 'DELAY '. The value is in seconds, causing NUsyslog to sleep that many seconds before starting the archival process. If the value is negative, then a random value up to, but not more than the given value. The random value changes everytime NUsyslog runs. An example: DELAY -600 # sleep random amount, but no more than 10 minutes (600 sec) Of the keywords, all but 'DELAY' are required. At least one instance of 'LOG', and only one instance of 'ZIP' (if compression is ever used), 'PID', and 'RUNLOG' can and must exist. Operation For static logfiles (non-wildcarded), NUsyslog creates a symlink with a temporary name that points to the current actively rotated and being processed logfile. If NUsyslog dies or the system crashes, then NUsyslog will detect the symlink and attempt to re-process that logfile before acting on the newest logfile. Normally, NUsyslog needs to run as root, but if no ownership changes are required nor root write permissions to archive or log directories, then it can be run as another user. NUsyslog also refuses to work on any logfiles that are not regular files including symlinks. MISC Send bug reports and feature requests to mikeh@mindspring.com. Latest version can be pulled from: ftp://bashful.gcsu.edu/pub/mikeh This document last updated 12/22/06 NUsyslog Changelog: 1.6.2: - removed superfluous symlink created for non-existant expireonly wildcard logfiles - added additional check for wildcards to make sure we don't include already archived logfiles which may match pattern - added more debugging - improved parsing for field 10 log entries - added checks for target logfiles to make sure they are regular files - added 'o' flag to no keep one "active" file on expire-only wildcards - fixed wrong user:group and permissions on archived wildcard files - fixed bug with expiring archived expire-only files when nothing rotated 1.6.1: - fixed rotate by day of month bug - fixed minor file descriptor leak Previous changes not recorded here. TODO: 1.7 - add filename passing option to prearchival execution script - add post archival script execution - enhance DELAY option to allow for a range