Post on 14-Apr-2018
7/27/2019 cPanellogd.pdf
1/14
1
WHM/cPanel Log ProcessingCopyright 2010 cPanel, Inc.
Revision History
Revision 3 2010-01-05 dn
Revision 4 2010-01-05 kpRevision 5 2010-01-07 kpRevision 6 2010-01-10 kpRevision 7 2010-01-11 kp
Abstract
This document details the WHM/cPanel log processing related to Apache and WHM Accounts. Particular attention is paid to
version 11.25.0 which introduced interface and behavioral changes.
Versions >= 11.25.0
Table of Contents
1. Just the Facts ........................ ................................................... ........................................................ .. 1
2. Overview ............................ ............................................................................................................... 2
2.1. cpanellogd ..................... .................................................... ................................................ 2
2.2. Version 11.25.0 ........................................................................................................................ 2
3. How Log Processing Works .............................................................. ................................................... 3
3.1. Statistics Generation ................................................................... .............................................. 3
3.2. Rotation & Retention ................................................ ........................................................ ........ 5
3.3. Synchronicity ........................................................................................................................... 6
4. Configuration ...................................... ....................................................... ........................................ 7
4.1. Cycle Frequency ......................................................................... ............................................. 7
4.2. Log Rotation ............................................................................. .............................................. 8
4.3. WHM Tweak Settings ......................................................... .................................................... 10
4.4. cPanel Options . ...... ..... ........................................................................................................ 11
4.5. Flag Files ...................................................... ....................................................... ................. 12
4.6. Other Logs .......................................................... ........................................................ .......... 12
5. Troubleshooting ...................................................... ....................................................... ................... 12
6. Known Issues and Caveats .. ........................................................ ....................................................... 13
6.1. Apache Restarts ................................................. ........................................................ ............. 13
7. Conclusion .................................................. ........................................................ ............................. 13
Glossary ................................................................... ........................................................ ................... 13
1. Just the FactsQ: Why is Apache restarting so often?
A: Depending upon the log processing configuration cpanellogd will restart Apache to preserve data integrity
within the account domain and bytes logs. See Section 3.1.1, Apache Restart Requirement
Q: How can I stop or lower the frequency of Apache restarts?
A: There are several ways to do this:
7/27/2019 cPanellogd.pdf
2/14
WHM/cPanel Log Processing
2
Reduce the frequency of log processing
Use the following values for the Schedule Configuration section of the Statistics Software Configuration
interface within WHM:
Process log files every 24 hours.
Process bandwidth every 8 hours.
Modify the timestamps of the files in /var/cpanel/lastrun. See "Contents of/var/cpanel/las-
trun" for more information..
Change the following:
Uncheck the Tweak Setting labeledDelete each domain's access logs after stats run
Caution
Leaving the log files in place may cause disk utilization issues, especially on systems that cannot
handle files larger than 2 GB
Disable log archiving. This is something the end user may configure.
Configure Log Processing Blackout hours ( see Black-out Hours ) to prevent log processing during busy traffic
periods.
2. Overview
2.1. cpanellogd
WHM/cPanel processing related to the domain logs and bandwidth logs for Accounts, as well as service level logs for
cPanel and Apache managed by the cpanellogd daemon.cpanellogd, in all versions of WHM/cPanel, is respon-
sible for tracking log file size, rotation and parsing frequency. Any utility available to affect cpanellogd's behavior is
exposed in the web interface as a value field, checkbox, or note for custom configuration.
2.2. Version 11.25.0
The Web Host Manager and cPanel suite version 11.25.0 processes statistics in a different, more robust way than past
versions. At a glance one will find two immediate differences:
1. A new configuration setting in Statistics Software Configuration within the WHM interface.
2. Periodic entries in the Apache log that Graceful restart[s] were requested and performed.
These differences highlight a new approach to statistics processing that aims to reduce processing time, CPU demand,and increase scalability.
2.2.1. Interface
Statistics processing is scheduled in WHM via Statistics Software Configuration under Server Configuration in the side-
bar. Previous versions of the WHM interface had a Statistics Frequency hourly value field only. Now the Schedule Con-
figuration subsection has two hourly value fields,Log Processing Frequency andBandwidth Processing Frequency. The
differences and affect of these values are detailed below.
7/27/2019 cPanellogd.pdf
3/14
WHM/cPanel Log Processing
3
Note
These value fields are mirrored in 'Main >> Server Configuration >> Tweak Settings >> Stats and Logs.'
However, since these value fields are most appropriately placed in the "Schedule Configuration" interface,
it should be inferred such mirroring is only present for legacy purposes and is likely to be deprecated.
Note
More interface changes are expected as respective functionality is refactored in future versions.
2.2.2. Apache Behavior
Log processing is determined by the combination of several values within cPanel. Optimal, balanced performance from
v11.25.0 can often result in Apache restarts approximately every 15 mins. Details on frequency and the affecting factors
are outlined below.
3. How Log Processing Works
cPanel will manage logs related to statistical information as well as the Apache logs. This consists of statistical log
parsing, rotation, archiving and deletion.
All other system or sub-system logs (i.e., messages, maillog, etc.) are beyond the of scope for this document.
3.1. Statistics Generation
cpanellogd triggers the log parsers which, in turn, report data forDaily Process Log, View Bandwidth Usage and the
various report engines available at the account & domain level.
Triggers are governed, in part1, by the values set in WHM's Statistics Software Configuration. Hourly values defined
in Log and Bandwidth Frequency fields do not determine the "run" frequency of cpanellogd, but instead determines the
threshold of time that respective log files are allowed to lapse between parsing.
cpanellogd checks the Logs and Bandwidth files about every 15 minutes. More precisely, cpanellogd will attempt
to evaluate a set of logs 500 seconds (8'20") proceeding the last evaluation and parse-cycle's completion; alternating
betweenLog (domain) files andBandwidth files. See Example 1, cpanellogd Check Times. If any file has not been
parsed since "X hourly value", the file is queued for parsing.
Example 1. cpanellogdCheck Times
Server blue.example.com has the following configuration for Bandwidth and Log processing frequency:
Bandwidth: 2 Hrs.
Log: 6 Hrs.
At 14:00:00 UTC cpanellogd checks whether it needs to process any Bandwidth files. At 14:08:20 UTC cpanel-
logd checks whether it needs to process any Log files. The next check at 14:16:40 UTC, where it checks for Bandwidth
files that need processed, is approximately 15 minutes since the last check of the Bandwidth files.
1Additional cpanellogd governance is found in "Schedule Configuration >> Statistics Schedule," "Tweak Settings" and '/var/cpanel/cpanel.config'. See the
Configuration portion of this document for more detail.
7/27/2019 cPanellogd.pdf
4/14
WHM/cPanel Log Processing
4
"X" is the respective value ofLog Processing Frequency orBandwidth Processing Frequency value fields; defined in
Main >> Server Configuration >> Statistics Software Configuration >> Schedule Configuration. Once all appropriate
items are queued, the parsing cycle begins.
During a parse cycle 5 key events occur:
1. A zero length file is made in /var/cpanel/lastrun/ indicating last processing date time. See "Contents of/var/cpan-
el/lastrun" for more details.
2. All queued log files are renamed with a '.bkup' extension.
3. Apache is gracefully restarted.
4. After the Apache restart, cpanellogd forks a child process for the necessary parser.
5. Once the parser completes operation on the .bkup file, rotation needs are determined:
if the give log is scheduled for rotation, ${logfile}.bkup is prepended to the active log file,
else if ${logfile} is not scheduled for rotation, no action is taken.
6. The ${logfile}.bkup files are discarded.
Contents of /var/cpanel/lastrun
Each cPanel account has a corresponding directory in/var/cpanel/lastrun. Within the account directory
are two files used to indicate the last time cpanellogd processed log files for the account. Each file corresponds
to type of log processed.
Table 1. Last Run Flag Files
File Log Type
bandwidth bytes_log
stats The Apache access logs in /usr/local/apache/domlogs
3.1.1. Apache Restart Requirement
As noted above, an Apache restart is part of the parsing process. If none of the items queued for processing need to be
deleted or rotation, Apache will not restart.
A restart of Apache is required to prevent Apache from continuing to write the files cpanellogd will process. Without
the restart there is potential for data loss. See "File Descriptors" for a detailed explanation.
When restarting Apache, a graceful restart is requested. This type of restart typically places the least load on a server
and should maintain service up-time
2
.
After processing is complete, respective statistical information is available from the normal interfaces: Daily Process
Log, View Bandwidth Usage interfaces, or various domain statistic portals (i.e., Webalizer, AWStats, Analog).
Report Discrepancies
Any discrepancy between statistical reports will stem from:
2See Section 6.1, Apache Restarts for known issues with Apache restarts.
7/27/2019 cPanellogd.pdf
5/14
WHM/cPanel Log Processing
5
1. Timing between log/bandwidth parsings
2. Inherent differences of the report engines and the data they report
File Descriptors
When an applicationopens a file the application receives a file descriptor( hereafter FD) from the operating system.This FD is an index into a kernel-resident data structure containing the details of all open files. The common system
utility lsofallows the admin to view the list of all open files and which applications are using them.
The FD remains associated with the requested file as long as the application keeps the file open. Changes to the
underlying file have little impact on the FD or the holding application. The application only knows about the FD.
When performing certain operations on files that are opened by long running services it is often necessary to restart
the services. This prevents the situations shown in Example 2, Renaming a file with an open FD and Example 3,
Deleting a file with an open FD.
Example 2. Renaming a file with an open FD
1. Application Foo opens /etc/named.conf for reading and writing.
2. Foo receives a FD associated with the file.
3. The system admin renames /etc/named.conf to /etc/named.conf.bak
4. Foo writes new data to the FD obtained in step 1
5. The new data is added to /etc/named.conf.bak
Example 3. Deleting a file with an open FD
1. Application Foo opens /etc/named.conf for reading and writing.
2. Foo receives a FD associated with the file.
3. The system admin deletes /etc/named.conf
4. Foo writes new data to the FD obtained in step 1
5. The new data disappears as the FD no longer points to a valid file.
3.2. Rotation & Retention
Log files can be selected in the respective interface for rotation inclusion.(See Section 4.2, Log Rotation).
If a given cPanel or Apache log file is rotated, it will occur once the file exceeds 300MB (See Section 4.2, Log
Rotation for specifying a custom value). File limits are checked during each parse cycle.
7/27/2019 cPanellogd.pdf
6/14
WHM/cPanel Log Processing
6
Domain access Apache logs do not rotate by default. However, the logic and configuration that governs the retention of
this data is separate of cPanel and Apache log files. The most obvious benefit of this approach is domain users can
access and manage these raw logs. The threshold for rotation and file trimming on domain logs is set at 1GB3.
Domain users have the option to archive domain logs on a monthly basis. Additionally, they have the ability to automat-
ically remove the previous months archive.
The WHM/cPanel administrator has the ability retain domain logs on a monthly basis (See Domain Log Rotation).
Additionally, they can elect to delete domain logs after parsing completes.
Note
Deletion is negated if the administrator has also elected to keep logs each month or if the affected domain
user has elected to archive their files.
3.3. Synchronicity
Log parsers are not created equal. It is by design that different log parsers expend varying levels of system resources
determining similar, if not identical, information. Additionally, each domain experiences different traffic, generating
various length logs that must be parsed.
Depending on the number of accounts and the traffic they receive, it is possible, and at some point highly probable on
certain server setups, that parsing will not complete for all accounts within a parsing cycle. Processing domain logs is
the slowest part of the log parsing process.
The log parsers provided by cPanel include:
Analog
AWStats
Webalizer
Only the parsers enabled in WHM are active during a log processing period. The parsers are executed serially, rather than
in parallel, to reduce system load the parsers generate.
3.3.1. Simple Example
Server "blue.example.com" has 5 accounts:
1. example.com
2. ejemplo.info
3. beispiel.com
4. voorbeeld.net
5. esempio.org
The majority of "blue.example.com"'s traffic is on domain esempio.org. The first time cpanellogd processes the
domains, it does so in the following order:
1. beispiel.com
3As of 11.25.0, this threshold is fixed and cannot be configured to a different value.
7/27/2019 cPanellogd.pdf
7/14
WHM/cPanel Log Processing
7
2. ejemplo.info
3. esempio.org
4. example.com
5. voorbeeld.net
The first two domains are processed very quickly. Because esempio.org has more data, it takes many minutes to process.
When the appropriate entry is made in /var/cpanel/lastrun the timestamps for example.com and voorbeeld.net
are 'not in sync' with the first three domains because the gap between the earliest and latest time stamp will be greater
than 500 seconds.
Once the Log Processing Frequency threshold is exceeded, beispiel.com, ejemplo.info, esempio.org will be processed
but not the other two domains since they were completed sometime just within the threshold. At cpanellogd's next
domain log parse cycle, cpanellogd will now process the remaining domains.
Due to the above, there are now 2 Apache restarts, 15 minutes apart4.
4. Configuration
4.1. Cycle Frequency
Log Process Frequency
Interface WHM >> Main >> Server Configuration >> Statistic Software Configuration >>
Statistics Configuration
Description Minimum time threshold before attempting to parse Log files.
Affected Files and Directories /usr/local/apache/logs/
/home/${domainowner}/tmp/analog//home/${domainowner}/tmp/awstats/
/home/${domainowner}/tmp/webalizer/
/home/${domainowner}/tmp/webalizerftp/
Default Value 24
Units Hours
Bandwidth Process Frequency
Interface WHM >> Main >> Server Configuration >> Statistic Software Configuration >>
Statistics Configuration
Description Minimum time threshold before attempting to parse Bandwidth files.
Affected Files and Directories /usr/local/cpanel/logs/
/var/cpanel/bandwidth/
/home/${domainowner}/tmp/cpbandwidth/
/home/${domainowner}/tmp/bw-${domainowner}-7days.png
/home/${domainowner}/tmp/bw-${domainowner}-today.png
4Approximate. cpanellogd makes a processing pass every 500 seconds. Each pass alternates between domain traffic logs and bandwidth monitoring logs.
This simple example excludes any other processes influencing Apache.
7/27/2019 cPanellogd.pdf
8/14
WHM/cPanel Log Processing
8
/home/${domainowner}/tmp/bw-${domainowner}-year.png
Default Value 2
Units Hours
Black-out Hours
Interface WHM >> Main >> Server Configuration >> Statistic Software Configuration >>
Statistics Configuration >> Statistics Schedule
Description Specific hours to suspend cpanellogd operation. Any active parse cycle that
lapses into a black-out time slot will complete for the current domain being pro-
cessed; all remaining domains queued are left unprocessed until the next parse cy-
cle. Processing is suspended and black-out is honored . Once black-out period ends
cpanellogd commences with a normal parse cycle.
Affected Files and Directories
Default Value No black-outs by default
Units Hour long blocks via form checkbox.
Deference to Backup
Interface WHM >> Main >> Server Configuration >> Statistic Software Configuration >>
Statistics Configuration >> Statistics Schedule
Description If enabled cpanellogd will be suspended while backups are run. Any active
parse cycle will complete for the current domain before deference to backup is
honored. Any remaining domains queued will be processed at the next parse cycle
proceeding backups. This behavior is identical to Black-out Hours.
Affected Files and Directories /var/cpanel/cpanel.config
Default Value Disabled
Units Boolean via form checkbox.
Emulating Apache's Prior Behavior
Description Emulating Apache and log parsing behavior prior to v11.25.0 is not directly supported. However,
approximate behavior can be accomplished by configuring Black-out Hours or modifying /var/cpan-
el/lastrun/* timestamps5.
5
4.2. Log RotationApache Log Rotation
Interface WHM >> Main >> Server Configuration >> Apache Configuration >> Log Rota-
tion
Description Select which Apache log files to include in rotation.
5Modifying sub-system upkeep files is discouraged.
7/27/2019 cPanellogd.pdf
9/14
WHM/cPanel Log Processing
9
Affected Files and Directories /var/cpanel/cpanel.config
/var/cpanel/log_rotation/
Default Value Disabled
Units Boolean via form checkbox.
Domain Log Rotation
Interface WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs
Description Elect to rotate domain specific Apache log files by 'checking' "Keep log files at
the end of the month."
Important
Domain logs are hard coded to rotate at 1GB, as well as at the end
of each month.
Elect to retain data within domain Apache logs by 'unchecking' "Delete each
domain's access log after stats run."
Affected Files and Directories /var/cpanel/cpanel.config
/usr/local/apache/domlogs/
Default Value Disabled
Units Boolean via form checkbox.
cPanel Log Rotation
Interface WHM >> Main >> Server Configuration >> cPanel Log Rotation Configuration
Description Select which cPanel log files to include in rotation.
Affected Files and Directories /var/cpanel/cpanel.config
/var/cpanel/log_rotation/
Possible Side Effects See Threshold for Log Rotation
Default Value no files are included in rotation
Units Boolean via form checkbox.
Threshold for Log Rotation
Interface /var/cpanel/cpanel.config
Description Defining variable "rotatelogs_size_threshhold_in_megabytes" within
cpanel.config will determine how large cPanel or Apache logs files are allowed
to grow before rotation occurs.
Warning
This value applies to both Apache and cPanel log files but has no
affect of Domain access Apache logs
7/27/2019 cPanellogd.pdf
10/14
WHM/cPanel Log Processing
10
Note
This variable is excluded in the initial configuration of cpanel.config
and therefore must be appended manually. Additionally any value
> Main >> Server Configuration >> Tweak Settings >> Stats and Logs
Description See Log Process Frequency
Status Invalid exposure of functionality.
Schedule for deprecation 11.25.X.
Bandwidth Processing Frequency
Interface WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs
Description See Bandwidth Process Frequency
Status Invalid exposure of functionality.
Schedule for deprecation 11.25.X.
Keep Stats Log
Interface WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs
Description Elect to not purge cPanel's statistic log upon cPanel restart.
Note
cPanel log rotation also affects this file.
Affected Files and Directories /usr/local/cpanel/logs/stats_log
Default Value Disabled
Units Boolean via form checkbox.
Keep Logs
Interface WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs
Description Elect to retain domain access logs, rotating them at the end of the month.
7/27/2019 cPanellogd.pdf
11/14
WHM/cPanel Log Processing
11
Note
Rotation and retention policy is enacted anytime these logs exceed
1GB. See Domain Log Rotation.
Warning
Election to retain these logs can quickly fill disk space!
Affected Files and Directories
Default Value Disabled
Units Boolean via form checkbox.
Purge Logs
Interface WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs
Description Delete domain access logs after parsing occurs.
Note
Electing to Keep Logs will override this configuration. See Keep log
files at the end of the month.
Note
If a domain user electing to Archive Logs, it will override this con-
figuration for their affected files. See Archive Logs in your home di-
rectory at the end of each month.
Warning
Election to retain these logs can quickly fill disk space!
Affected Files and Directories
Default Value Enabled
Units Boolean via form checkbox.
4.4. cPanel Options
Archive Logs
Interface cPanel >> Logs >> Raw Access Logs >> Configure Logs
Description Archive domain access logs on a monthly basis.
Note
Purge Logs will be negated for domain user if this configuration is
enabled. See Delete each domain's access logs after stats run.
7/27/2019 cPanellogd.pdf
12/14
WHM/cPanel Log Processing
12
Affected Files and Directories
Default Value Disabled
Units Boolean via form checkbox.
Purge Archive
Interface cPanel >> Logs >> Raw Access Logs >> Configure Logs
Description Delete previous months archive on a monthly basis.
Warning
Electing to NOT Purge Archive can consume excessive disk space
and disk quota. See Delete each domain's access logs after stats run.
Affected Files and Directories
Default Value Disabled
Units Boolean via form checkbox.
4.5. Flag Files
Interface CLI
Description Zero length file that disables Bandwidth monitoring.
Flag File /etc/rrdtooldisable
Affected Files and Directories
Default Value Nonexistent
Units Zero length file.
4.6. Other Logs
N/A
5. Troubleshooting
5.1. Unprocessed .bkup files?
In the extreme event that a parser does not properly parse it's data, the result will be two-fold:
1. cpanellogd will attempt to reprocess ${logfile}.bkup.
2. If one is trying to view the report while the parser is working, a report should be rendered based on the older,
finished report, with reference to unparsed or incomplete data 6.
6
5.2. Why are all domains not processed during a specific cycle?
7/27/2019 cPanellogd.pdf
13/14
WHM/cPanel Log Processing
13
Domain logs are parsed end to end. If processing for one domain takes several minutes, it is likely that the following
domains will not complete their log parsing within the same 500 second parsing cycle; ergo, the latter domains
will receive future parsing in a cycle time different than the former domains.
6. Known Issues and Caveats
6.1. Apache Restarts
6.1.1. Apache 2.2 and Child Processes in 'G' Status
Reference Bug Apache 2.2 Child Processes [http://docs.cpanel.net/twiki/bin/vief/EasyA-
pache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_i]
Errata Apache Foundation considers this issue resolved as of Apache v2.2.12+
Status Resolved
6.1.2. Apache and Child Processes in 'K' Status
Behavior If Apache has several child processes in 'K' (KeepAlive) status, a graceful restart can take several
minutes, during which Apache will not accept any new connections.
Possible Cause KeepAlive directives in httpd.conf
Possible Fix Limit or exclude KeepAlive directives in httpd.conf.
Status Unconfirmed
7. Conclusion
WHM/cPanel 11.25.0 provides new mechanisms for maximizing server control and better resource utilization.cpan-ellogd, the daemon responsible for parsing various logs and triggering report engines, has changed internally and how
it negotiates with server processes; specifically, Apache is managed for scalability and performance. New configurations
can be managed in WHM.
GlossaryGlossary of terms used in this document and within the WHM/cPanel produce.
B
Bandwidth The measure of data that traverses a network interface.See Also Bandwidth Logs.
Bandwidth Logs Log files detailing all traffic on a per domain basis. This includes all HTTP (without
headers), HTTPS (with headers), mail and FTP transmissions.
D
Domain Access Apache Logs Access logs generated by Apache specific to a domain.
http://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_i7/27/2019 cPanellogd.pdf
14/14
WHM/cPanel Log Processing
Domain Logs See Domain Access Apache Logs.
L
Logs See Domain Access Apache Logs.