cPanellogd.pdf

7/27/2019 cPanellogd.pdf

1/14

1

WHM/cPanel Log ProcessingCopyright 2010 cPanel, Inc.

Revision History

Revision 3 2010-01-05 dn

Revision 4 2010-01-05 kpRevision 5 2010-01-07 kpRevision 6 2010-01-10 kpRevision 7 2010-01-11 kp

Abstract

This document details the WHM/cPanel log processing related to Apache and WHM Accounts. Particular attention is paid to

version 11.25.0 which introduced interface and behavioral changes.

Versions >= 11.25.0

Table of Contents

1. Just the Facts ........................ ................................................... ........................................................ .. 1

2. Overview ............................ ............................................................................................................... 2

2.1. cpanellogd ..................... .................................................... ................................................ 2

2.2. Version 11.25.0 ........................................................................................................................ 2

3. How Log Processing Works .............................................................. ................................................... 3

3.1. Statistics Generation ................................................................... .............................................. 3

3.2. Rotation & Retention ................................................ ........................................................ ........ 5

3.3. Synchronicity ........................................................................................................................... 6

4. Configuration ...................................... ....................................................... ........................................ 7

4.1. Cycle Frequency ......................................................................... ............................................. 7

4.2. Log Rotation ............................................................................. .............................................. 8

4.3. WHM Tweak Settings ......................................................... .................................................... 10

4.4. cPanel Options . ...... ..... ........................................................................................................ 11

4.5. Flag Files ...................................................... ....................................................... ................. 12

4.6. Other Logs .......................................................... ........................................................ .......... 12

5. Troubleshooting ...................................................... ....................................................... ................... 12

6. Known Issues and Caveats .. ........................................................ ....................................................... 13

6.1. Apache Restarts ................................................. ........................................................ ............. 13

7. Conclusion .................................................. ........................................................ ............................. 13

Glossary ................................................................... ........................................................ ................... 13

1. Just the FactsQ: Why is Apache restarting so often?

A: Depending upon the log processing configuration cpanellogd will restart Apache to preserve data integrity

within the account domain and bytes logs. See Section 3.1.1, Apache Restart Requirement

Q: How can I stop or lower the frequency of Apache restarts?

A: There are several ways to do this:


2/14

WHM/cPanel Log Processing

2

Reduce the frequency of log processing

Use the following values for the Schedule Configuration section of the Statistics Software Configuration

interface within WHM:

Process log files every 24 hours.

Process bandwidth every 8 hours.

Modify the timestamps of the files in /var/cpanel/lastrun. See "Contents of/var/cpanel/las-

trun" for more information..

Change the following:

Uncheck the Tweak Setting labeledDelete each domain's access logs after stats run

Caution

Leaving the log files in place may cause disk utilization issues, especially on systems that cannot

handle files larger than 2 GB

Disable log archiving. This is something the end user may configure.

Configure Log Processing Blackout hours ( see Black-out Hours ) to prevent log processing during busy traffic

periods.

2. Overview

2.1. cpanellogd

WHM/cPanel processing related to the domain logs and bandwidth logs for Accounts, as well as service level logs for

cPanel and Apache managed by the cpanellogd daemon.cpanellogd, in all versions of WHM/cPanel, is respon-

sible for tracking log file size, rotation and parsing frequency. Any utility available to affect cpanellogd's behavior is

exposed in the web interface as a value field, checkbox, or note for custom configuration.

2.2. Version 11.25.0

The Web Host Manager and cPanel suite version 11.25.0 processes statistics in a different, more robust way than past

versions. At a glance one will find two immediate differences:

1. A new configuration setting in Statistics Software Configuration within the WHM interface.

2. Periodic entries in the Apache log that Graceful restart[s] were requested and performed.

These differences highlight a new approach to statistics processing that aims to reduce processing time, CPU demand,and increase scalability.

2.2.1. Interface

Statistics processing is scheduled in WHM via Statistics Software Configuration under Server Configuration in the side-

bar. Previous versions of the WHM interface had a Statistics Frequency hourly value field only. Now the Schedule Con-

figuration subsection has two hourly value fields,Log Processing Frequency andBandwidth Processing Frequency. The

differences and affect of these values are detailed below.


3/14


3

Note

These value fields are mirrored in 'Main >> Server Configuration >> Tweak Settings >> Stats and Logs.'

However, since these value fields are most appropriately placed in the "Schedule Configuration" interface,

it should be inferred such mirroring is only present for legacy purposes and is likely to be deprecated.

Note

More interface changes are expected as respective functionality is refactored in future versions.

2.2.2. Apache Behavior

Log processing is determined by the combination of several values within cPanel. Optimal, balanced performance from

v11.25.0 can often result in Apache restarts approximately every 15 mins. Details on frequency and the affecting factors

are outlined below.

3. How Log Processing Works

cPanel will manage logs related to statistical information as well as the Apache logs. This consists of statistical log

parsing, rotation, archiving and deletion.

All other system or sub-system logs (i.e., messages, maillog, etc.) are beyond the of scope for this document.

3.1. Statistics Generation

cpanellogd triggers the log parsers which, in turn, report data forDaily Process Log, View Bandwidth Usage and the

various report engines available at the account & domain level.

Triggers are governed, in part1, by the values set in WHM's Statistics Software Configuration. Hourly values defined

in Log and Bandwidth Frequency fields do not determine the "run" frequency of cpanellogd, but instead determines the

threshold of time that respective log files are allowed to lapse between parsing.

cpanellogd checks the Logs and Bandwidth files about every 15 minutes. More precisely, cpanellogd will attempt

to evaluate a set of logs 500 seconds (8'20") proceeding the last evaluation and parse-cycle's completion; alternating

betweenLog (domain) files andBandwidth files. See Example 1, cpanellogd Check Times. If any file has not been

parsed since "X hourly value", the file is queued for parsing.

Example 1. cpanellogdCheck Times

Server blue.example.com has the following configuration for Bandwidth and Log processing frequency:

Bandwidth: 2 Hrs.

Log: 6 Hrs.

At 14:00:00 UTC cpanellogd checks whether it needs to process any Bandwidth files. At 14:08:20 UTC cpanel-

logd checks whether it needs to process any Log files. The next check at 14:16:40 UTC, where it checks for Bandwidth

files that need processed, is approximately 15 minutes since the last check of the Bandwidth files.

1Additional cpanellogd governance is found in "Schedule Configuration >> Statistics Schedule," "Tweak Settings" and '/var/cpanel/cpanel.config'. See the

Configuration portion of this document for more detail.


4/14


4

"X" is the respective value ofLog Processing Frequency orBandwidth Processing Frequency value fields; defined in

Main >> Server Configuration >> Statistics Software Configuration >> Schedule Configuration. Once all appropriate

items are queued, the parsing cycle begins.

During a parse cycle 5 key events occur:

1. A zero length file is made in /var/cpanel/lastrun/ indicating last processing date time. See "Contents of/var/cpan-

el/lastrun" for more details.

2. All queued log files are renamed with a '.bkup' extension.

3. Apache is gracefully restarted.

4. After the Apache restart, cpanellogd forks a child process for the necessary parser.

5. Once the parser completes operation on the .bkup file, rotation needs are determined:

if the give log is scheduled for rotation, ${logfile}.bkup is prepended to the active log file,

else if ${logfile} is not scheduled for rotation, no action is taken.

6. The ${logfile}.bkup files are discarded.

Contents of /var/cpanel/lastrun

Each cPanel account has a corresponding directory in/var/cpanel/lastrun. Within the account directory

are two files used to indicate the last time cpanellogd processed log files for the account. Each file corresponds

to type of log processed.

Table 1. Last Run Flag Files

File Log Type

bandwidth bytes_log

stats The Apache access logs in /usr/local/apache/domlogs

3.1.1. Apache Restart Requirement

As noted above, an Apache restart is part of the parsing process. If none of the items queued for processing need to be

deleted or rotation, Apache will not restart.

A restart of Apache is required to prevent Apache from continuing to write the files cpanellogd will process. Without

the restart there is potential for data loss. See "File Descriptors" for a detailed explanation.

When restarting Apache, a graceful restart is requested. This type of restart typically places the least load on a server

and should maintain service up-time

2

.

After processing is complete, respective statistical information is available from the normal interfaces: Daily Process

Log, View Bandwidth Usage interfaces, or various domain statistic portals (i.e., Webalizer, AWStats, Analog).

Report Discrepancies

Any discrepancy between statistical reports will stem from:

2See Section 6.1, Apache Restarts for known issues with Apache restarts.


5/14


5

1. Timing between log/bandwidth parsings

2. Inherent differences of the report engines and the data they report

File Descriptors

When an applicationopens a file the application receives a file descriptor( hereafter FD) from the operating system.This FD is an index into a kernel-resident data structure containing the details of all open files. The common system

utility lsofallows the admin to view the list of all open files and which applications are using them.

The FD remains associated with the requested file as long as the application keeps the file open. Changes to the

underlying file have little impact on the FD or the holding application. The application only knows about the FD.

When performing certain operations on files that are opened by long running services it is often necessary to restart

the services. This prevents the situations shown in Example 2, Renaming a file with an open FD and Example 3,

Deleting a file with an open FD.

Example 2. Renaming a file with an open FD

1. Application Foo opens /etc/named.conf for reading and writing.

2. Foo receives a FD associated with the file.

3. The system admin renames /etc/named.conf to /etc/named.conf.bak

4. Foo writes new data to the FD obtained in step 1

5. The new data is added to /etc/named.conf.bak

Example 3. Deleting a file with an open FD

1. Application Foo opens /etc/named.conf for reading and writing.

2. Foo receives a FD associated with the file.

3. The system admin deletes /etc/named.conf

4. Foo writes new data to the FD obtained in step 1

5. The new data disappears as the FD no longer points to a valid file.

3.2. Rotation & Retention

Log files can be selected in the respective interface for rotation inclusion.(See Section 4.2, Log Rotation).

If a given cPanel or Apache log file is rotated, it will occur once the file exceeds 300MB (See Section 4.2, Log

Rotation for specifying a custom value). File limits are checked during each parse cycle.


6/14


6

Domain access Apache logs do not rotate by default. However, the logic and configuration that governs the retention of

this data is separate of cPanel and Apache log files. The most obvious benefit of this approach is domain users can

access and manage these raw logs. The threshold for rotation and file trimming on domain logs is set at 1GB3.

Domain users have the option to archive domain logs on a monthly basis. Additionally, they have the ability to automat-

ically remove the previous months archive.

The WHM/cPanel administrator has the ability retain domain logs on a monthly basis (See Domain Log Rotation).

Additionally, they can elect to delete domain logs after parsing completes.

Note

Deletion is negated if the administrator has also elected to keep logs each month or if the affected domain

user has elected to archive their files.

3.3. Synchronicity

Log parsers are not created equal. It is by design that different log parsers expend varying levels of system resources

determining similar, if not identical, information. Additionally, each domain experiences different traffic, generating

various length logs that must be parsed.

Depending on the number of accounts and the traffic they receive, it is possible, and at some point highly probable on

certain server setups, that parsing will not complete for all accounts within a parsing cycle. Processing domain logs is

the slowest part of the log parsing process.

The log parsers provided by cPanel include:

Analog

AWStats

Webalizer

Only the parsers enabled in WHM are active during a log processing period. The parsers are executed serially, rather than

in parallel, to reduce system load the parsers generate.

3.3.1. Simple Example

Server "blue.example.com" has 5 accounts:

1. example.com

2. ejemplo.info

3. beispiel.com

4. voorbeeld.net

5. esempio.org

The majority of "blue.example.com"'s traffic is on domain esempio.org. The first time cpanellogd processes the

domains, it does so in the following order:

1. beispiel.com

3As of 11.25.0, this threshold is fixed and cannot be configured to a different value.


7/14


7

2. ejemplo.info

3. esempio.org

4. example.com

5. voorbeeld.net

The first two domains are processed very quickly. Because esempio.org has more data, it takes many minutes to process.

When the appropriate entry is made in /var/cpanel/lastrun the timestamps for example.com and voorbeeld.net

are 'not in sync' with the first three domains because the gap between the earliest and latest time stamp will be greater

than 500 seconds.

Once the Log Processing Frequency threshold is exceeded, beispiel.com, ejemplo.info, esempio.org will be processed

but not the other two domains since they were completed sometime just within the threshold. At cpanellogd's next

domain log parse cycle, cpanellogd will now process the remaining domains.

Due to the above, there are now 2 Apache restarts, 15 minutes apart4.

4. Configuration

4.1. Cycle Frequency

Log Process Frequency

Interface WHM >> Main >> Server Configuration >> Statistic Software Configuration >>

Statistics Configuration

Description Minimum time threshold before attempting to parse Log files.

Affected Files and Directories /usr/local/apache/logs/

/home/${domainowner}/tmp/analog//home/${domainowner}/tmp/awstats/

/home/${domainowner}/tmp/webalizer/

/home/${domainowner}/tmp/webalizerftp/

Default Value 24

Units Hours

Bandwidth Process Frequency


Statistics Configuration

Description Minimum time threshold before attempting to parse Bandwidth files.

Affected Files and Directories /usr/local/cpanel/logs/

/var/cpanel/bandwidth/

/home/${domainowner}/tmp/cpbandwidth/

/home/${domainowner}/tmp/bw-${domainowner}-7days.png

/home/${domainowner}/tmp/bw-${domainowner}-today.png

4Approximate. cpanellogd makes a processing pass every 500 seconds. Each pass alternates between domain traffic logs and bandwidth monitoring logs.

This simple example excludes any other processes influencing Apache.


8/14


8

/home/${domainowner}/tmp/bw-${domainowner}-year.png

Default Value 2

Units Hours

Black-out Hours


Statistics Configuration >> Statistics Schedule

Description Specific hours to suspend cpanellogd operation. Any active parse cycle that

lapses into a black-out time slot will complete for the current domain being pro-

cessed; all remaining domains queued are left unprocessed until the next parse cy-

cle. Processing is suspended and black-out is honored . Once black-out period ends

cpanellogd commences with a normal parse cycle.

Affected Files and Directories

Default Value No black-outs by default

Units Hour long blocks via form checkbox.

Deference to Backup


Statistics Configuration >> Statistics Schedule

Description If enabled cpanellogd will be suspended while backups are run. Any active

parse cycle will complete for the current domain before deference to backup is

honored. Any remaining domains queued will be processed at the next parse cycle

proceeding backups. This behavior is identical to Black-out Hours.

Affected Files and Directories /var/cpanel/cpanel.config

Default Value Disabled

Units Boolean via form checkbox.

Emulating Apache's Prior Behavior

Description Emulating Apache and log parsing behavior prior to v11.25.0 is not directly supported. However,

approximate behavior can be accomplished by configuring Black-out Hours or modifying /var/cpan-

el/lastrun/* timestamps5.

5

4.2. Log RotationApache Log Rotation

Interface WHM >> Main >> Server Configuration >> Apache Configuration >> Log Rota-

tion

Description Select which Apache log files to include in rotation.

5Modifying sub-system upkeep files is discouraged.


9/14


9


/var/cpanel/log_rotation/



Domain Log Rotation

Interface WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs

Description Elect to rotate domain specific Apache log files by 'checking' "Keep log files at

the end of the month."

Important

Domain logs are hard coded to rotate at 1GB, as well as at the end

of each month.

Elect to retain data within domain Apache logs by 'unchecking' "Delete each

domain's access log after stats run."


/usr/local/apache/domlogs/



cPanel Log Rotation

Interface WHM >> Main >> Server Configuration >> cPanel Log Rotation Configuration

Description Select which cPanel log files to include in rotation.


/var/cpanel/log_rotation/

Possible Side Effects See Threshold for Log Rotation

Default Value no files are included in rotation


Threshold for Log Rotation

Interface /var/cpanel/cpanel.config

Description Defining variable "rotatelogs_size_threshhold_in_megabytes" within

cpanel.config will determine how large cPanel or Apache logs files are allowed

to grow before rotation occurs.

Warning

This value applies to both Apache and cPanel log files but has no

affect of Domain access Apache logs


10/14


10

Note

This variable is excluded in the initial configuration of cpanel.config

and therefore must be appended manually. Additionally any value

> Main >> Server Configuration >> Tweak Settings >> Stats and Logs

Description See Log Process Frequency

Status Invalid exposure of functionality.

Schedule for deprecation 11.25.X.

Bandwidth Processing Frequency


Description See Bandwidth Process Frequency

Status Invalid exposure of functionality.

Schedule for deprecation 11.25.X.

Keep Stats Log


Description Elect to not purge cPanel's statistic log upon cPanel restart.

Note

cPanel log rotation also affects this file.

Affected Files and Directories /usr/local/cpanel/logs/stats_log



Keep Logs


Description Elect to retain domain access logs, rotating them at the end of the month.


11/14


11

Note

Rotation and retention policy is enacted anytime these logs exceed

1GB. See Domain Log Rotation.

Warning

Election to retain these logs can quickly fill disk space!




Purge Logs


Description Delete domain access logs after parsing occurs.

Note

Electing to Keep Logs will override this configuration. See Keep log

files at the end of the month.

Note

If a domain user electing to Archive Logs, it will override this con-

figuration for their affected files. See Archive Logs in your home di-

rectory at the end of each month.

Warning

Election to retain these logs can quickly fill disk space!


Default Value Enabled


4.4. cPanel Options

Archive Logs

Interface cPanel >> Logs >> Raw Access Logs >> Configure Logs

Description Archive domain access logs on a monthly basis.

Note

Purge Logs will be negated for domain user if this configuration is

enabled. See Delete each domain's access logs after stats run.


12/14


12




Purge Archive

Interface cPanel >> Logs >> Raw Access Logs >> Configure Logs

Description Delete previous months archive on a monthly basis.

Warning

Electing to NOT Purge Archive can consume excessive disk space

and disk quota. See Delete each domain's access logs after stats run.




4.5. Flag Files

Interface CLI

Description Zero length file that disables Bandwidth monitoring.

Flag File /etc/rrdtooldisable


Default Value Nonexistent

Units Zero length file.

4.6. Other Logs

N/A

5. Troubleshooting

5.1. Unprocessed .bkup files?

In the extreme event that a parser does not properly parse it's data, the result will be two-fold:

1. cpanellogd will attempt to reprocess ${logfile}.bkup.

2. If one is trying to view the report while the parser is working, a report should be rendered based on the older,

finished report, with reference to unparsed or incomplete data 6.

6

5.2. Why are all domains not processed during a specific cycle?


13/14


13

Domain logs are parsed end to end. If processing for one domain takes several minutes, it is likely that the following

domains will not complete their log parsing within the same 500 second parsing cycle; ergo, the latter domains

will receive future parsing in a cycle time different than the former domains.

6. Known Issues and Caveats

6.1. Apache Restarts

6.1.1. Apache 2.2 and Child Processes in 'G' Status

Reference Bug Apache 2.2 Child Processes [http://docs.cpanel.net/twiki/bin/vief/EasyA-

pache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_i]

Errata Apache Foundation considers this issue resolved as of Apache v2.2.12+

Status Resolved

6.1.2. Apache and Child Processes in 'K' Status

Behavior If Apache has several child processes in 'K' (KeepAlive) status, a graceful restart can take several

minutes, during which Apache will not accept any new connections.

Possible Cause KeepAlive directives in httpd.conf

Possible Fix Limit or exclude KeepAlive directives in httpd.conf.

Status Unconfirmed

7. Conclusion

WHM/cPanel 11.25.0 provides new mechanisms for maximizing server control and better resource utilization.cpan-ellogd, the daemon responsible for parsing various logs and triggering report engines, has changed internally and how

it negotiates with server processes; specifically, Apache is managed for scalability and performance. New configurations

can be managed in WHM.

GlossaryGlossary of terms used in this document and within the WHM/cPanel produce.

B

Bandwidth The measure of data that traverses a network interface.See Also Bandwidth Logs.

Bandwidth Logs Log files detailing all traffic on a per domain basis. This includes all HTTP (without

headers), HTTPS (with headers), mail and FTP transmissions.

D

Domain Access Apache Logs Access logs generated by Apache specific to a domain.
http://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_ihttp://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_i


14/14


Domain Logs See Domain Access Apache Logs.

L

Logs See Domain Access Apache Logs.

cPanellogd.pdf

Documents

Transcript of cPanellogd.pdf