Minimizing Apache httpd downtime with logrotate

Armijn Hemel, April 3, 2009, 17226 views.

Tags: , ,

Our company hosts quite a few websites. Because of log reporting scripts we wanted to move from a weekly logrotate to a daily logrotate. Apart from reporting scripts we also wanted to do this because of performance. The Apache documentation recommends to rotate logs every now and then.

We use logrotate to rotate our logs. Our logrotate scripts initially were something like this:

/home/vhosts/example.org/logs/access_log
/home/vhosts/example.org/logs/error_log {
        daily
        missingok
        compress
        rotate 31
        create 640 root adm
        sharedscripts
        prerotate
                /path/to/awstats.pl -config=example.org -update > /dev/null 2>&1
        endscript
        postrotate
                /usr/sbin/service httpd restart > /dev/null
        endscript
}

If you do this for 200 domains it gets a bit painful, since you will be restarting the webserver 200 times. In our case this was not a major disaster since the script ran at 4 AM and all websites were targeted a European audience, that should have been sound asleep then.

This didn't sit well with us, so we rewrote it a bit. A newer version of the scripts did the following:

  1. stop Apache
  2. rotate and process the logs for all sites
  3. start Apache

The drawback with this is that the webserver won't be running during step 2 and clients will get a 'connection refused'. A better solution is to run a small webserver that will process requests. The hard part here is which return code to send to the client. You don't want to send the wrong code to for example a search engine, because your websites will be indexed improperly. Sending 200 (OK), 403 (Forbidden) or 404 (Not Found) is absolutely not correct to send. Luckily there is 503 ("Temporarily Unavailable"), which the most important search engines will use as a hint that they should not use the current page they get for indexing.

The process becomes a bit more complicated:

  1. stop Apache
  2. start temporary Apache
  3. rotate and process the logs for all sites
  4. stop temporary Apache
  5. start Apache

All requests will be redirected to one page. The script for this 503 page is a very simple PHP script:

<?php header("HTTP/1.0 503 Service Temporarily Unavailable");
echo "Temporarily down for maintenance";
?>

The Apache configuration looks like this:

ServerTokens Prod
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
LoadModule cgi_module modules/mod_cgi.so
LoadModule alias_module modules/mod_alias.so
# use this for Apache httpd 2.0
LoadModule access_module modules/mod_access.so
# Use the following line for Apache httpd 2.2
# LoadModule authz_host_module modules/mod_authz_host.so
LoadModule php5_module modules/libphp5.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule mime_module modules/mod_mime.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule log_config_module modules/mod_log_config.so
AddHandler php5-script .php
AddType text/html .php
RewriteEngine On
RewriteRule ^.*$ /503.php
TypesConfig /etc/mime.types
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>
Listen 80
User apache
Group apache
ServerAdmin root@localhost
UseCanonicalName Off
DocumentRoot "/etc/httpd/alt-boot"
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
ErrorLog     /var/log/httpd/alt_error_log
CustomLog    /var/log/httpd/alt_access_log combined
ScriptAlias / "/etc/httpd/alt-boot/"
<Directory "/etc/httpd/alt-boot/">
Order allow,deny
Allow from all
Options +ExecCGI
</Directory>

The shell script to start the temporary webserver looks like this:

#!/bin/sh
/sbin/service httpd stop
sleep 5
/usr/sbin/httpd -f /etc/httpd/conf/httpd-alt.conf

Finally, the shell script to restart the normal webserver looks like this:

#!/bin/sh
/sbin/service httpd restart

These scripts are run each night from /etc/cron.daily/ (we use CentOS Linux, on other operating systems this might be different).

The temporary webserver would run for 5 to 10 minutes and serve nothing but 503 pages.

A considerable speedup was achieved by running awstats just prior to starting the temporary webserver (awstats can run on live logfiles) and run awstats on the few additional requests that could have been made between running awstats and stopping the webserver.

  1. run awstats for all logs
  2. stop Apache
  3. start temporary Apache
  4. rotate and process the logs for all sites
  5. stop temporary Apache
  6. start Apache

With this setup the temporary webserver would run for about 40 to 45 seconds, which was a huge improvement.

But, it can be done even better. Log processing, no matter how few requests, can be done at any time. Logrotation in itself is fast, because you just need to move a few files, compress a few others, and touch new logfiles, after which Apache you can start Apache again. Log processing can be done afterwards. In our setup logfiles are compressed nightly. With a slight change in the configuration (reading from a pipe instead of a file) awstats can read from a compressed file.

LogFile="/bin/zcat -f /home/vhosts/example.org/logs/access_log.1.gz |"

Every night cron runs awstats with this configuration after logrotation instead of during logrotation.

The process now is:

  1. stop Apache
  2. start temporary Apache
  3. rotate logs
  4. stop temporary Apache
  5. start Apache
  6. process logs for all sites

The temporary webserver runs for about 5 seconds per night and handles at most two or three requests.

If you don't want any downtime because of rotating logs, you might want to look at piped logging. There are various solutions for Apache for that.

Social networking: Tweet this article on Twitter Pass on this article on LinkedIn Bookmark this article on Google Bookmark this article on Yahoo! Bookmark this article on Technorati Bookmark this article on Delicious Share this article on Facebook Digg this article on Digg Submit this article to Reddit Thumb this article up at StumbleUpon Submit this article to Furl

Talkback

respond to this article

Re: Minimizing Apache httpd downtime with logrotate (radoslav kolev, 2009-07-08 15:46 CEST)
Why not just make apache reopen the logs after rotation by sendiing a SIGHUP?
Re: Minimizing Apache httpd downtime with logrotate (Armijn Hemel, 2009-07-23 09:56 CEST)
> Why not just make apache reopen the logs after rotation by sendiing
> a SIGHUP?

That is, of course, also an option. We didn't explore it for unknown reasons, but we will definitely look at it in the future.
Re: Minimizing Apache httpd downtime with logrotate (logic, 2014-02-07 10:11 CET)
Instead of shutting down apache, you can just add "copytruncate" to logrotate option file path(s), and no need of downtime (you can just loose few seconds of log recording).

:?