Related articles
Minimizing Apache httpd downtime with logrotate
Armijn Hemel,
April 3, 2009,
7251 views.
Tags: administration, apache, linux
Our company hosts quite a few websites. Because of log reporting scripts we wanted to move from a weekly logrotate to a daily logrotate. Apart from reporting scripts we also wanted to do this because of performance. The Apache documentation recommends to rotate logs every now and then.
We use logrotate to rotate our logs. Our logrotate scripts initially were something like this:
/home/vhosts/example.org/logs/access_log
/home/vhosts/example.org/logs/error_log {
daily
missingok
compress
rotate 31
create 640 root adm
sharedscripts
prerotate
/path/to/awstats.pl -config=example.org -update > /dev/null 2>&1
endscript
postrotate
/usr/sbin/service httpd restart > /dev/null
endscript
}
If you do this for 200 domains it gets a bit painful, since you will be restarting the webserver 200 times. In our case this was not a major disaster since the script ran at 4 AM and all websites were targeted a European audience, that should have been sound asleep then.
This didn't sit well with us, so we rewrote it a bit. A newer version of the scripts did the following:
- stop Apache
- rotate and process the logs for all sites
- start Apache
The drawback with this is that the webserver won't be running during step 2 and clients will get a 'connection refused'. A better solution is to run a small webserver that will process requests. The hard part here is which return code to send to the client. You don't want to send the wrong code to for example a search engine, because your websites will be indexed improperly. Sending 200 (OK), 403 (Forbidden) or 404 (Not Found) is absolutely not correct to send. Luckily there is 503 ("Temporarily Unavailable"), which the most important search engines will use as a hint that they should not use the current page they get for indexing.
The process becomes a bit more complicated:
- stop Apache
- start temporary Apache
- rotate and process the logs for all sites
- stop temporary Apache
- start Apache
All requests will be redirected to one page. The script for this 503 page is a very simple PHP script:
<?php header("HTTP/1.0 503 Service Temporarily Unavailable");
echo "Temporarily down for maintenance";
?>
The Apache configuration looks like this:
ServerTokens Prod
ServerRoot "/etc/httpd"
PidFile run/httpd.pid
Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
LoadModule cgi_module modules/mod_cgi.so
LoadModule alias_module modules/mod_alias.so
# use this for Apache httpd 2.0
LoadModule access_module modules/mod_access.so
# Use the following line for Apache httpd 2.2
# LoadModule authz_host_module modules/mod_authz_host.so
LoadModule php5_module modules/libphp5.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule mime_module modules/mod_mime.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule log_config_module modules/mod_log_config.so
AddHandler php5-script .php
AddType text/html .php
RewriteEngine On
RewriteRule ^.*$ /503.php
TypesConfig /etc/mime.types
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
Listen 80
User apache
Group apache
ServerAdmin root@localhost
UseCanonicalName Off
DocumentRoot "/etc/httpd/alt-boot"
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
ErrorLog /var/log/httpd/alt_error_log
CustomLog /var/log/httpd/alt_access_log combined
ScriptAlias / "/etc/httpd/alt-boot/"
<Directory "/etc/httpd/alt-boot/">
Order allow,deny
Allow from all
Options +ExecCGI
</Directory>
The shell script to start the temporary webserver looks like this:
#!/bin/sh /sbin/service httpd stop sleep 5 /usr/sbin/httpd -f /etc/httpd/conf/httpd-alt.conf
Finally, the shell script to restart the normal webserver looks like this:
#!/bin/sh /sbin/service httpd restart
These scripts are run each night from /etc/cron.daily/ (we use CentOS Linux, on other operating systems this might be different).
The temporary webserver would run for 5 to 10 minutes and serve nothing but 503 pages.
A considerable speedup was achieved by running awstats just prior to starting the temporary webserver (awstats can run on live logfiles) and run awstats on the few additional requests that could have been made between running awstats and stopping the webserver.
- run awstats for all logs
- stop Apache
- start temporary Apache
- rotate and process the logs for all sites
- stop temporary Apache
- start Apache
With this setup the temporary webserver would run for about 40 to 45 seconds, which was a huge improvement.
But, it can be done even better. Log processing, no matter how few requests, can be done at any time. Logrotation in itself is fast, because you just need to move a few files, compress a few others, and touch new logfiles, after which Apache you can start Apache again. Log processing can be done afterwards. In our setup logfiles are compressed nightly. With a slight change in the configuration (reading from a pipe instead of a file) awstats can read from a compressed file.
LogFile="/bin/zcat -f /home/vhosts/example.org/logs/access_log.1.gz |"
Every night cron runs awstats with this configuration after logrotation instead of during logrotation.
The process now is:
- stop Apache
- start temporary Apache
- rotate logs
- stop temporary Apache
- start Apache
- process logs for all sites
The temporary webserver runs for about 5 seconds per night and handles at most two or three requests.
If you don't want any downtime because of rotating logs, you might want to look at piped logging. There are various solutions for Apache for that.
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Netherlands License.
