Howto parse a Squid logfile with Awstats

Thanks

During the last couple of days a had the chance to configure squid caching on a large scaled high traffice webserver. After all was running well, traffic statistics were needed. We tried using Awstats for this on plone.org a couple of weeks ago but badly failed. Today HyperJohnG found the problem and made it possible for to find a workaround. Thanks John for making this Howto possible.

Requirements

• Squid Cache from http://www.squid-cache.org.
• Awstats from http://awstats.sourceforge.net.
• Squid with custom log format support. See http://devel.squid-cache.org/customlog.

Squid2 or Squid3

Squid2 offers no built in way to generate a awstats compatible access logfile. To make this possible a patch is needed. This patch has been totally incorporated into Squid3. So if you wanna trust Squid3 (its not final by writing this howto) you dont need any patching. If you use Squid2 you need to patch the source with this patch http://devel.squid-cache.org/old_projects.html#customlog in addition to Squid and Awstats.

Patching Squid2

Please ignore this for Squid3. For Squid2 the patch needs to be applied by using the unix patch tool.

Compile Squid

>>> ./configure; make; make install

Configure LogFormat

The patched Squid knows the logformat directive inside the squid.conf configuration File. These log format can be configured like apaches combined log format, including all information needed (referrers, client information, etc.).

Browse to the logformat section and uncomment the combined definition. Squid tells you about the default date format it writes to the logfile. This is wrong (at least for Squid2). Change the line to the following:

logformat combined %>a %ui %un [%{%d/%b/%Y:%H:%M:%S +0000}tl] \
      "%rm %ru HTTP/%rv" %Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh

Note the argument passed to the %tl variable which is a full featured strftime format string which tells squid to not use its builtin default which does not work with awstats. The +0000 is the timezone offset . Change this according your requirements. For a full list of available options see http://devel.squid-cache.org/customlog/logformat.html.

This newly defined combined logformat now has to be used for the access log by adding 'combined' after the log filename in the access_log line.

Restart Squid

After restarting squid with the changed configuration, it will write awstats compatible access_log files.