Software for Webserver Log Analysis? [closed]

Question

Can I get some recommendations (preferably with some reasons) for good log analysis software for Apache 2.2 access log files?

I have heard of Webalizer and AWStats, but have never really used any of them, and would like to know:

• What they can do
• Why they are useful
• Interesting uses for them

Any and all comments and thoughts are welcome.

Istanbul

Here's a relatively recent analysis of on premises web analytics software, all of which do log analysis: ----- analyticsmarket.com/blog/website-analytics-software-review – Ceremonial

Answer 1

AWStats and Webalizer are both good and free (I think both free speech as well as free beer). I generally prefer the look of AWStats - it has a nice modern look whereas Webalizer looks like something created in about 1992.

They both give roughly the same information which includes:

• Most frequently accessed pages
• Which hosts (IPs and Domain Names) visitors come from
• Proportion of users using different browsers
• Proportion of downloads of different file types

All of this information is usually viewable on a hour by hour, day by day, month by month and year by year basis. Normally the raw data is available but also with bar charts and pie charts. Both AWStats and Webalizer will (I think) try and work out where your visitors come from by using services such as GeoIP, although I never bothered to set this up. Some also try to work out what order people have visited pages in and things like that - but that is very difficult to do so the results are guesses at the best.

I generally find them both useful - even if just to get an overview of what is going on with my server and who is accessing it. They are both relatively easy to install - although I seem to remember Webalizer being a little easier than AWStats, and they both have varied configuration options to let you decide exactly what you want to get out of them.

For more information see their sites at awstats.sourceforge.net/ and http://www.webalizer.org/.

Hope that helps.

Robin

Brotherinlaw

Answer 2

I use Analog because it's free, but it is rather dated now (wow, last release was 4 years ago!) and I'm sure it doesn't have as many fancy features as newer ones.

Crinoid

Answer 3

Splunk > is great, and free. It allows you to visualize and search all of your logs in real time. And it's all web based, so you can view your logs from just about anywhere.

Etching

Wow! that's an amazingly powerful tool! – Ogpu

Splunk doesn't do a good job with sessionization (i.e. visitors/sessions/pageviews). It's a good hit counter, that's about it. – Ceremonial

Answer 4

I'd go for awstats. I was getting my site usage reports from awstats for a while and then turned on Google Analytics to do this job as well. And, surprisingly, Google turned out to be rather unreliable. I get usually have no visits from a country like Finland and once I knew for sure a friend of mine visited the site from Finland and what happend? Awstats says - hey, you had a visit from Finland! And Google Analytics? I move my mouse over Finland and the bubble says "visits: 0". After that I could never quite make myself trust it again.

Rosenzweig

Since Google relies on javascript I've noticed on many occasions that it does not track a certain amount of users. Typically due to script blocking or adblock in firefox. Whereas serverside logging tools always capture these hits. I like the google analytics but I think it would be wise to cover yourself with 2 metric frameworks if you can for a more complete picture. – Piave

Answer 5

It seems that as javascript page-tagging becomes the more popular choice as a way of processing web stats, there's not as much work being done on log-based analysis tools anymore in the marketplace. My office used to use a product called LiveStats.XSP. It wasn't the greatest tool by any means, but it did have some nice features. It was recently bought by Microsoft and is no longer supported however. It abandoned log analysis turned into a proposed Google Analytics killer called Microsoft Gatineau, which supposedly is good at determining the demographics of your visitors, including age and gender (yeah sure...)

When I was looking for log analysis software a while ago, I wanted to avoid anything that looked overly bloated and enterprisey, which is what most stuff seemed to be, focusing more on the marketing and advertising aspects of reports.

One thing you may want to look at is the new version of Urchin, Urchin6 (see features here). Urchin I believe was bought by Google a few years ago. It's offered as a locally installed solution, and with it you have the option to use either page-tagging or log file analysis for any site that it monitors. There also seems to be some interface ties between Google's own web-based Google Analytics and Urchin. It's not free though, unfortunately, and I think you can only get it through authorized partners.

It does all the standard logfile analysis stuff, everything is browser-based, the reports it offers are pretty deep and comprehensive, and it also seems to have a few bells and whistles that other services don't offer. For example, I remember it being able to present a view of a web page it tracks with colored hot spots overlayed on top of it, based on how often users click on items on that area of the page. Worth checking out the demo of it anyways.

Standard

Urchin has now been discontinued... – Antipathy

Answer 6

I get Awstats and Webalizer with my web hosting account and I find that neither is accurate or very useful. The reported numbers are inflated by up to 1000%, because the tools don't properly identify bots and spiders. Here is a comparison of the Visits metric between three tools over the past 3 weeks (I think Awstats has only partial data for 3/23, and no data for today 3/24, which is why I did not include the numbers).

                            Google  Woopra  Awstats
Sunday, March 1, 2009           10      11      69
Monday, March 2, 2009           13      14      85
Tuesday, March 3, 2009          13      14      96
Wednesday, March 4, 2009        21      28      91
Thursday, March 5, 2009         19      25     107
Friday, March 6, 2009           12      10      88
Saturday, March 7, 2009         12      14     100
Sunday, March 8, 2009           10      11      65
Monday, March 9, 2009           13      14      78
Tuesday, March 10, 2009         17      13      96
Wednesday, March 11, 2009       18      16      87
Thursday, March 12, 2009        19      18      87
Friday, March 13, 2009          12      13      66
Saturday, March 14, 2009        11       7      52
Sunday, March 15, 2009          11      12      57
Monday, March 16, 2009          13      15      92
Tuesday, March 17, 2009         24      22     102
Wednesday, March 18, 2009       18      16      79
Thursday, March 19, 2009        17      18      73
Friday, March 20, 2009          16      11      70
Saturday, March 21, 2009        24      26      67
Sunday, March 22, 2009         103     114     216
Monday, March 23, 2009         232     223     117

I personally prefer Woopra over Google. While it is still in Beta, it can take a long time for your site to get approved, and it will probably be a paid service at some point, the real-time monitoring capabilities are amazing. The new custom reporting capabilities on Google Analytics are superior to Woopra, though. Woopra does not have any capabilties to produce printed reports

Emancipated

Answer 7

My old company always used WebLog Expert. There is a free 'lite' version. Its still has on-going development and can be combined with geo-location databases if you use one of the paid versions.

Apartment

Answer 8

I typically do a search every 30 or so days for new projects for better stats. I currently (still) use AWStats (and JAWStats as I mentioned in another thread).

One of the best things about AWStats when compared to it counterparts is ability to break up your datafiles to create reports by day, week, month, year or even hour using the databasebreak feature.

Why people are still using Webalizer and Analog (well, speed aside) is beyond me. At least in my experience they tend to bury as much information as they expose and I end up reading raw logs.

Homopolar

Answer 9

Try out XpoLog log analysis for web servers it provide log analysis platform to anlyze multiple web and application servers for centrelized location and create automatic reports and monitors for applications health. find it at XpoLog log analysis

Folsom

Answer 10

If you are looking for stats that are meant to be given to a customer and provide them with realistic data (not number of hits based on every file), then check out Google Analytics. Free and looks pretty.

Feinstein

Google Analytics doesn't analyse log files. – Yokoyama

Agreed, it does not, but if you want to give your customer realistic stats, it's a much better method than using log files as your customer typically doesn't care about the number of hits to their CSS files. – Feinstein