Mining that Data

Let's assume that you've generated a nice long report, using the log analysis tool of your choice. We'll go through each section of a typical traffic report, and see how many traffic-building brainstorms we can come up with.

Most Requested Pages

This section is a gold mine. Here you can see which of your pages are bringing in the most hits. Most sites will expect to see their home page at the top, followed by the home or "hub" pages of the major sections of your site.

Most log analysis programs can filter out user-specified file types for this report, and many people set things up to show only .html or .htm files, to avoid cluttering the report up with image files, scripts and so forth. Sometimes it's useful to see what's going on with those other types of files, however. For example, a WebTrends report can include a list of the most submitted forms and scripts. Looking at this info for my site, I realized that there was an old CGI script, that we thought we had retired, still linked somewhere and churning out a few impressions a day.

Looking at the relative amounts of traffic that certain pages (or directories) get, you can get an idea of what people are using your site for. Are most of your visitors coming for product support? for product information? Which products are they asking for information about? It's tempting to conclude that pages or sections that get more traffic are the ones that visitors like more, but things aren't that simple.

The amount of traffic a particular page or section gets may have more to do with the number and prominence of links that lead to it than with how much visitors like it. For example, looking at the report for my site, The Web Developer's Journal, I can see that the top five pages each generate a big percentage of the total impressions. Now, I happen to know that all five of these pages are ones that have earned a listing in Yahoo. That's why they get lots of hitskys - there are plenty of other pages that readers would probably like just as much if they were listed, too.

There's one pearl of wisdom for you, in case you didn't already know: the big directories generate lots of traffic, and Yahoo is the biggest of the big. If you don't already have a listing in Yahoo, it's well worth your time to try to get listed.

The prominence of links on your site obviously has a lot to do with the amount of traffic an interior page will get. A page that's linked from your navbar on every page, or that has a big prominent link on the home page, will get a lot more action than one that's just referenced by a hot link in some text somewhere.

Keep in mind that the popularity of an individual page can be interpreted in two opposite ways. On the one hand, if a certain type of content is generating more hits than other types, then it makes sense to go with a winner - push that type of content even more, and create new stuff in the same vein. On the other hand, if a section isn't getting much traffic, and you think it should, it might be time to push it a little more, by giving it more prominent internal links, and perhaps submitting it individually to directories and such.

Some programs can report not only the most popular pages, but things like:

Least popular pages - Remember, this doesn't mean people didn't like those pages, it means few people saw them in the first place. If this list contains pages of limited importance, that you wouldn't expect many people to visit anyway, then you're looking good. If you see important content pages on this list, you may want to take steps to steer more people toward them.

Top entry pages - Useful for finding out which pages are linked from other sites, search engines, etc.

Top exit pages - Keeping visitors on your site as long as possible is a worthy goal, especially if you're an ad-supported site. Analyzing pages that tend to make people want to leave may help you figure out how to make them a little stickier.

Single access pages - Pages that visitors access and exit without viewing any other page. These pages are entry pages to your site, but for some reason they don't entice very many people to visit other sections. If you can figure out why not, you may be able to boost your overall traffic by a hair.

The most useful insights come when you start combining the individual page traffic with other data. For example, let's say you're selling products on your site. Your traffic report tells you that your home page is getting healthy traffic, but your ordering page is getting only a tiny percentage of that. You replace the small text link to your ordering page with a big bright yellow image that flashes on and off, and says "Click here NOW! Or else!" A month later, you run another report, and find that traffic to your ordering page has skyrocketed.

When you compare the traffic data for that page with your actual orders, however, you find that only a tiny fraction of the people who visited the order page actually placed an order. That is, your "conversion rate" (the percentage of visitors converted into customers) is low. This indicates that you should direct your efforts toward making the order page better before you spend time trying to build more traffic to it.

Often tiny changes in page layout or wording can make a big difference in visitor behavior. Keep careful track of any modifications you make to your pages, then compare the dates of those changes to the dates of any increases or decreases in traffic, as revealed by your log reports.

How Log Files Work
There's gold in them there log files!
Insights from your Error Logs