Making Your Pages Easy for Search Engines to Index

Voodoo

First, let's dispense with the zombies and headless chickens. Many companies and experts offer methods of deceiving search engine spiders to gain high rankings. And a few of these methods even work — at least for a while.

On the other side of the front line are the search engines themselves, engaged in a perpetual battle to identify these techniques, commonly known as spamming, and penalize the sites that use them because they distort search results. And of course the search engines want the sole right to do that themselves, through accepting payments for high rankings, but that's another story.

Often the techniques used by spamming experts backfire — they're spotted and punished by low rankings. Even more often, the same techniques are attempted by regular webmasters, who are not dedicated search engine experts and can't keep up with the progress of the battle, so they get wiped out soon after the opening credits. As a small example, just renaming your pages with spider-friendly filenames, without changing their content, can get them demoted in the rankings.

The easiest way to avoid this virtual war is to step aside and provide the search engines with what they want and observe their rules. Few sites manage to do this, and yet it's both effective and "legal".

The Time Element

There will be no instant results. Getting a good search engine ranking takes time. That's one of the best ways to identify charlatans from real experts. Anybody who promises instant gratification is in the voodoo business.

Search engines do not publish their precise methods of working, but most anecdotal evidence points to the importance of time. They simply do not trust new sites to deliver the goods. Also the engines are oversubscribed and have too many sites to index, so they're prone to dropping new sites from their listings on the grounds that many young sites are destined not to reach maturity, so it would be silly to take them too seriously while they're still in diapers.

You need to allow six months to a year for a decent search engine strategy to work. It will then continue to work for a long time, with minimal effort. But in the first few months it might not be a star performer. If your site doesn't provide what visitors want to see then it might never perform at all, but once again, that's a different story.

During the first year of promoting a new site, you may have to resubmit your site a number of times because it's repeatedly dropped by the search engines. That's fine. It's part of the game. They're just checking that you're serious.

You may also experience long delays between submission and actual listing. Look at the details provided by the search engine when you make your submission — often they'll tell you how long the delay will be. Allow a few weeks extra, to be on the safe side, before resubmitting.

All your submissions should be done manually, and it's a good idea to keep a record of them so you don't bug specific engines too often. If you do, you will be penalized. Automatic submission systems ("With one keystroke register your site with 1,500 search engines!") are for suckers. The search engines quickly identify them and ignore them, or, worse still, punish the sites that use them.

Spiders are Machines

Spiders (or robots) are software programs the search engine companies create to trawl the Web and index sites. They create massive databases that the engines then use to return search results.

They follow rules of logic, impeccably, and have no flexibility. They have no idea what your site really looks like, nor do they have a sense of humor. It's highly unlikely that a real human will look at your site as part of the indexing process. The exception is when you apply for a listing with a directory such as Yahoo!.

When designing your site, it's important to remember that it will be read by machines. This means, for example, if you turn all your major page headings into graphics, the spiders won't be able to recognize where your main heading are and identify the core text areas that follow, even though this would pose no problem for a human viewer.

On the other hand, if you go all out for machine-readability, you may well get the thumbs-down from a Yahoo reviewer, who has no interest in how your pages appear to a machine. It's all a question of balance.

Text - Page 2