How Search Engine Spiders Work
The more you understand about any subject, the more interesting
it becomes. As you read this article you'll find that the
subject of SEO is certainly no exception.
There are hundreds of search engines available today, but some
are far more complex than others. This article will give you an
overview of how some of the most popular ones work.
Let's start with a smaller engine: InfoSeek. They only index
about 200 words of your web page, so it's important to make sure
that you have meta tags on your site, and that the most
important things are listed first. The information you put in
your meta tags will be used to display a description of your
site, and most meta tags can contain about 200 characters of
text. The keywords meta tag, however, can have up to 1,000
characters.
These simple rules are important to keep in mind for all search
engines. The more important that the information is, the closer
it has to be to the beginning of your meta tags or even the
beginning of your site's content. Many search engines won't even
touch your meta tags so it is important that you have the same
information in your body that you have in your meta tags
(although you obviously cannot simply enter lists and lists of
key words as this would be detrimental to your site's
content).
The AltaVista search engine will send Scooter, its spider, to
check out your entire site. Scooter can take as long as three
months to spider and fully index your site - the average spider
only takes 6-8 weeks. Scooter will normally spider somewhere
between two and ten pages from your site each week. This means
that the longer that your web site lasts, the better it will be
indexed which is in example of how search engines implement
Darwin's Theory into their ideology.
Excite used to be a search powerhouse, but has now been dropped
as the provider of AOL and Netscape search, so it's less
important than it once was. The algorithm it uses to determine
keyword relevance is very complicated: it indexes your pages and
then attempts to summarize them by selecting only the most
relevant sentences. Expect to have your pages reviewed roughly
once every two weeks. Keep in mind, though, that with meta tags
have no meaning to Excite when it comes to rankings, even though
it will use your description tags as long as the words are
relevant to your pages' content.
See how much you can learn about SEO when you take a little time
to read a well-researched article? Don't miss out on the rest
of this great information.
Let's move on to Lycos. Lycos has fully integrated the Open
Directory Project (ODP) into their mainstream results pages, and
they also use search results from AllTheWeb. Lycos also runs
click-throughs to their sister site HotBot. Lycos is one of the
harder search engines to understand, as their submission pages
say one thing but then they index your site in a completely
different way. As a general rule of thumb, your site will be
indexed in Lycos in due time as long as you get indexed in ODP
and AllTheWeb.
Even though WebCrawler is owned by Excite, it still has its own
search engine and indexer. If you happen to be listed with
WebCrawler, you should try to stay listed with them, as it isn't
the easiest search engine to get listed with. Its hit-and-miss
standards combined with the sporadic indexing methods makes the
submission process tough, although not impossible.
The biggest player is, of course, Google, who use a page ranking
system as the central basis of their index. It was once nearly
impossible to manipulate this page ranking system to drive up
your rankings, but people quickly figured out that the more
links they could generate to their site on the rest of the net,
the better Google ranked them. Google is not thought to be using
context-sensitive rankings. Context-sensitive information is
used at Yahoo, Looksmart and the ODP, however, and Google
regularly spiders those sites when it re-indexes its own
database.
MSN is another important search engine. The holy trinity of
search engines at the moment is Google, Yahoo!, and MSN. These
three search engines combine to provide you with the vast
majority of the traffic that you will receive from search
engines. MSN will generally be the first search engine to index
your site and it will almost certainly list the most pages the
fastest.
Although no-one can tell you exactly when you will be indexed on
any search engine, it's best to check back at least weekly.
Whatever you do, though, don't re-submit your site more often
than every two months or so - you might not get indexed at all
if you do this.
Take time to consider the points presented above. What you learn
may help you overcome your hesitation to take action.
SEO Tips
All About Domain
Names
Articles for
Website Content
About the author:
James Mahony is the founder of Search Engine Optimization
Tips - A site dedicated to Search Engine Optimization
SEO Tips
All About Domain
Names
Articles for
Website Content