Obviously you will want to monitor your investment of time and money in

getting high search engine rankings so you can see if it’s worth your time

and if you need to make any changes. This section will show you how to be

a crack search engine analyst.

The effectiveness of your efforts in submitting your pages for listing on

search engines can be monitored and evaluated by two methods: spider

spotting and URL check.

Spiders from search engines that visit your site and crawl pages leave some

unique trace marks in your access log. This can tell you whether a spider has

visited or not, what pages they have visited and also the frequency or

duration of their visit

The Best Way To Track
The best way to identify spider visits is by finding out which visitors asked
for the file robots.txt from your site. Only spiders make such a request, as
this file is an indication to them to avoid covering the page in question. So
the first thing a crawler would do is to check for this file. If you see the
access log and analyze it using some convenient software, you would be able
to spot all the visits that were initiated with this request. Then one can spot
the host name and relate that to major search engines. Host names are
135
related to the search engine company’s name (it is the name of the site that
hosts the spider). Another name that is used to identify such visits is the
agent or browser names used by respective search engines. Get a list of host
names and agent names from available resources (these names tend to
change often) and also develop your own intuitive list by searching your access logs for all occurrences of known engine, host or agent names.
Concentrate only on the top engines; though you may find several other
smaller and less known search engines visiting your site.
Pay attention to not only the total number of visits but to the activity pattern
for each of the recent visits to actually judge how many pages they covered.
This is a very good way of ensuring if submissions have worked or if other
inducements such as links from other sites have worked or not. This also
helps you to distinctly evaluate the effectiveness of submission, indexing and
page ranking characteristics of your site

Some examples of hostnames and agent names
AltaVista: hostname may have altavista.com
Lycos uses lycos.com within its host name
indexed. These search strings have been identified and compiled by some
useful resources on SEO.
For searching the pages from your URL in Google for example, insert the
following search string in Google search

allinurl:yourdomain.com/webmasters/meta.html (this depends on the
index pages of your site). In the Yahoo directory use the command
u:yourdomain.com to find the listings for this URL. There are similar,
but specific search strings applicable to each search engine