The Truth About Robots

Home > Promote > Search Engines > Articles

by David Notestine

Robot Travel

If there is one thing I have learned about robots, it is that there is absolutely no pattern to them. Most robots are stupid and wander randomly. For example, 50% of robot hits to my sites ask for the robots.txt page and then go away never asking for anything else. Then they come back a week later, ask for the same thing and then go away, again. This happens over and over again for months. I have never figured it out. What are they doing? If they wanted to see if the Web site was really a Web site, they could just Ping it. This would be much faster and much more efficient. They seldom visit another page and if they do, they ask for one other page every visit or so. Some come in and issue rapid-fire requests for every page in the Web site. How rude! You have to quit worrying so much about robots. It takes 6 months before they request enough pages to do you any good. I really quit thinking about them a long time ago. Build a lot of pages correctly, and, if you have reciprocal links to them, the robots will find them someday.

Try this: Go to AltaVista and type into the search box "link:YourSite.com" (Leave off the www). This will list the reciprocal links to your Web site. Try link:crownjewels.com and you get 136 links to it. Think about this now: The robots say to themselves, "Here is a site that must be popular or why would so many Web sites SIMILAR to it have it's link on their pages?" Remember that only SIMILAR sites with SIMILAR THEMES would probably have a link to your site. They give more importance to this than you submitting your link to them. Wouldn't you?

Go to heavily trafficked sites matching your Web site's Themes and use AltaVista to find out how many reciprocal links they have. This will prove to you I am right.