Search engines help people find relevant information on the Internet. Major search engines such as Google, Yahoo, MSN and many others maintain huge databases of web sites that users can search by typing in some text.
Search Engine Spiders
To compile their databases, search engines rely on computer programs called "robots" or "spiders." These programs "crawl" across the web visiting websites and cataloguing them. Each search engine uses its own specific set of criteria to decide which parts of each page to include with its database. Some search engines catalogue every page within a site, others only index certain pages. Most search engines record various bits of information about your site such as the page titles, keywords and main textual content. Once the robot or spider has catalogued your page and added it to its database it then follows any links on your page to the next page and catalogues that.
Be kind to the spiders
Making sure that your pages are easy for search engine robots to read is an essential part of Search Engine Optimisation.
Once your site is catalogued within a search engines database, the search engine will rank it for its relevancy for particualr keywords.
When someone uses a search engine to search for words contained within pages on your site, your site will be displayed in the search engines results. The more relevant your site to the search criteria, the higher up the list of results your page will appear.
Each Search Engine uses its own special and highly secretive algorithms for determining a sites relevancy. These algorithms are kept a secret because it is a search engines competitive advantage to provide more accurate and relevant results than the other engines. It is however known that some list the results of a user's search according to which sites have the most links from other sites (a system known as link popularity). Other search engines prioritize results according to the summary information contained in sites' meta tags, and still others look for common themes used throughout a site. There are many other ways to organize results, and most search engines use a combination of several of them.