Just how does a search engine find exactly what you are looking for when it has to search through millions of web pages? Software robots called Spiders, through a process called Web Crawling help you find your website. Web Crawling is when the spider makes lists of words that are found on the Internet. The spider begins its search at more popular web pages or servers. It follows the links to different web pages and makes lists of the words that were searched for and where they were found. It builds its index based on this.

            Different search engines program their spiders differently. Google uses three spiders at a time. Google spiders are programmed to index all the important words on a page while leaving out words like “a”, “an”, and “the”. However, other search engines prefer their spiders to index every word on a page. While others index the 100 most frequently occurring words in the first 20 lines.

To see how a search engine ranks its results go to PageRank.

How Internet Search Engines Work