Home Page Image
Next Topic >
Internet Vocabulary
 


Search Engines, Meta Search, & Web Directories

There are three basic types of search services available to Internet users: search engines, directories, and so-called meta search services. Search engines offer users the most comprehensive indexing of Web sites and have user interfaces that allow searching their indexes in a variety of ways. Directories are more akin to traditional online databases common to most libraries in that their content is selective and is organized into subject categories. Meta search services, such as Dogpile, piggyback on search engines and directories and provide the means for sending a search to numerous Internet services at one time.

Google reputedly has the largest index of the Web, although a rival service known as Cuil claims to outnumber Google's searchable index by three times. Yahoo is probably the best known directory of the Web and offers users a hierarchically organized subject listing of millions of Web sites. The primary difference between search engines and directories is that search engines find Internet resources automatically using sophisticated software with little or no human intervention, while directories are created by people who have actually visited and verified the content of websites and categorized the sites selected according to predetermined subject categories.

If you are looking for a more comprehensive view of materials available on the Internet, use a search engine. If you want an organized, mediated approach to Web content, use a directory. If you want a quick overview with links of what numerous search services have to offer in their indexes, try a meta search service. Most of the major search engines have developed into hybrids, still providing machine-generated, searchable indexing, but also providing classified listings of available materials. In essence, a searcher can get the best of both worlds from one source. Another trend is toward partnering with other Internet services. AltaVista, for example, provides multiple language translation of Web pages through its alignment with Yahoo's Babel Fish and Netscape, through its NetCenter, offers Internet search services and directory listings based on its partnerships with a number of search services.

Piggy-back riders of the major search engines, meta search services like MetaCrawler and WebCrawler provide users with one-stop shopping in numerous search engine databases. The idea behind a meta index is to accept a user's search words, formulate searches that comply with criteria used in the search engines it covers, and to deliver the top listings from each of the search engines in one easily clickable index. Users can customize their meta searches, selecting the services to be searched as well as the number of entries to be returned for each service.

For the most complete view of what is currently indexed on the Internet, a searcher should learn the techniques that make for successful searching of one or more of the major search engines. Keep in mind that no two search engines are going to give you the same results and that the results you get from one search engine one day may be different than what you get from another. The Web is constantly changing and the search engines are constantly finding and indexing the changes. If you don't find what you need in one search engine, try another. If you don't find it today, try again tomorrow. The Internet is constantly changing as people add sites, remove sites, and continuously revise information.

When you do use the search engines, keep in mind that no two are completely alike and no two will give you the exact same resources. Each has its own criteria for selecting and indexing and no two do it exactly the same. When you review your lists of results for your keyword searches and wonder why Google gave you 5,000 documents and Yahoo gave you 3,000, and the duplicates between the two services amount to only 1,000, keep in mind that the search engines differ in a number of ways: Some index the entire content of Internet; others index only certain parts, such as Web pages only or newsgroups only. If you are wanting to scan newsgroups only, for example, use a service like Google Groups. If you want a comprehensive look at anything available on the Internet, use Cuil, or Google, or Live Search.

The frequency with which a search engine updates its index varies from engine to engine. Some update constantly, revisiting sites on at least a weekly basis; others may update particular sites infrequently and others more frequently. If you really want to learn more about how the search engines compare at what they do, visit Search Engine Watch. Some search engines retrieve the full text of everything in a Web site, while others may only retrieve the title and the first paragraph of each individual document. Others may follow a Web site's structure only so far, retrieving only a portion of the full content of the site and ignoring documents that reside at deeper levels. Of course, once you find a relevant document, you can always follow its links to related information either in the same Web site or in other sites. Ahhhh...the wonders of hypertext.

One of the primary differences in the search engines is their user interfaces. Most have a simple user interface even if they also provide more advanced means for searching. Some offer multiple fill-in-the-blank boxes to further facilitate searching for more specific information; others offer command line interfaces where an experienced searcher looking for very specific information can use Boolean logic and proximity and field searching to further narrow a search. Keep in mind that when you see an option labeled "Power Search" or "Advanced Search" that you should not shy away. In most cases, these options provide you with additional and more efficient means for searching for content. How you enter a search makes a difference in how successful your search will be. Whether you should use a search engine, a directory of the Web, or a meta index depends on how specific your topic is and how fast you want results.

Following are some general suggestions about what type of service to use to find information.

USE
WHEN
A directory like Yahoo You have a general subject in mind but want to browse for more specific topics. (This is similar to going to a subject area in a library's collection and flipping through a number of books for things of interest.)
A directory like Yahoo You are looking for organizations that specialize in a certain field. (Think of this as a yellow pages browse where you begin by locating a product or service and then browse through listings of companies or organizations that provide them.)
A directory like Yahoo You need to quickly locate both domestic and foreign government agencies.
A directory like Yahoo

You don't have time to search through hundreds or thousands of hits for general information on a topic that you know will be covered in somebody's Web pages.

A directory like Yahoo You need to avoid irrelevant hits.
A directory like Yahoo You have hours and hours to burn and want to browse everything they have identified on Entertainment.
A search engine like Google You have a very specific topic that you want to pin down in anything available on the Internet.
A search engine like Google You want to know just how much information is available on your topic, regardless of its quality.
A search engine like Google You need to search using phrases instead of single words.
A search engine like Google You want to further limit your search by adding additional terms to your original topic.
A search engine like Google You want to search for specific kinds of files on the Internet (audio, graphics, video, etc.).
A search engine like Google You have nothing to do for hours and hours and don't mind occasionally meandering aimlessly through the Wild Wild Web. If you enjoy hiking, backpacking, mountain climbing, or joy riding, you'll enjoy using a search engine as a starting point for a long, suprise-filled journey.
A search engine like Google You need a quick answer to a tough question and don't shy away from "advanced" or "power" searches.
A Reference Librarian None of the above suggestions helps you decide where to begin.

 

 

RESOURCES

Cuil Help
Google Help
Live Search Help
MetaCrawler Help
Yahoo Help

   
  Guide by Jim Alderman. September 2008.   Contact Jim.