Today, it is impossible to imagine the Internet without search engines, and without them it would be virtually impossible to make meaningful use of the World Wide Web with its billions of pages. However, in order to constantly improve the findability of information on the Internet, search engines must collect and statistically evaluate targeted information about search behavior and the quality of search hits. The ever-growing number of Internet pages alone makes it necessary for search engine operators to process an ever-increasing amount of information.
Source: FDPIC – Information and tips on dealing with search engines