Caffeine gives Google search a jolt

Google's new search engine architecture, which it calls "Caffeine," will make recently updated Web pages load faster when you search. Caffeine also makes it easier for Google to scale as the Web grows, and add new features like annotations, the company said.

"It's like swapping out the engine of a car -- you can do more with it," said Matt Cutts, Google software engineer and head of the company's Webspam team, in a phone interview. "This is probably the biggest update or change to our indexing system since 2006."

The most visible, immediate change to users: Search results will be more up-to-date. "Documents will be a bit fresher," he said. Previously, Google crawled the Web in sections, then made batch decisions on indexing each section.

In 2000, it took Google 30 days to crawl the Web, and 7-10 days to index it -- meaning search results were likely to be 7-10 days old, Cutts said.

By 2003, Google had improved that turnaround to less than a day, he said.

Now, with Caffeine, indexing happens as soon as a Web page is discovered, meaning search results are added in seconds.

Try it yourself. Search for something that's likely to be updated frequently with new results -- like the word "Obama," Cutts said. Click "Latest" in the tools on the left side. You'll see search results updated in a minute or less.

 

Caffeine also makes it easier for Google's search index to scale. Google's index is now 10 petabytes. Each petabyte is 1,024 terabytes, or a million gigabytes.

Caffeine's index will scale an order of magnitude larger than the pre-caffeinated capacity, Cutts said.

The final big change is the ability to incorporate annotations into search results. Annotations are additional information that Google displays when you run a search; they don't change search ranking, but they do make search more useful, Cutts said. Google discovers potential annotations by reading the contents of a Web site and identifying the kinds of information available there.

For example, if you search on businesses, Google will often include Yelp results in the search. Here is a search for the restaurant Serious Pie in Seattle, where Cutts said he had dinner Tuesday:

Previously, Google had to rewrite its index code to add annotations, but Caffeine will make it easier to add annotations on the fly. For example, in the future, Google might add business hours to searches for a business, or whether a shop accepts MasterCard or Visa. "Anytime we want to add new information, Caffeine makes it easier for us," he said.

Google has been spending a lot of time recently on diversifying its business, with Android mobile phones, the Google Chrome browser and operating system, and upcoming Google TV. Caffeine is an investment in Google's traditional, core business: Search.