Google pushes for an official web crawler standard

One of the cornerstones of Google's business (and really, the web at large) is the robots.txt file that sites use to exclude some of their content from the search engine's web crawler, Googlebot. It minimizes pointless indexing and sometimes keeps s...

Google Webmaster Tools is Now Google Search Console


Google decided to rename Google Webmaster Tools to Google Search Console. The essential tools for webmasters have been around for nearly ten years. Google has found out that the traditional idea of...

Google bots learning to read webpages like humans, one step closer to knowing everything

Google bots now read webpages more like humans, one step closer to knowing everything

Google just launched its Knowledge Graph, a tool intended to deliver more accurate information by analyzing the way users search. Of course, with a desire to provide better search results comes a need for improved site-reading capabilities. JavaScript and AJAX have traditionally put a wrench in Google bots' journey through a webpage, but it looks like the search engine has developed some smarter specimens. While digging through Apache logs, a developer spotted evidence that bots now execute the JavaScript they encounter -- and rather than just mining for URLS, the crawlers seem to be mimicking how users click on objects to activate them. That means bots can dig deeper into the web, accessing databases and other content that wasn't previously indexable. Looks like Google is one step closer to success on its quest to know everything.

Google bots learning to read webpages like humans, one step closer to knowing everything originally appeared on Engadget on Thu, 17 May 2012 00:36:00 EDT. Please see our terms for use of feeds.

Permalink Ars Technica  |  sourceswapped.cc/blog  | Email this | Comments