Skip to main content

Google slows web crawlers to help blackout websites during protest effort

Google and many other websites went black today to oppose the Stop Online Piracy Act (SOPA) in the House and the Protect IP Act (PIPA) in the Senate, but now the globally popular search engine has taken the protest one step further.

Pierre Far, a Webmaster trends analyst at Google, announced on Google+ today that the Mountain View, Calif.-based company slowed its web crawlers to continue support against U.S. anti-piracy bills.

More information is available below.

“We realize many webmasters are concerned about the medium-term effects of today’s blackout,” said Far. “As a precaution, the crawl team at Google has configured Googlebot to crawl at a much lower rate for today only so that the Google results of websites participating in the blackout are less likely to be affected.”

Far previously shared blackout tips in a post called “Website outages and blackouts the right way” on Jan. 16 where he advised webmaster’s to use a 503 HTTP status code to effectively tell spiders that a website is temporary unavailable.

Google will not index the page’s content when the status code is implemented; thus, the code physically blacks —or blocks— the website from the Internet. This would normally be a fatal decision, but the code is also a temporary solution for actively protesting SOPA and PIPA.

With that said, the code may only work with Google, so remember that Bing or Yahoo may blackout websites in a different manner.

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Google — experts who break news about Google and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Google on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel