Google News now crawling with Googlebot

Thursday, August 25, 2011 at 2:00 PM

Webmaster Level: intermediate

(Cross-posted on the Google News Blog)

Google News recently updated our infrastructure to crawl with Google’s primary user-agent, Googlebot. What does this mean? Very little to most publishers. Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot.

Our Help Center provides detailed guidance on using the robots exclusion protocol for Google News, and publishers can contact the Google News Support Team if they have any questions, but we wanted to first clarify the following:

• Although you’ll now only see the Googlebot user-agent in your site’s logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies. (You can always check whether your site is included in Google News by searching with the “site:” operator. For instance, enter “site:yournewssite.com” in the search field for Google News, and if you see results then we are currently indexing your news site.)
• Your analytics tool will still be able to differentiate user traffic coming to your website from Google Search and traffic coming from Google News, so you should see no changes there. The main difference is that you will no longer see occasional automated visits to your site from the Googlebot-news crawler.
• If you’re currently respecting our guidelines for Googlebot, you will not need to make any code changes to your site. Sites that have implemented subscriptions using a metered model or who have implemented First Click Free will not experience any changes. For sites which require registration, payment or login prior to reading any full article, Google News will only be able to crawl and index the title and snippet that you show all users who visit your page. Our Webmaster Guidelines provide additional information about “cloaking” (i.e., showing a bot a different version than what users experience). Learn more about Google News and subscription publishers in this Help Center article.
• Rest assured, your Sitemap will still be crawled. This change does not affect how we crawl News Sitemaps. If you are a News publisher who hasn’t yet set up a News Sitemap and are interested in getting started, please follow this link.
• For any publishers that wish to opt out of Google News and stay in Google Search, you can simply disallow Googlebot-news and allow Googlebot. For more information on how to do this, consult our Help Center.

As with any website, from time to time we need to make updates to our infrastructure. At the same time, we want to continue to provide as much control as possible to news web sites. We hope we have answered any questions you might have about this update. If you have additional questions, please check out our Help Center.

Posted by David Smydra, Google News Product Specialist

Making the most of improvements to the +1 button

Wednesday, August 24, 2011 at 10:22 AM

Webmaster level: All

For the past few months, you might have used +1 buttons to help visitors recommend your content on Google Search and on their Google Profiles. We’ve just announced a few changes that make +1 even more useful.

First, the +1 button now lets visitors share links to your pages on Google+. If someone wants to start a conversation about your content, it’s easy for them to do so. Second, you can use +Snippets to customize the name, image and description that appear when your content is shared. Finally, new inline annotations help increase engagement after users see a friend’s recommendation right on your page.

Here are a couple of tips to help you take full advantage of these improvements:

+Snippets
The +1 button opens up your site to a valuable new source of traffic with sharing on Google+. +Snippets let you put your best face forward by customizing exactly what appears when your content is shared.

For example, if you’re running a movie review site, you might want visitors to share posts containing the title, movie poster, and a brief synopsis:

You may already be using this markup to build rich annotations for your pages on Google Search. If not, marking up your pages is simple. Just add the correct schema.org attributes to the data already present on your pages. You’ll set a name, image, and description in your code:

<body itemscope itemtype="http://schema.org/Article">
<h1 itemprop="name">This is the article name</h1>
<img itemprop="image" src="thumbnail.jpg" />
<p itemprop="description">This is the description of the article.</p>
</body>

Example code containing each of the +Snippet attributes

For more details on alternate markup types, please see our technical documentation.

Inline annotations
Now, when a person visits a page that someone they know has +1’d, they can see a name and face reminding them to pay special attention to your content. Here’s how it looks:

Inline annotations let people see which of their friends +1’d your content

To add inline annotations, you need to update your +1 button code. Visit the configuration tool, select ‘inline’ from the ‘Annotation’ menu, and grab a new snippet of code.

Both sharing from +1 and inline annotations are rolling out fully over the next few days. To test these improvements right now, join our Platform Preview group.

Update later the same day, August 24, 2011: If you have any thoughts or feedback you'd like to share, continue the conversation on Google+.

Posted by Daniel Dulitz, Group Product Manager

Help us improve Google Search

Tuesday, August 23, 2011 at 9:45 AM

Webmaster level: Advanced

Yes, we're looking for help improving Google search—but this time we're not asking you to submit more spam reports. Although we still appreciate receiving quality spam reports, today we've got a different opportunity for you to improve Google Search: how about YOU join our team and do the webspam fighting yourself?

Interested? Here's what we're looking for: open-minded academic graduates willing to work in a multinational environment in our Dublin office. Looking at a site's source code should not scare you. You should be excited about search engines and the Internet. It’s also essential that you share our aversion to webspam and the drive to make high-quality content accessible. PlayStation or foosball skills are a plus.

This is an actual work environment photo taken at the Dublin Google office.

If you'd like to know more about the positions available, here's the full list of requirements and responsibilities. Great candidates should be able to email the recruiter directly.

Posted by Kaspar Szymanski, Search Quality

A new rich snippets format for music

Thursday, August 18, 2011 at 10:40 AM

Webmaster level: All

Since we introduced Rich Snippets back in 2009, we’ve created rich snippet formats for a variety of different content types, such as Events, People and Reviews, to show users relevant information about the content they can find on a site. Today, we announced the launch of rich snippets for music. With this new feature, site owners can mark up their pages using the newly created music markup spec on schema.org, and search results for that site may start displaying song information in the snippet so that users know that there are songs or samples there for them to listen to:



Several initial partners have implemented the music markup on their sites, including MySpace, Rhapsody and ReverbNation. As with other rich snippet formats, implementing the markup does not guarantee that your site will be displayed with the UI shown above on a given search; a variety of factors affect whether a particular rich snippet type will appear in our search results. However, having correct markup is a prerequisite for music rich snippets to ever be displayed. You can use the rich snippets testing tool to test the markup of your page and see an illustration of what the search result would look like with music rich snippets.

For now, music rich snippets will display song information when users search for artists, album names or song names. We’ll continue working both on expanding our existing rich snippets formats and on creating new ones, so keep watching for updates about new types of content that you can surface for users right in your site’s snippets.

If you have any questions about this new rich snippets format, you can head over to our Webmaster Forum and ask. You can also check out our page on rich snippets in our Webmaster Tools Help Center, which includes an article specifically about music rich snippets.

Posted by Gideon Wald, Associate Product Manager

Introducing new and improved sitelinks

Tuesday, August 16, 2011 at 12:15 PM

Webmaster level: All

This week we launched an update to sitelinks to improve the organization and quality of our search results. Sitelinks are the two columns of links that appear under some search results and ads that help users easily navigate deeper into the site. Sitelinks haven’t changed fundamentally: they’re still generated and ranked algorithmically based on the link structure of your site, and they’ll only appear if useful for a particular query.

Sitelinks before today’s changes

Here’s how we’ve improved sitelinks with today’s launch:

• Visibility. The links have been boosted to full-sized text, and augmented with a green URL and one line of text snippet, much like regular search results. This increases the prominence of both the individual sitelinks and the top site overall, making them easier to find.


• Flexibility. Until now, each site had a fixed list of sitelinks that would either all appear or not appear; there was no query-specific ranking of the links. With today’s launch, sitelink selection and ranking can change from query to query, allowing more optimized results. In addition, the maximum number of sitelinks that can appear for a site has been raised from eight to 12, and the number shown also varies by query.


• Clarity. Previously, pages from your site could either appear in the sitelinks, in the regular results, or both. Now we’re making the separation between the top domain and other domains a bit clearer. If sitelinks appear for the top result, then the rest of the results below them will be from other domains. One exception to this is if the top result for a query is a subpart of a domain. For instance, the query [the met exhibitions] has www.metmuseum.org/special/ as the top result, and its sitelinks are all from within the www.metmuseum.org/special section of the site. However, the rest of the results may be from other parts of the metmuseum.org domain, like store.metmuseum.org or blog.metmuseum.org/alexandermcqueen/about.


• Quality. These user-visible changes are accompanied by quality improvements behind the scenes. The core improvement is that we’ve combined the signals we use for sitelinks generation and ranking -- like the link structure of your site -- with our more traditional ranking system, creating a better, unified algorithm. From a ranking perspective, there’s really no separation between “regular” results and sitelinks anymore.

Sitelinks after today’s changes

These changes are also reflected in Webmaster Tools, where you can manage the sitelinks that appear for your site. You can now suggest a demotion to a sitelink if it’s inappropriate or incorrect, and the algorithms will take these demotions into account when showing and ranking the links (although removal is not guaranteed). Since sitelinks can vary over time and by query, it no longer makes sense to select from a set list of links -- now, you can suggest a demotion of any URL for any parent page. Up to 100 demotions will be allowed per site. Finally, all current sitelink blocks in Webmaster Tools will automatically be converted to the demotions system. More information can be found in our Webmaster Tools Help Center.

It’s also worth mentioning a few things that haven’t changed. One-line sitelinks, where sitelinks can appear as a row of links on multiple results, and sitelinks on ads aren’t affected. Existing best practices for the link structure of your site are still relevant today, both for generating good quality sitelinks and to make it easier for your visitors. And, as always, you can raise any questions or comments in our Webmaster Help Forum.

Written by Harvey Jones, Software Engineer, & Raj Krishnan, Product Manager, Sitelinks team

High-quality sites algorithm launched in additional languages

Friday, August 12, 2011 at 4:00 AM

(Cross-posted on the Inside Search blog)

Webmaster level: All

For many months, we’ve been focused on trying to return high-quality sites to users. Earlier this year, we rolled out our “Panda” change for searches in English around the world. Today we’re continuing that effort by rolling out our algorithmic search improvements in different languages. Our scientific evaluation data show that this change improves our search quality across the board and the response to Panda from users has been very positive.

For most languages, this change impacts typically 6-9% of queries to a degree that a user might notice. This is distinctly lower than the initial launch of Panda, which affected almost 12% of English queries to a noticeable amount. We are launching this change for all languages except Chinese, Japanese, and Korean, where we continue to test improvements.

For sites that are affected by this algorithmic change, we have a post providing guidance on how Google searches for high-quality sites. We also have webmaster forums in many languages for publishers who wish to give additional feedback and get advice. We’ll continue working to do the right thing for our users and serve them the best results we can.

Posted by Amit Singhal, Google Fellow

New webmaster tutorial videos

Tuesday, August 09, 2011 at 10:35 AM

Webmaster level: All

Over the past couple of years, we’ve released over 375 videos on our YouTube channel, with the majority of them answering direct questions from webmasters. Today, we’re starting to release a freshly baked batch of videos, and you might notice that some of these are a little different. Don’t worry, they still have Matt Cutts in a variety of colored shirts. Instead of only focusing on quick answers to specific questions, we’ve created some longer videos which cover important webmaster-related topics. For example, if you were wondering what the limits are for 301 redirects at Google, we now have a single video for that:



Thanks to everyone who submitted questions for this round. You can be the first to hear about the new videos as they’re released by subscribing to our channel or following us on Twitter.

Posted by Michael Wyszomierski, Search Quality Team