Annotations and User-Agent Detection for Mobile

The internet is no longer confined to the desktop or laptop web browser. The massively growing population of smartphone and tablet users has broadened the field of user engagement, mandating that websites be flexible in how they render and respond to a multitude of devices.

A recent study by Internet Analyst, Mary Meeker, just this month provided an Internet Trends report that by the end of Q2 2013, the global smartphone plus tablet install base will surpass the install base of the PC, and that 1 out of every 4 searches will be from a mobile device by 2013.

Due to the rapid increase in mobile internet users, and the fact that 96% of current mobile users have researched (searched) a product of service using their device, there is an increasing demand and opportunity for mobile search-friendly content that cannot be ignored.

On June 6, 2012 Google publicly provided recommendations for smartphone optimized websites.  The recommendation is to use “responsive web design” on the same URL as the best way to configure a mobile version of your website.

The reasons Google suggests the one URL responsive mobile design approach are as follows:

  1. Using a single URL for a piece of content makes it easier for your users to interact with, share, and link to your content, and a single URL for the content helps Google’s algorithms assign the indexing properties for the content.
  2. No redirection is needed for users to get to the device-optimized view, which reduces loading time, and increases crawling efficiencies. Also, user-agent-based redirection is prone to error and can degrade your site’s user experience, as well as your overall search engine visibility.
  3. Responsive web design saves resources for both your website and Google’s crawlers. When implemented, responsive web design under a one-URL construct means that Googlebot user-agents only need to crawl your content once, as opposed to crawling multiple times with different user-agents. This enhancement of crawling efficiencies can indirectly help Google index more web content, and help to keep the content fresh.

Google affirms that it will still support same URL with different HTML, which is actually best for mobile SEO efforts, as well as separate mobile sites under a subdomain or subdirectory construct within the same domain.

Responsive Web Design

(RWD) is an approach to web design in which a site is crafted to provide an optimal viewing experience no matter what type of device is being used. This means that content is made readily available for easy reading and navigation with a minimum of resizing, panning, and scrolling—across a wide range of devices, whether access the content from a desktop computer, tablet, smartphone or feature phone.

While RWD provides an excellent user experience, in and of itself, it is not considered to be mobile SEO optimal since you can’t optimize keywords based on device type search intent, which can be done with dynamic serving. Regardless of the mobile approach, user agent detection is required for any mobile redirection (if to another URL) or rending (on the same URL), not only to maximize your mobile SEO potential, but to increase conversions by providing the proper user experience on the customer’s device.

For many website owners responsive web design on the same URL is not feasible, and often companies rely on completely different websites and/or subdomains with similar content of the “desktop” version of a website, to serve mobile devices; therefore, it is imperative that alternate methods are in place to instruct search engines where to find the right version of the experience to serve searchers.

However; in addition to Google’s recommendation just discussed, Google does support smartphone-optimized sites other configurations:

  1. Sites that dynamically serve all devices on the same set of URLs, but each URL serves different HTML (and CSS) depending on whether the user agent is a desktop or a mobile device.
  2. Sites that have separate mobile and desktop URLs.

Sites that dynamically serve all devices on the same set of URLs

Dynamically serving different HTML on the same URL

Dynamic serving is a setup where the server responds with different HTML (and CSS) on the same URL depending on the user agent requesting the page. As it is not immediately apparent in this setup that the site alters the HTML for mobile user agents (the mobile content is “hidden”), we recommend that the server send a hint to request that Googlebot-Mobile should crawl the page, too, and thus discover the mobile content. This hint is implemented using the Vary HTTP header.

The Vary HTTP header

The Vary HTTP header has two important and useful implications:

  1. It signals to caching servers used in ISPs and elsewhere that they should consider the user agent when deciding whether to serve the page from cache or not.
  2. It helps Googlebot discover your mobile-optimized content faster, as a valid Vary HTTP header is one of the signals used to crawl URLs that serve mobile-optimized content.

The Vary HTTP header is part of the server’s response to a request and means the contents of the response will vary depending on the user agent that requests the page. If your server already uses the Vary HTTP header, you can add “User-Agent” to the list that’s already served.

Sites that have separate mobile and desktop URLs.

Often companies rely on completely different websites and/or subdomains and with similar content of the “desktop” version of a website, to serve mobile devices. In instances such as this alternate methods should be in place to instruct search engines where to find the right version of the experience to serve searchers.

Separate mobile URLs (Sites that have separate mobile and desktop URLs)

In this configuration, each desktop URL has an equivalent mobile URL serving mobile-optimized content.

  • Desktop Users experience page:  http://www.example.com/page-1
  • Mobile Users Corresponding page:  http://m.example.com/page-1

Annotations for Desktop and Mobile Smartphone URLs:

Desktop

On the Desktop page, add a special link rel=”alternate” tag pointing to the corresponding mobile URL.  This helps Googlebot discover the location of you site’s mobile pages.

For example, the desktop page is located at: http://www.example.com/page-1, and the equivalent mobile page is located at: http://m.example.com/page-1. The recommendation would be to include the link rel=”alternate” tag on the Desktop page pointing to the corresponding mobile page like this:

<link rel=”alternate” media=”only screen and (max-width: 640px)” href=”http://m.example.com/page-1″ >

Mobile

On the mobile page, add a link rel=”canonical” tag pointing to the corresponding desktop page.

For example, the mobile page is located at: http://m.example.com/page-1, and the equivalent desktop page is located at: http://www.example.com/page-1.  The recommendation would be to include the link rel=”canonical” tag on the Mobile page pointing to the corresponding desktop page like this:

<link rel=”canonical” href=”http://www.example.com/page-1″ >

Annotations for Separate Feature Phone URLs

Since feature phones do not have the capability to follow CSS media queries, webmasters wanting to serve feature phones would need to configure their sites to either use dynamic serving or separate URLs to serve the feature phone content.

If you’re planning to serve feature phone users with your mobile content specific for feature phone users; under separate URLs, it is recommended that you implement annotations to help Google’s algorithm understand the relationship between the equivalent pages targeting feature phones:

For example, the desktop page is located at: http://www.example.com/page-1, and the equivalent Feature phone page is located at: http://m.example.com/page-1

The recommendation would be to include the link rel=”alternate” tag on the desktop page pointing to the corresponding mobile page like this:

<link rel=”alternate” media=”handheld” href=”http://m.example.com/page-1″>

Annotations in detail

The above two-way (“bidirectional”) annotations helps Googlebot discover your content and helps its algorithms understand the relationship between your desktop and mobile pages and treat them accordingly. When you use different URLs to serve the same content in different formats, the annotation tells Google’s algorithms that those two URLs have equivalent content and should be treated as one entity instead of two entities.  If they are treated separately, both desktop and mobile URLs are shown in desktop search results, and their positions may be lower than they would otherwise be.

User Agent Detection

Once you have decided on an approach, you’ll need to set up your user agent detection. This is the process by which you recognize incoming user agents. When you see an incoming user agent that is mobile device specific, send them to the mobile version of your site. Make sure that your user agent detection includes Googlebot-Mobile.

User-Agent:

The user-agent is any software on any device which requests a resource (web page) from a server (website).  An example of a user-agent is a web browser (Google Chrome, Mozilla Firefox, Microsoft Internet Explorer, etc.). The website server can detect (or sniff) the type of user-agent accessing its content. Mobile phones and tablets each use unique versions of web browsers that indicate what device they’re running on.

User-Agent Detection

Server side user-agent detection takes place when the web server, having received the user-agent type with a request, either renders mobile ready content or redirects the user-agent (browser) to a different site or page (typically sub-domain) based on the kind of device browsing. For example, navigating to www.facebook.com on a mobile phone will redirect the user to m.facebook.com, the mobile variation. Logic can also be placed to render a given web page differently depending on the user-agent detected by the server.

Benefit of User Agent Detection

User agent detection allows for mobile search elements to be finely tuned for variances in mobile and desktop search results, as well as account for localized terms and such.  Without user agent detection in place, searchers get a one-query-fits-all search experience, which is suboptimal.  Websites need to be able to differentiate the search optimization elements by way of user agent detection in order to provide visitors a search optimal experience. Why is User Agent Detection so important?

One of the decisions you’ll want to make is what types of devices you’re going to be targeting. If you don’t have many feature phone users who are trying to access your website, you may not need to support them, and you can then focus your mobile site on smartphone devices.

Mobile device differences:

  1. Traditional mobile phones/feature phones: Phones with browsers that cannot render normal desktop webpages. This includes browsers for cHTML (iMode), WML, WAP, and the like.
  2. Smartphones: Phones with browsers that render normal desktop pages, at least to some extent. This category includes a diversity of devices, such Windows Phone 7, Blackberry devices, iPhones, and Android phones, and also tablets and eBook readers.
  3. Tablets: a kind of mobile computer, usually having a touch-screen or pen-enabled interface; which can render both desktop and mobile content.

 

Given the diversity of capabilities across mobile web browsers, the kind of content that you should serve Googlebot-Mobile lies in the User-agent that Googlebot-Mobile supplies when crawling. There are several User-agent strings in use by Googlebot-Mobile, all of which use this format:

[Phone name(s)](compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Here are the main user-agent strings that Googlebot-Mobile now uses:

  • Feature phones Googlebot-Mobile:
    • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
    • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • Smartphone Googlebot-Mobile:
    • Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

To decide which content to serve, assess which content your website has that best serves the phone(s) in the User-agent string. A full list of Googlebot-Mobile User-agents can be found here.

The key to any successful Mobile SEO strategy is to make sure, along with adjusting the design and content to focus on each device type’s search behavior and intent, that it is further amplified in your metadata. As always, when you think about metadata, keep in mind the following:

  • Mobile Web User Requirements are Different
  • Mobile Search Engine Behaviors are Different
  • Mobile Web User Activity is Different
  • Building Your Mobile Site for SEO is Different