This week, we're taking a look at the new X-Robots-Tag that is now supported by Google and Yahoo. How can this tag help you to increase your search engine rankings?

In the news: Google changed the ranking algorithm so that less pages from the same domain are returned in the search results, Ask.com releases a new privacy feature and more.

Table of contents:

• Facts of the week
• Search engine news of the week
• Articles of the week
• Recommended resources
• Previous articles

Last week, Yahoo announced that they now support the X-Robots-Tag in the HTTP header. This new tag allows you to influence how Google and Yahoo index your website pages.

What is the X-Robots-Tag?

Google introduced the new X-Robots-Tag directive in 2007 to allow webmasters to control access to non-web page documents, such as Adobe PDF files, video and audio files.

The X-Robots-Tag is included in the HTTP header of a document. The HTTP header is the initial reply of a server to a query. It contains information about the document that follows, including content type, creation date, character set, encodings, etc.

The new X-Robots-Tag allows webmasters to add information about search engine indexing to the HTTP header.

Which commands are supported by the X-Robots-Tag?

At this time, the X-Robots-Tag supports the following commands:

• X-Robots-Tag: NOINDEX (Use this tag if you don't want to show the URL in Google's search results)
  
  
• X-Robots-Tag: NOARCHIVE (Use this tag if you don't want to see a Cache link in the search result pages for the document)
  
  
• X-Robots-Tag: NOSNIPPET (Use this tag if you don't want to display a summary in the search result pages.)
  
  
• X-Robots-Tag: NOFOLLOW (Use this tag if you don't want Googe and Yahoo to index the links in the page.)

The X-Robots-Tag is currently supported by Google and Yahoo. Other search engines don't support the tag yet.

How to add the X-Robots-Tag to your documents

It depends on your server and on the method you use to create your web pages how you can add additional tags to the HTTP header. If your web server uses Apache, you can use the htaccess file to modify your HTTP headers.

In general, you shouldn't mess with the HTTP header if you're not 100% sure what you're doing. A broken HTTP header can keep search engines away so that no search engine will index your website.

Instead of using the X-Robots-Tag, you can also add meta tags with the same functionality to your web pages:

<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex">

You can edit these tags directly in the head part of your web pages. They have the same effect as the corresponding X-Robots-Tags explained above.

Do you need the X-Robots-Tag or the corresponding meta tags?

If you don't want to limit access to your documents of if you only have normal web pages on your website then you don't need the X-Robots-Tag or the meta tags.

The X-Robots-Tag has been designed to restrict access to your documents. That means that it does not help to increase search engine rankings.

The new X-Robots-Tag and the meta robots tag allow you to restrict access to your documents. If you want to make sure that search engines index all of your web pages then you have to optimize your web pages.

Search engine robots must be able to easily parse your web pages and they must be able to find the right elements on your web pages. If you optimize the right web page elements with the right keywords, your web page will get high search engine rankings for these keywords.

"Historically, it’s been kind of a wash about when to use subdomains vs. subdirectories. [...]

For several years Google has used something called “host crowding,” which means that Google will show up to two results from each hostname/subdomain of a domain name.

That approach works very well to show 1-2 results from a subdomain, but we did hear complaints that for some types of searches (e.g. esoteric or long-tail searches), Google could return a search page with lots of results all from one domain. In the last few weeks we changed our algorithms to make that less likely to happen."

"Matt Cutts, the head of Google's webspam team, provides some useful tips on how to optimize the images you include on your site, and how simply providing useful, accurate information in your ALT attributes can make your photos and pictures more discoverable on the web."

"AskEraser is a new privacy feature from Ask.com. When AskEraser is enabled your search activity will be deleted from Ask.com servers."

• Google enters UK schools market.
• More server problems with Overture's keyword tool.
• Google privacy: emails, off-the-record chats.
• Wikia looking to topple Google, but will fail miserably.
• You sleigh me! (track Santa)
• Success without ads.
• MSN launches a new mobile portal.
• Google disables Gmail accounts by mistake.

"We have achieved TOP 10 rankings on Google."

"IBP was recommended to me by a colleague, and I have to say that I was somewhat sceptical about its claims.

However, after having used IBP's for just three months we have achieved TOP 10 rankings on Google for the majority of related searches. IBP is simply unbeatable, in both performance and in value. Great work Axandra!"
Jan Franklin, Inspiration Sounds

The Google ranking mystery is solved

No need to speculate anymore. There is a proven solution.

IBP's Top 10 Optimizer decrypts Google's ranking algorithm and it tells you what exactly you have to do to get top 10 rankings on Google.

Click here for further information.

Get your site mentioned in front of 140,000+ subscribers

We want to hear from you about your successes with IBP or ARELIS. Just write us 2-3 sentences and you might get featured in this newsletter along with your web site address.

• Google warns against cloaking: we can detect it
• New robots.txt commands: make sure that Google can index your site
• Are you paying too much for your AdWords ads? Change it now!
• Troubleshooting: how search engines can misunderstand your web pages
• Top 6 reasons why your search engine rankings have dropped
• Reciprocal links are alive and kicking
• How long does it take until a new site ranks in Google, Yahoo and MSN?
• View all past issues...