Last
week, Yahoo announced that they
now support the X-Robots-Tag in
the HTTP header. This new tag
allows you to influence how Google
and Yahoo index your website pages.
What is the X-Robots-Tag?
Google introduced the new
X-Robots-Tag directive in
2007 to allow webmasters to
control access to non-web
page documents, such as Adobe
PDF files, video and audio
files.
 The
X-Robots-Tag is included in
the HTTP
header of a document.
The HTTP header is the initial
reply of a server to a query.
It contains information about
the document that follows,
including content type, creation
date, character set, encodings,
etc.
The new X-Robots-Tag allows
webmasters to add information
about search engine indexing
to the HTTP header.
Which commands are
supported by the X-Robots-Tag?
At this time, the X-Robots-Tag
supports the following commands:
- X-Robots-Tag: NOINDEX (Use
this tag if you don't want
to show the URL in Google's
search results)
- X-Robots-Tag: NOARCHIVE (Use
this tag if you don't want
to see a Cache link in the
search result pages for
the document)
- X-Robots-Tag: NOSNIPPET (Use
this tag if you don't want
to display a summary in
the search result pages.)
- X-Robots-Tag: NOFOLLOW (Use
this tag if you don't want
Googe and Yahoo to index
the links in the page.)
The X-Robots-Tag is currently
supported by Google and Yahoo.
Other search engines don't
support the tag yet.
How to add the X-Robots-Tag
to your documents
It depends on your server
and on the method you use
to create your web pages how
you can add additional tags
to the HTTP header. If your
web server uses Apache, you
can use the htaccess file
to modify your HTTP headers.
In general, you shouldn't
mess with the HTTP header
if you're not 100% sure what
you're doing. A broken HTTP
header can keep search engines
away so that no search engine
will index your website.
Instead of using the X-Robots-Tag,
you can also add meta tags
with the same functionality
to your web pages:
<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex">
You can edit these tags directly
in the head part of your web
pages. They have the same
effect as the corresponding
X-Robots-Tags explained above.
Do you need the X-Robots-Tag
or the corresponding meta
tags?
If you don't want to limit
access to your documents of
if you only have normal web
pages on your website then
you don't need the X-Robots-Tag
or the meta tags.
The X-Robots-Tag has been
designed to restrict access
to your documents. That means
that it does not help to increase
search engine rankings.
The new X-Robots-Tag and the
meta robots tag allow you to
restrict access to your documents.
If you want to make sure that
search engines index all of
your web pages then you have
to optimize your
web pages.
Search engine robots must be
able to easily parse your web
pages and they must be able
to find the right elements on
your web pages. If you optimize
the right web page elements
with the right keywords, your
web page will get
high search engine rankings for
these keywords.
|
Customer Support • Code of ethics
No
need to speculate anymore.
There is a proven solution. 