Axandra news archive: 15 May 2007

• Facts of the week
• Search engine news of the week
• Articles of the week
• Recommended resources
• Previous articles

A recent thread in a webmaster forum indicated that some search engine spammers might exploit the new XML sitemaps files. Has your sitemaps file been abused by spammers? Can using a sitemaps file harm your search engine rankings?

What is a sitemaps XML file?

The big search engines (Google, Yahoo, MSN and Ask) introduced the Sitemaps protocol earlier this year.

In its simplest form, a sitemap is an XML file that lists URLs for a site along with additional metadata about each URL: when it was last updated, how often it usually changes, how important it is, relative to other URLs in the site, etc.

That information helps search engines to more intelligently crawl your site. The Sitemaps protocol is a standard that makes it easier to create a sitemap that can be parsed by all search engines.

How can such a file harm your rankings?

Some webmasters reported problems with duplicate content after adding a sitemaps XML file to their web sites.

The content of their websites appeared on dubious websites that had nothing to do with the original sites. The content of the original websites had been duplicated on many other sites. The result was that the original sites might have received ranking penalties due to duplicate content.

What happened?

Some search engine spammers used the sitemaps XML files to easily find contents for their scraper sites.

A scraper site is a website that pulls all of its information from other websites using automated tools. The scraper software pulls different contents from other websites to create new web pages that are designed around special keywords. The scraped pages usually show AdSense ads with which the spammers hopes to make money.

The new sitemaps XML files make it very easy for scraper tools to find content rich pages. Although the original intention of the sitemaps files was to inform search engines about every single page of your web site, they can also be used to inform spam bots about your pages.

What can you do to avoid problems with your sitemaps file?

One possible solution is not to use any sitemaps file at all. In that case, scraper bots can still parse your web pages through the normal links on your web pages but that would be more difficult for them than using your sitemaps file.

Another solution is to set up a sitemaps file and delete as soon as search engines have indexed that file.

Do not use free sitemap generator tools. You don't know what they will do with your data and they might even use it to create scraper sites with your content.

Unfortunately, there's not much that you can do to stop spammers from abusing your content. Use a tool such as CopyScape to find sites that have duplicated your content.

"These spam emails have created some confusion about their authenticity, since we send very similar email notifications [...]

Because these emails are easy to mistake for authentic ones from Google, we've temporarily discontinued sending them as we work on ways to provide more secure communication mechanisms. We hope this will reduce confusion."

"Google Inc.'s corporate tagline has become 'Search, Ads and Apps,' reflecting a shift beyond search and advertising into online software applications."

"Algorithms and algorithmic spamfighting are an essential way to improve Google’s quality, but Google does reserve the right to take manual action on spam. [...]

Our current algorithm detected the paid links [...] just fine, but these outside reports are a great way to measure (and then improve) the precision and recall of our existing algorithms on independent data."

• Who's afraid of Google?
• Google shareholders vote against anti-censorship proposal.
• Google layouts timeline.
• IAC to launch Ask Mobile location-based search service.
• Yahoo starts "Greenest City in America" contest.
• Chinese search engine Baidu sued for trade name violation.
• Trial scheduled in Google "Adwords" case.
• A Google Search URL that removes AdSense ads.
• Google Categories.
• Google Gadgets in your search results.
• A clarification on accidental AdSense clicks.
• Google Earth in Portuguese, Dutch, Russian, Polish, Korean, Arabic, and Czech.
• Dutch domain owner faces legal action from Google.

"Google has become more comfortable doing big acquisitions but still sees small technology deals as its primary thrust for buying businesses".

"Internet giant Google has drawn up plans to compile psychological profiles of millions of web users by covertly monitoring the way they play online games.

The company thinks it can glean information about an individual's preferences and personality type by tracking their online behaviour, which could then be sold to advertisers."

MSN, Yahoo! partnership envisioned

"The marriage of Microsoft and Yahoo! didn't happen as the tabloids predicted, but I wonder if the aging Internet giants will try a domestic partnership instead. [...]

Search is a great business, but the fascination with it is starting to fade. Search is becoming a utility we take for granted; content is where the Web's sizzling now."

"We would highly recommend IBP as an easy to use, comprehensive, high quality SEO software product."

"As a highly professional UK search marketing organisation, we have used IBP9 and ARELIS for some years now, and would highly recommend it as an easy to use, comprehensive, high quality SEO software product. We use it every day and wouldn't be without it."
Paul McIntyre, www.searchhigh.co.uk

Do you want to be mentioned in this newsletter?

• Two new Google ranking patents
• The influence of semantic indexing on your search engine rankings
• What Google thinks about hidden and paid links
• Search engine submission with the new Sitemap protocol
• How to deal with Google's latest ranking algorithm changes
• What Google could do if they were evil
• All about Google's new PPA advertising product
• How to work around Google' spam filters - Part V
• View all past issues...