Search Engine War Blog : « BBC.co.uk - another miserable failure | Search marketing ad spend to increase »

The power of links – non-indexed pages out-ranking optimised ones + robots.txt flaw.

Tuesday, 10 April 2007

My friend Mike Grehan often talks in his linking presentations about how a page not even in Google can out rank one that has actually been indexed and optimised, well here's a good example of exactly that.  The links in position 2 and 3 are on Googles own website and explicitly blocked by their robots.txt file. Notice no cache or description snippet, that’s because the pages themselves have not been crawled (not indexed, not in Google). All that Google has to go on is the fact that someone else linked to the page and the text they used.

The anomaly this highlights though, is that when a page is linking to another that is blocked by the robots.txt file Google opts to display the link text from the linking page as the result title, which when you think about it is actually quite a serious flaw in their treatment of the robots.txt protocol.

It means Google has an exploitable loophole allowing one website to control the representation of another website within the algorithmic results page, and effectively own the title space of another websites results, for content that website did not want included in the index. Google is showing priority to the decision of the linker to link, over the content owner who wants it excluded. In a worst case scenario this could be defamatory text or something libellous and it would appear to be coming from the website being linked to. In the examples below the Title text is coming from links on a review page of UK search engine marketing agencies http://www.sci7.com/cms/62/uk-google-qualified-professionals.html supposedly we are one of the trendy ones, so it's definitely relevant to the query ;-) but there is potential this would not always be the case.

It's my opinion that Google would be much better off displaying only the URL and not third party link text in the results page.

Comments

algoholic

Great example Teddy, This also happens a lot with affiliate links, with enough links they outrank the non parameter versions of landing pages, even when they are blocked via robots.txt or the meta robots tag.

Teddie

If I had a large affiliate network that was causing that particular problem, I'd be more inclined to set a server side trigger in my pages activated by the presence of the affiliate tracking ID, that allowed them to register the affiliate, drop the cookie and then force a 301 permanent redirect to the page I wanted to rank.

More sales + better desired rankings + more control of the brand. Problem fixed :-)

Spanish Property

Someone needs to come in and change the whole way it works,that would shake things up!!
Lucy

mac makeup bags

nice post and i like it very much~it's so useful,thanku for sharing it~

Properties in Spain

Im so amazed, things on Google are still this way. It was set up to help small companies out rank big companies, but like everything you can pay to get ahead of the rest.. great article.

The comments to this entry are closed.

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451c37d69e200d8341c8d5253ef

Listed below are links to weblogs that reference The power of links – non-indexed pages out-ranking optimised ones + robots.txt flaw.:

» SearchCap: The Day In Search, April 11, 2007 from Search Engine Land: News About Search Engines & Search Marketing
Below is what happened in search today, as reported on Search Engine Land and from other places across the web:... [Read More]

» SearchCap: The Day In Search, April 11, 2007 from Minefeed.com
Below is what happened in search today, as reported on Search Engine Land and from other places across [...] [Read More]

Subscribe to this blog's feed

Add to My Yahoo!
Subscribe with Bloglines
Add to Google
Subscribe in NewsGator Online

Add to My AOL
Add to Technorati Favorites!
Add to netvibes