Share what you know with millions of people
Focus is the best place to turn what you know into remarkable content
0
How do you hide URLs that you don't want indexed from a spider?
I have some pages on my site that I don't want to show up in search results. How can I hide these from spiders?
Events
- Dos and Don'ts of Small Business Marketing May 29 @ 11 am PT
- Lead Nurturing 202: The Next Generation May 31 @ 11 am PT
- The Tricks to Paid Media June 6 @ 11 am PT
- Display Advertising for Brand Awareness June 20 @ 11 am PT
Related Activity
5 Answers
Use a "No Follow Command" in your HTML. Aaron Wall's community and free Toolbar is a great resource. Here is a link with more background on this:
http://www.seobook.com/robots-txt-vs-rel-nofollow-vs-meta-robots-nofollow
robots.txt has to be the way to go.
http://www.robotstxt.org/robotstxt.html
SEO Book is out of date now on that particular subject. The table of data was created based on an interview with Matt Cutts Published: October 8, 2007
I can see Google checks my robots.txt by checking my weblogs. What they outline would only apply if you control all links to the page, even those on third party sites.
Google does respect robots.txt
"Google won't crawl or index the content of pages blocked by robots.txt"
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
It may end up indexing a page as a result of a third party link to the page however, in which case use the removal request tool:
https://www.google.com/webmasters/tools/removals?pli=1
Thanks Andrew - should have been more specific about Robots.txt. Great answer....kudos.
My apologies I should have noted that adding the Meta Robots tag would also be a good idea
\\ META NAME="ROBOTS" CONTENT="NOINDEX" //
In addition to the robots.txt exclusion
There is no no harm, so where possible also use the "nofollow" as Lee pointed out when linking to the page.
By using a Robots.txt file in the directory.
Answer This Question