1. How to allow all search engine spiders to index all files
Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your web site:
User-agent: *
Disallow:
2. How to disallow all spiders to index any file
If you don't want search engines to index any file of your web site, use the following:
User-agent: *
Disallow: /
3. Where to find more complex examples.
If you want to see more complex examples, of robots.txt files, view the robots.txt files of big web sites:
*
o http://www.cnn.com/robots.txt
o http://www.nytimes.com/robots.txt
o http://www.spiegel.com/robots.txt
o http://www.ebay.com/robots.txt
Your web site should have a proper robots.txt file if you want to have good rankings on search engines. Only if search engines know what to do with your pages, they can give you a good ranking.
Sumber :..
 |
Informasi 
Subscribe to this comment's feed
Show/hide comments
Show/hide comment form
RSS FEED