What is robots.txt?
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots like Google, Bing, Yahoo) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
A robots.txt file lives at the root of your site. So, for site www.icreativez.com, the robots.txt file lives at www.icreativez.com/robots.txt. robots.txt is a plain text file that follows the Robots Exclusion Standard.
Here is a simple robots.txt file with two rules, explained below:
# Group 1
User-agent: Googlebot
Disallow: /nogooglebot/
# Group 2
User-agent: *
Allow: /
robots.txt disallow all
Blocking all web crawlers from all content
User-agent: * Disallow: /
Using this syntax in a robots.txt file would tell all web crawlers not to crawl any pages on www.yourwebsite.com, including the homepage.
robots.txt no index
You can prevent a page from appearing in crawlersh by including a noindex meta tag in the page's HTML code, or by returning a noindex header in the HTTP request.
<meta name="robots" content="noindex">