No. Robots.txt it’s a formal parsing guide for web crawlers (especially for search engines).
With robots.txt you can avoid to appear in unwanted page or sections in search engines, but this can’t stop bots to parse this pages.
No. Robots.txt it’s a formal parsing guide for web crawlers (especially for search engines).
With robots.txt you can avoid to appear in unwanted page or sections in search engines, but this can’t stop bots to parse this pages.