robots.txt is a text file that tells search engine robots which pages on your website to index and which to ignore. The robots meta tag can also be...

read more