robots.txt protocolThe robots.txt protocol, also known as the Robots Exclusion Protocol is to prevent web spiders from accesing all or part of a website that is publicly viewable. Robots are used by search engines to categorize and archive web sites. A robots.txt file will function as a request that specified robots ignore specified files or directories in their search.You can use a robots.txt generator to create one and upload it to your public_html directory. |
