A robots.txt file (inserted on the root of your website) is a file that prevents search engines from crawling your entire website. You may want to prevent certain parts of your website from being exposed and searchable to the world. Additionally, as it relates to SEO, several folders and files may not be related to the website’s targeted keywords.

Without a robots.txt file, a search engines assumes it can crawl your entire site for content.

A robots.txt file can be created with a simple notepad editor. Following the standard format, you can choose what files and folders to exclude. You can additionally use a robots.txt file to tell search-engine spiders were your sitemap is located.

You may find that there are some online robots.txt generators that will do the job for you. You can google ‘robots.txt generator’ to find one. After generating your robots.txt file, you would need to save this file and upload, via ftp or your host control panel to the root of your website’s folder.

Simple example of a robots.txt file:

User-Agent: *

Disallow: /images/

Disallow: /directory/file.html

As always, it is recommended that may want to check with some experienced help on how to setup your file correctly. SEO(search engine optimization ) is a gimmick if not applied correctly. Every file, directory, and image of your website needs to be carefully reviewed for content in order to determine exclusions.