Robots.txt is an instruction file on a website, that gives search crawlers instructions, on how to crawl, and index a website. Most times it is only obeyed by good crawlers, because some bad bots, disobey the robots.txt file completely.

How to Find Website Robots.txt File

To find your website robots.txt file, just add robots.txt to the end of your website url. For example, if your url is (https://www.yesterblog.com), your robots.txt will be located at (https://www.yesterblog.com/robots.txt