How to enable robot.txt file in WordPress?
Having problems with robots.txt file is not good for website performance. In order to fix Robots.txt, first you need to check what is wrong with the file. In order to see live robots you can try to open live robots file by writing your domain name followed by robots.txt in url. For example you have to write example.com/robots.txt and hit enter to see live robots.
However, You can also test the robots file in webmaster. All you need to do is, login to webmaster and select the property and choose crawl from the left hand side menu and hit robots.txt tester.
Refer the screenshot below to test robots file
You can test the urls and see if there’s any problem with indexing by using this option in google webmaster console aka search console.
How to fix robots.txt file in wordpress
If you are using wordpress and facing this problem then you can solve this problem in minutes. The best way to solve this problem is by using yoast plugin. You have to install yoast plugin into your wordpress. Once you install and activate the plugin, navigate to the SEO › Tools tab in your dashboard, and look for the option that says File editor:
Clicking on that link will send you to a new page, where you can edit your .htaccess file without leaving your dashboard. There’s also a handy button that reads Create robots.txt file, which does exactly what you’d expect:
Once you click on this button, the tab will display a new editor where you can modify your robots.txt file directly. Keep in mind that Yoast SEO sets its own default rules, which override your existing virtual robots.txt file.
Whenever you add or remove rules, remember to click on the Save changes to robots.txt button, so they persist:
That’s easy enough! Now let’s see how another popular plugin does the same thing.
However, if you don’t have plugin installed then you can try to download or create generic robots.txt file and upload it by using filezilla.
Follow Upload Files Using the FileZilla FTP Client (Windows & Mac) to know how to upload files using filezilla.