How to Enable Robots.txt file in wordpress

Robots.txt file

How to enable robot.txt file in WordPress?

If your wordpress website or blog has problems with robots.txt file, then your website or blogs performance might be in trouble. It is not at all good for website performance. In order to fix Robots.txt, first you need to check what is wrong with the file. Today I am going to share some steps by which you can learn how to enable robots.txt file in wordpress. 

Before getting into the process of how to enable robots.txt in wordpress, let’s understand what is robots file and how it works.

Robots.txt file

Robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. By using robots file we make sure that crawlers follow website according to our requirements. For example, I have a blog which has some pages/post which I don’t want search engines to cache or follow then I can use this file to do that. I can adjust my robots file for the website so that crawlers would read the information which I want them to read.

Here is how to see live robots

In order to see live robots you can try to open live robots file by writing your domain name followed by robots.txt in url. For example type the full root domain of your website or blog like example.com and add robots.txt at the end and hit enter. Let’s take example of example domain. If I want to see the live robots for example domain then I will write example.com/robots.txt and hit enter. After that I will be able to see live robots for example domain. 

However, You can also test the robots file in webmaster. All you need to do is, login to webmaster and select the property and choose crawl from the left hand side menu and hit robots.txt tester.

Refer the screenshot below to test robots file

robots

You can test the urls and see if there’s any problem with indexing by using this option in google webmaster console aka search console.

**NOTE** : This option was available in old google webmaster (Search Console). New search console doesn’t have this feature. But you can still see the live robots by following URL method as explained earlier.

How to enable or fix robots.txt file in wordpress

If you are using wordpress and facing this problem then you can solve this problem in minutes. The best way to solve this problem is by using yoast plugin. You have to install yoast plugin into your wordpress. Once you install and activate the plugin, navigate to the SEO › Tools tab in your dashboard, and look for the option that says File editor:

wordpress step one

Clicking on that link will send you to a new page, where you can edit your .htaccess file without leaving your dashboard. There’s also a handy button that reads Create robots.txt file, which does exactly what you’d expect:

Once you click on this button, the tab will display a new editor where you can modify your robots.txt file directly. Keep in mind that Yoast SEO sets its own default rules, which override your existing virtual robots.txt file.

Whenever you add or remove rules, remember to click on the Save changes to robots.txt button, so they persist:

wordpress step three

That’s easy enough! 

However, if you don’t have plugin installed then you can try to download or create generic robots.txt file and upload it by using FTP clients such as filezilla. You can take help of article Upload Files Using the FileZilla FTP Client (Windows & Mac) to know how to upload files using filezilla.

So these were the steps to enable robots.txt file in wordpress. Hope you get the answer of “how to enable robots.txt in wordpress” If you still have any question, feedback or suggestion for this information then do comment below.

Written by 

I am a SEO strategist who loves to solve search engine optimization problems. Created TheSEOGuy with the AIM to provide a better information platform for people who are looking for SEO Answers.

Leave a Reply

Your email address will not be published. Required fields are marked *