How to Enable Robots.txt file in wordpress

Robots.txt file

Do you know how to enable robots.txt file in WordPress?

Welcome to TheSEOGuy. Today we are going to talk about robots.txt file. In this blog you will learn how to enable or create robots.txt file if you are running a website or maybe a blog on WordPress CMS.

If your wordpress website or blog has problems with robots.txt file, then it will impact performance. Crawlers will have difficulty in reading your website and it’s not good for any portal. In order to fix Robots.txt, first you need to check what is wrong with the file. Today I am going to share some steps by which you can learn how to enable robots.txt file in wordpress. 

Before getting into the process of how to enable or fix robots.txt in wordpress, let’s understand what is robots file and how it works.

Robots.txt File

Robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites.

By using robots file we make sure that crawlers follow website according to our requirements. For example, I have a blog which has some pages/post. I don’t want search engines to cache or follow certain pages or directories. In that case I will use robots.txt file to give directions to crawlers to avoid those pages.

Here is how to see live robots

However, You can also test the robots file in webmaster. All you need to do is, login to webmaster and select the property and choose crawl from the left hand side menu and hit robots.txt tester.

Refer the screenshot below to test robots file

robots

You can test the urls and see if there’s any problem with indexing by using this option in google webmaster console aka search console.

Above mention method is old. In order to test your robots you can now use tool provided by google for free. Given below is the tool which you can use:

Robots Testing Tool

**NOTE** : This option was available in old google webmaster (Search Console). New search console doesn’t have this feature. But you can still see the live robots by following URL method as explained earlier.

How to enable or fix robots.txt file in wordpress

If you are using wordpress and facing this problem then you can solve this problem in minutes. The best way to solve this problem is by using yoast plugin. You have to install yoast plugin into your wordpress. Once you install and activate the plugin, navigate to the SEO › Tools tab in your dashboard, and look for the option that says File editor:

wordpress step one

Clicking on that link will send you to a new page, where you can edit your .htaccess file without leaving your dashboard. There’s also a handy button that reads Create robots.txt file, which does exactly what you’d expect:

Once you click on this button, the tab will display a new editor where you can modify your robots.txt file directly. Keep in mind that Yoast SEO sets its own default rules, which override your existing virtual robots.txt file.

Whenever you add or remove rules, remember to click on the Save changes to robots.txt button, so they persist:

wordpress step three

That’s easy enough! 

However, if you don’t have plugin installed then you can try to download or create generic robots.txt file and upload it by using FTP clients such as filezilla. You can take help of article Upload Files Using the FileZilla FTP Client (Windows & Mac) to know how to upload files using filezilla.

So these were the steps to enable robots.txt file in wordpress. Hope you get the answer of “how to enable robots.txt in wordpress”.

You may also visit Useful SEO Plugins to boost your SEO score. A complete list of plugins which can help you to boost SEO score and improve your blog/website performances.

Hope this information would help you to enable robots.txt file in wordpress. However, if you still have any question, feedback or suggestion for this information then do comment below. All the feedback and suggestions are very meaningful and I take every feedback very seriously.

Keep exploring TheSEOGuy for more amazing SEO Tricks.

Written by 

I am a SEO strategist who loves to solve search engine optimization problems. Created TheSEOGuy with the AIM to provide quality information and case studies on Digital Marketing.

Leave a Reply

Your email address will not be published. Required fields are marked *