Here is how to enable robots.txt file in WordPress
If your wordpress website or blog has problems with robots.txt file, then your website or blogs performance might be in trouble. It is not at all good for website performance. In order to fix Robots.txt, first you need to check what is wrong with the file. Today I am going to share some steps by which you can learn how to enable robots.txt file in wordpress.
Before getting into the process of how to enable robots.txt in wordpress, let’s understand what is robots file and how it works.
Robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. By using robots file we make sure that crawlers follow website according to our requirements. For example, I have a blog which has some pages/post which I don’t want search engines to cache or follow then I can use this file to do that. I can adjust my robots file for the website so that crawlers would read the information which I want them to read.
Here is how to see live robots
In order to see live robots you can try to open live robots file by writing your domain name followed by robots.txt in url. For example type the full root domain of your website or blog like example.com and add robots.txt at the end and hit enter. Let’s take example of example domain. If I want to see the live robots for example domain then I will write example.com/robots.txt and hit enter. After that I will be able to see live robots for example domain.
However, You can also test the robots file in webmaster. All you need to do is, login to webmaster and select the property and choose crawl from the left hand side menu and hit robots.txt tester.
Refer the screenshot below to test robots file
You can test the urls and see if there’s any problem with indexing by using this option in google webmaster console aka search console.
**NOTE** : This option was available in old google webmaster (Search Console). New search console doesn’t have this feature. But you can still see the live robots by following URL method as explained earlier.
How to enable or fix robots.txt file in wordpress
If you are using wordpress and facing this problem then you can solve this problem in minutes. The best way to solve this problem is by using yoast plugin. You have to install yoast plugin into your wordpress. Once you install and activate the plugin, navigate to the SEO › Tools tab in your dashboard, and look for the option that says File editor:
Clicking on that link will send you to a new page, where you can edit your .htaccess file without leaving your dashboard. There’s also a handy button that reads Create robots.txt file, which does exactly what you’d expect:
Once you click on this button, the tab will display a new editor where you can modify your robots.txt file directly. Keep in mind that Yoast SEO sets its own default rules, which override your existing virtual robots.txt file.
Whenever you add or remove rules, remember to click on the Save changes to robots.txt button, so they persist:
That’s easy enough!
However, if you don’t have plugin installed then you can try to download or create generic robots.txt file and upload it by using FTP clients such as filezilla. You can take help of article Upload Files Using the FileZilla FTP Client (Windows & Mac) to know how to upload files using filezilla.
So these were the steps to enable robots.txt file in wordpress. Hope you get the answer of “how to enable robots.txt in wordpress”.
FAQs – Robots.txt file
To find live robots file of a website you can simply type the robots.txt after URL and hit enter. For example: www.example.com/robots.txt.
Robots. txt file informs search engine crawlers about page availability and tell them about which pages or files the crawler can or cannot request from your site. This is used mainly to avoid overloading your site with requests. It is not a mechanism for keeping a web page out of Google. You can find more about Robotx.txt file here
You may also visit Useful SEO Plugins to boost your SEO score. A complete list of plugins which can help you to boost SEO score and improve your blog/website performances.
Hope this information would help you to enable robots.txt file in wordpress. However, if you still have any question, feedback or suggestion for this information then do comment below. All the feedback and suggestions are very meaningful and I take every feedback very seriously.
Keep exploring TheSEOGuy for more amazing SEO Tricks.