How to create your WordPress Robots.txt file for better SEO

Robot.txt file acts as instruction manual for web crawlers while visting your site or blog. It instructs the bots /crawlers/spiders where they can go on your website.

With this file, you are asking bots- what they are allowed to see on your site and what is off limits.

Robot.txt is an incredibly powerful tool when it comes to your site SEO. It’s a way to increase your SEO by taking advantage of a natural part of every website that rarely gets talked about.

Here is why.

Search bots have a crawl quota for each website. Meanwhile, they crawl a certain number of pages during a crawl session. If they don’t end up crawling all pages on your site, then they will come back and resume crawl in the next session.

This can slow down your website indexing rate.

As a solution, you can disallow search bots from attempting to crawl discretionary pages. Doing this, you will save your crawl quota and also helps search engines crawl even more pages on your site and index them as quickly as possible.

The syntax for a robots.txt file:

User-agent: [user-agent name]
Disallow: [URL string not to be crawled]
User-agent: [user-agent name]
Allow: [URL string to be crawled]
Sitemap: [URL of your XML Sitemap]

This is the basic format of Robot.txt file. You can add multiple instructions to allow or disallow specific URLs/Directories. Similarly, you can add multiple sitemaps. If you don’t input any URL to disallow, then search engine bot can crawl any URL (as mentioned above).

For example:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml

In the above example, I’ve disallowed search bots to crawl WordPress plugin files, WordPress admin area.

So, this is an ideal look of the Robot.txt file. Now, let’s see the way to create Robot.txt file for your site.

For this task, I strongly recommend using Yoast SEO plugin to create and edit it directly from your WordPress admin area. Yoast SEO automatically creates Robot.txt file for you.

Install Yoast SEO and navigate to Tools page under SEO menu. Here you can see the robots.txt file of your website/blog. If you don’t have a robots.txt file, create one by hitting Create robots.txt file button.

Creating robots.txt file using Yoast SEO plugin
Creating robots.txt file using Yoast SEO plugin

If you don’t see option for file editor, it is because you have harden WordPress security. You can enable file editing option by removing following line from wp-config.php file.

define('DISALLOW_FILE_EDIT', true);

Your existing Robot.txt file will appear as: 

User-agent: *
Disallow: /

You have to delete the text as it blocks all search engines. Once done with deleting, you can edit the file as I noted above.

Now test the file using a robots.txt tester tool. The best tool you have- Google Search Console.

Visit your Google search console >> launch the robots.txt tester tool located under ‘Crawl’ menu.

It will automatically fetch your Robot.txt file for your website and highlight the errors and warnings if it found any.

NOTE: Do not block CSS or JS folders as a bot can view your site as a real user and if your pages need the JS and CSS to function properly, they should not be blocked.

Conclusion:

The basic aim of optimizing your Robot.txt file is to hide publicly unavailable pages from search engine bots. Through this guide, I hope- you don’t have to invest too much time configuring or testing your robots.txt.

Comment your reviews in the comment section below.

 

ALSO READ

Subscribe to our newsletter

To be updated with all the latest news, offers and special announcements.

Comment Policy: Your words are your own, so be nice and helpful if you can. Please, only use your real name and limit the amount of links submitted in your comment. We accept clean XHTML in comments, but don't overdo it please.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

GET STARTED

Start WordPress Blog: Ultimate Guide For Beginners

Here is our step by step guide for beginners to start a WordPress blog.

Top 6 Best WordPress Theme Frameworks

There are many websites themes which are created by using a framework. Likewise, WordPress Theme frameworks also becoming so popular. The framework allows you...

The Most Essential WordPress Plugins

Here, I have listed some excellent plugins which will surely add more features and new functionality to your WordPress blog.

Tips to Speed Up WordPress Performance

Here, I am sharing some tips to speed up WordPress performance. All you need to modify .htaccess file on your server and add some plugins.

The Ultimate Guide to Harden WordPress Security

Learn how to improve WordPress security. Here are some basic security concepts that will help you to harden WordPress security.

How to Setup CloudFlare Free SSL for WordPress Blog

Google has announced that they will count HTTPS as a ranking factor which means if you using HTTPS you will get a higher position...

DEALS

WP Coupons

The #1 WordPress coupon plugin for affiliate marketers to increase your CTR and conversions!

POPULAR NOW

How to Remove jQuery Migrate in WordPress?

jQuery Migrate greatly simplifies the process of moving older jQuery code to a higher jQuery version by identifying deprecated features. It then restores deprecated...

Get almost 100% PageSpeed on Newspaper theme

The newspaper theme is the most popular premium theme for a self-hosted WordPress blog. The theme is developed by team tagDiv. You...

How to Delete Multiple Images in WordPress Media Gallery

Hey, guys today one of my friends asked me a question, How can I delete Multiple Images in WordPress? I replied, "It's very simple." Yeah,...

Xperia Home Launcher: Install on any Android Device (No Root)

Hey, guys today through this tutorial you are going to learn how to install Xperia Home Launcher on any Android device without any root. Also...

Tips to Speed Up WordPress Performance

Here, I am sharing some tips to speed up WordPress performance. All you need to modify .htaccess file on your server and add some plugins.

How to Display Featured Image Before or After Entry Title in Genesis

Learn how to add featured image to a single post or a page in Genesis child themes. You can set it to display featured...