Many person does not know importance of robots.txt file and how to use robots.txt file in wordpress. robots.txt file is important as per search engine.  The robots.txt file is very important as per search engine and SEO prospective.

how to use robots.txt file in wordpress

how to use robots.txt file in wordpress

how to use robots.txt file in wordpress

What is robots.txt file?

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

In wordpress you should put robots.txt file. you can use following code in robots.txt file and put that file under your ROOT directory( wordpress main folder)

User-agent: *
Allow: /
Disallow: /*?s=
Disallow: /wp-admin/*
Disallow: /wp-content/*
Disallow: /wp-includes/*
Disallow: /wp-content/cache
Disallow: /wp-content/themes/*<code></code>
Disallow: /wp-content/plugins/*

Important Note: if you are using the QA or testing enviourment then put following lines in your theme header.php file. You can put the domain name condition also.

if($SERVER)['HTTP_HOST']=='Testing domain name') {

<?php } ?>

You should not disallow the images and comments also because that is more important for SEO purpose. I recommend to use always good permalink structure for your wordpress installation.

Written by Purab

My name is Purab, Here is my Personal Blog: I contributor of many wordpress plugins and Drupal Modules. I am core developer of Drupal. Connect to on LinkedIn or Follow @purab on Twitter or Follow me on Google+ or Write Mail to or Chat with me on Skype : purabdk

Leave a Comment

Your email address will not be published. Required fields are marked *