WordPress Tutorials

How To Stop Search Engines From Crawling A WordPress Site

Stopping search engines from crawling a WordPress is not as easy as pie. You will be able to overcome this hurdle if you go through this article with an apt attention. You may be having a duplicate content right through your website which you do not want to be deleted and also you do not want your search engines to penalize and index your website. For example, consider that you want a simple and quick page that guards users from getting content till they provide their email. This is a good experience for those people who make endeavor to make their mailing list. They will provide a free resource of some kind in return for your email address.

From the complicated registrations that require username and password to password protecting files in a directory making use of .htaccess to the use of Meta tags on robots.txt file or pages, there are so many ways to conceal pages from both search engines and users. Below, some tips on how to how to stop search engines from crawling a WordPress site are being explained:

Ascertain The Cause Why Anyone Would Like To Block Search Engines

It may sound incredible to you to know that there are lots of users who like to work on their sites by keeping them on accessible domain rather than making a development site. Some people make project management sites by using WordPress. There are numerous people who make use of WordPress for making private blogs. In all such conditions, you perhaps do not have intentions to be indexed by search engines and you are found in the condition where you are not prepared for that. A misunderstanding is that in case you do not have links that point to your domain, then search engines will perhaps never find your website.

There are many ways through which search engines can get a website linked elsewhere. For instance, consider the following things:

  • There are lots of pages with a list of domain names. There is probability that your site can emerge on one of those.
  • Some results of domain search site might be indexed with your link on them.
  • Maybe your domain name might have been owned by some other people and they have some links that point to your website now.

There are lots of things that happen on the web and none of these are controllable by you. Nevertheless, your website is under your control and so you can provide instructions to search engines for not following or indexing your website.

Stopping Search Engines From Indexing And Crawling A WordPress site

WordPress is available with such a feature that lets you provide instructions to search engines for not indexing your site. What you need to do is to pay a visit to Setting, read and check the box placed next to the option of Search Engine Visibility. Such lines call upon web crawlers for not indexing your pages. Nevertheless, it depends on search engines to accept this request or avoid it.  Although many search engines do pay respect to this, some random image or some page belonging to your site are indexed.

Ensuring Your Site Fails To Emerge In Search Results

The most powerful way of blocking search engines from a website is by password that provides protection to your whole WordPress site present on the server level. This thing reveals that when anybody has access to your website, he or she is asked to give his or her username and password prior to reaching WordPress. This involves search engines also. When he or she fails in login attempt, he or she is provided 401 error and the bots are turned away.

Password Providing Protection To Whole Site Using cPanel

In case, WordPress hosting provider provides cPanel access for management of your hosting account, then you can provide protection to your whole site by making use of cPanel. What you need to do is to login to cPanel dashboard before your click on password protect directories.

On another screen, you will have to choose the folder where site of WordPress is installed. Usually, it is public www directory or public_html. After that, you will have to check the box which is placed next to the option of ‘Password protect this directory’. Your next step involves providing your name to the protected directory and pressing the save button. After some time, you will get to perceive visually a success message and link for going back. Clicking on this link will bring you to the screen of password protection. At this time, you are supposed to give your user name and password, which is required for viewing your website.

So, these were some of the points that you need to chew on for stopping searching engines from crawling a WordPress site.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.