Write a Perfectly Optimized Robots.txt file [Wordpress] | GreatestFart

Optimize Your Robots.txt

Note: As you, people know I am using Blogger Network so I don't know many things about Wordpress. But Don't Worry, I will explain everything to you. So let's get Started.


Importance Of Robots.Txt:

One of the most important parts of SEO is to create the Robots.txt file for your blog/website. Robots.txt is a small file which indicates “Google bot” which page/post to crawl and index and which to leave. So Robots.txt file has importance in basic SEO, but sometimes webmasters are unaware of it.

Robots.txt in WordPress:

Robots.txt file is located in root directory of your blog. When Google bot comes to crawl your website, it first follows Robots.txt file to indicate which page/post to crawl and which page/post to avoid. This file is extremely important for your blog/website. Usually, some webmaster is not aware of this text file, which destroys their blogging career.

Robots.txt file is important for your blog/website safety too. It is a file which does not allow Google Bot to crawl and index your WP-Admin page and thus your blog safety is pretty much not compromised. Robots.txt file has same Importance Like Akismet

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
 }
});

Here User-agent: * means that Robots.txt allow every search engine bot i.e GOOGLE BOT, BING and YAHOO BOTS to crawl the website while DISALLOWING your ADMIN and Content PAGES to be crawled and thus doesn’t compromise your website security.

How Can I Check My Website Robots.txt file:

You can easily check your website/blog Robots.txt file by searching for your website/blog Robots.txt on your browser. Just Search www.your-blog-domain.com/robots.txt in your browser.

Robots.txt file for my Site

Basic Robots.txt File Content

Following Robots.txt file is negotiable for an average blog:

sitemap: http://your-blog-domain.com/sitemap.xml

User-agent:  *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cgi-bin/

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /
 }
});

How Can I Edit My Robots.txt File For WordPress?

There are two ways of editing your Robots.txt file:

1. Via FTP Client.
2. Via SEO Plugin by Yoast.

Update Robots.txt File Via FTP Client:

You can update your Robots.txt via FTP Client like FILE ZILLA. See Step by Step Procedure to update your Robots.txt file via FTP Client:

1. Connect to your blog/website through FTP Client.
2. Go to PUBLIC_HTML of your blog.
3. Right-click and create a new file with notepad on your desktop.
4. Name it “robots.txt”, click to edit it.
5. Copy/Paste the complete code from above.
6. Change the strike-through domain name to your domain name in Sitemap section.
7. Click Save.
8. Upload it to PUBLIC_HTML Folder via FTP CLIENT.
9. Check Your Robots.txt File.

Update Robots.txt File Via SEO Plugin by Yoast:

You can also edit your Robots.txt file if you are having SEO Plugin by Yoast. Following is the Step-by-Step Procedure to setup Robots.txt file via SEO Plugin by Yoast.

1. Connect to your blog/website through FTP Client.
2. Go to PUBLIC_HTML
3. Right-click and create a file.
4. Name it “robost.txt”
5. Log in to your dashboard.
6. Got to “SEO by YOAST” settings.
7. Click on ” Edit Files”.
8. Copy/Paste the Complete Code Above.
9. Change sitemap to your domain.
10. Click on “Save Changes”.

Free Robots.Txt Tip By GreatestFart!

If you have one or two copy pasted posts from other websites. Just disallow them or non-index them in robots.txt file of your blog. Thus Google will not crawl your copied posts and you will be safe from Google Penalization.

Need Help? Ask Us!

If you are not capable of little technical work and cannot edit your Robots.txt file. Ask us, let us do this for you! We will surely help you!