Some months ago, I was creating my first robots.txt. I even asked someone to help me make one. After a few months though, I’m a bit more confident in knowing WordPress and so I revisited my robots.txt again and start doing some more research to optimize my search engines traffic even further.

I found out that I still have a lot of duplicated contents that need to be filtered out to get a good SEO (Search Engine Optimization)!

If you don’t know what robots.txt is, it’s the file that search engine bots/crawlers will look first on their visits to your site/blog. The file tells them what to crawl/info to grab and what’s not.

The file robots.txt has to be put on your root site, even if your WordPress is installed on a sub-folder! So since my blog’s URL is http://www.michaelaulia.com/blogs/ , I still have to put the robots.txt under the http://www.michaelaulia.com/  (or your public_html/ folder).

Here is my new robots.txt file: (Feel free to comment about it)

# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.michaelaulia.com/blogs/sitemap.xml.gz
# END XML-SITEMAP-PLUGIN

User-agent: Googlebot-Image
Disallow:

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

User-agent: *
Disallow: /blogs/cgi-bin/
Disallow: /blogs/wp-admin/
Disallow: /blogs/wp-includes/
Disallow: /blogs/wp-content/plugins/
Disallow: /blogs/wp-content/cache/
Disallow: /blogs/wp-content/themes/
Disallow: /blogs/author/
Disallow: /blogs/archives/
Disallow: /blogs/trackback/
Disallow: /blogs/feed/
Disallow: /blogs/tag/
Disallow: /blogs/search-result/
Disallow: /blogs/smilies/
Disallow: /blogs/wp-au-backup/
Disallow: /blogs/category/
Disallow: /blogs/page/
Disallow: /blogs/2007/
Disallow: /blogs/2008/

———-

NOTE: If you want to copy my robots.txt to your WordPress blog, feel free to do so, BUT! This only works if your permalink structure is similar like mine (www……./%posttitle%……. IF your permalink structure has the year or category on it, it will be blocked by this robots.txt configuration! (i.e. the Disallow: /2008/ part)

As always, check whether your posts are accessible by using Google Webmaster Tools.

Once there, go to Tools-Analyze robots.txt.

You should then see your robots.txt contents there. If you’ve just updated your robots.txt file, you may still see the old one. It will be refreshed on the next Google’s crawl which may take a day or two.

Then, test if the crawler bots can access your actual content and can’t access the duplicated contents:

Then, look at the results to see if the bot can access only the actual content.

As you can see, the bot can now only access the actual post content and not the posts on archives, feeds, navigation pages, etc.

Have you re-visited your robots.txt? It’s very important for search engines, especially Google, that you get it right and optimized!

Like this post? Make sure that you get my latest posts delivered to you by subscribing via email or your favorite feed reader. If you are a blogger, feel free to leave a trackback to this post.