Post by satish11nov » Tue May 07, 2013 8:20 pm

Hi All,

I want to add robots.txt for opencart installation. Below is the content .


User-Agent: *
Disallow: /admin/
Disallow: /cache/
Disallow: /library/
Disallow: /system/
Disallow: /catalog/
Disallow: /download/
Disallow: /image/
Disallow: /cgi-bin/
Disallow: /vqmod
Allow: /

Is this fine ? do i need to following paths ?
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search
Disallow: /*?route=affiliate/

Please help me on this.

Newbie

Posts

Joined
Fri Nov 30, 2012 6:14 pm

Post by rph » Tue May 07, 2013 10:37 pm

It's important to remember that robots.txt doesn't provide security for those directories (the exact opposite in fact). It just suggests to search engines areas that are inefficient to spend time indexing. I would recommend using:

Code: Select all

User-agent: *
Disallow: /*&filter
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=account/
Disallow: /*?route=affiliate/
Disallow: /*?route=checkout/
Disallow: /*?route=product/search

-Ryan


rph
Expert Member

Posts

Joined
Fri Jan 08, 2010 5:05 am
Location - Lincoln, Nebraska

Post by Avvici » Thu May 09, 2013 6:17 am

satish11nov wrote:Hi All,

I want to add robots.txt for opencart installation. Below is the content .


User-Agent: *
Disallow: /admin/
Disallow: /cache/
Disallow: /library/
Disallow: /system/
Disallow: /catalog/
Disallow: /download/
Disallow: /image/
Disallow: /cgi-bin/
Disallow: /vqmod
Allow: /

Is this fine ? do i need to following paths ?
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search
Disallow: /*?route=affiliate/

Please help me on this.
Other than what RPH suggested, you also need to know that you should never use robots.txt to totally steer robots away from your website. You should always use a <meta tag="robots" content="noindex,nofollow"> in the header of your site. Otherwise you can run the risk in Google not re-indexing you for months.

User avatar
Expert Member

Posts

Joined
Tue Apr 05, 2011 12:09 pm
Location - Asheville, NC

Post by victorj » Thu May 09, 2013 6:23 am

[/quote]

Other than what RPH suggested, you also need to know that you should never use robots.txt to totally steer robots away from your website. You should always use a <meta tag="robots" content="noindex,nofollow"> in the header of your site. Otherwise you can run the risk in Google not re-indexing you for months.[/quote]

i was under the impression that it should be index, follow as noindex,nofollow would tell google etc to stay away

Koeltechnische deurrubbers eenvoudig online op maat bestellen.
Alle niet stekplichtige onderdelen zoals scharnieren, sloten, randverwarming en verlichting voor alle typen koelingen en vriezers.
https://koelcel-onderdelen.com


User avatar
Expert Member
Online

Posts

Joined
Sat Jun 25, 2011 4:09 am
Location - Alkmaar Holland

Post by Avvici » Thu May 09, 2013 4:00 pm

Please re-read what I wrote. And no, sometimes you need to tell robots to stay away if you are not ready to launch on the WWW. When developing websites sometimes it's necessary to launch the domain before it's actually time to want search engines to crawl. This is what the meta tag is used for.

User avatar
Expert Member

Posts

Joined
Tue Apr 05, 2011 12:09 pm
Location - Asheville, NC
Who is online

Users browsing this forum: No registered users and 33 guests