Post by uege_uege » Mon Jun 28, 2010 12:09 am

Hello!

I know it is good to have Robots.txt file.
So what should be the default?
This is what i done:

User-agent: *
Disallow: /adminFolder/
Disallow: /catalog/
Disallow: /download/
Disallow: /image/
Disallow: /system/



Is this right?

Newbie

Posts

Joined
Sun Jun 13, 2010 4:43 pm

Post by JAY6390 » Mon Jun 28, 2010 12:34 am

To be honest, I see no real advantage to using a robots.txt file other than to stop google indexing pages you don't want it to. If you don't have a link to your admin pages anywhere on your site (which you shouldn't) then google will never spider there as it doesn't guess what links you have, it reads your site, collates links and goes to where they are, recursively doing the same on those pages

The same is true for the image folders and the likes

Image


User avatar
Guru Member

Posts

Joined
Wed May 26, 2010 11:47 pm
Location - United Kingdom

Post by uege_uege » Mon Jun 28, 2010 2:49 am

JAY6390 wrote:To be honest, I see no real advantage to using a robots.txt file other than to stop google indexing pages you don't want it to. If you don't have a link to your admin pages anywhere on your site (which you shouldn't) then google will never spider there as it doesn't guess what links you have, it reads your site, collates links and goes to where they are, recursively doing the same on those pages

The same is true for the image folders and the likes
Yes, it is for All search engines...and not just the Google ones... Also, some of these bots are not so "clever" and they browse something, which is not need to.. ;)
That's why the smart people discovered the robots.txt file, there is it in the web masters tools into Google. And google will bot up all which it can, it is natural. So, you are not right at your note. There is tons of information about that...
So, do you know the proper "default" robots.txt for opencart?

Newbie

Posts

Joined
Sun Jun 13, 2010 4:43 pm

Post by SteveSherry » Mon Jun 28, 2010 4:52 am

My robots file is:

Code: Select all

User-agent: * 
Disallow: /admin/
Disallow: /index.php?route=common/home
Disallow: /index.php?route=product/search
Disallow: /index.php?route=product/category
Disallow: /index.php?route=information/information
not a bad idea on the other folders also, i think i'll add them to mine

My Website ¦ Summer Madness Special Offer ¦


Active Member

Posts

Joined
Thu Apr 08, 2010 7:47 am
Location - Wirral, UK

Post by Xsecrets » Mon Jun 28, 2010 5:46 am

why would you not want google to spider your information pages and category pages?

OpenCart commercial mods and development http://spotonsolutions.net
Layered Navigation
Shipment Tracking
Vehicle Year/Make/Model Filter


Guru Member

Posts

Joined
Sun Oct 25, 2009 3:51 am
Location - FL US

Post by SteveSherry » Mon Jun 28, 2010 5:54 am

I've got this set up for the new products that I haven't seo'd yet.

I sometimes set up a product in a rush, and don't always give it the full care and attention I want, so in order to not have the rushed or temporary product indexed by google I block the category page.

My Website ¦ Summer Madness Special Offer ¦


Active Member

Posts

Joined
Thu Apr 08, 2010 7:47 am
Location - Wirral, UK

Post by JAY6390 » Mon Jun 28, 2010 6:06 am

The thing is, unless you link to yoursite.com/admin/ then google or any other bot won't try and spider it, so I just find it pretty pointless. If you've got pages that are visible to the public and linked then that's fair enough, but otherwise it's pretty redundant. ALSO, giving a robots.txt file your admin if anything is worse, since anyone with half a brain can read your robots.txt file, find your admin folder name and start hammering away at trying to break into your admin area

Image


User avatar
Guru Member

Posts

Joined
Wed May 26, 2010 11:47 pm
Location - United Kingdom

Post by Karen » Wed Oct 27, 2010 11:56 am

There does seem to be one help in having a robots.txt file, which is to direct bots to your xml sitemap. I'm wondering what effect it has to not have a robots.txt file, since they all do seem to look for it.

Using OC version 1.4.8b
http://catandcaboodle.com/


User avatar
Active Member

Posts

Joined
Thu Jun 10, 2010 2:51 pm
Location - WA State, USA

Post by Qphoria » Thu Aug 18, 2011 8:59 pm

That was the default back in 1995. that is hte OLD "open-cart" project which was moved to "opencart"

Image
Donate!|OpenCart Basics|GeoZones
Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by pwelku » Wed Jan 04, 2012 11:59 pm

Hi Guys, I get the idea of what the txt should say but how do I find the this in the root file. For example break it down like cataloge-model-ect ect/ I am a complete newbie and need exact instructions. Also, once I have changed the files to disallow how do I upload that. Again please remember i am new. Thank you guys in advance.

Newbie

Posts

Joined
Wed Jan 04, 2012 4:03 am

Post by Avvici » Thu May 24, 2012 6:54 pm

I agree. There is no harm in having one as long as you have it configured correctly. Especially for sitemap location.

User avatar
Expert Member

Posts

Joined
Tue Apr 05, 2011 12:09 pm
Location - Asheville, NC

Post by Solan » Tue Jun 19, 2012 7:39 am

trankhanh89 wrote:in SEO's perspective, robots.txt is really good choice to improve your site as it helps SE to have a direction when crawling your site.

By defalt, Opencart is SEO friendly already. However, adding robots.txt will make your site better than your competitor's.

This one (below) in my SEO's experience is quite a good one to consider (to enable sitemap to google, log in to your admin and go to extension-->productfeed, then enable Google sitemap) :

user-agent: *
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search
Disallow: /*?route=affiliate/


Sitemap: http://yoursite.com/index.php?route=feed/google_sitemap

Do you mean when i go to extension-->productfeed and enable google sitemap i shall write (in the text area field) all of the below:

user-agent: *
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search
Disallow: /*?route=affiliate/

Sitemap: http://mysite.com/index.php?route=feed/google_sitemap


Is this correct?

Active Member

Posts

Joined
Sat Feb 19, 2011 7:34 pm

Post by Devbee » Wed Jan 15, 2014 10:55 pm

Why We are adding

Disallow: /*?route=product/search

It actually blocks the products for crawling right?

Newbie

Posts

Joined
Wed Jan 15, 2014 10:47 pm

Post by williamino » Thu May 09, 2019 4:20 am

create a file in the folder where you installed opencart
call the file robots.txt
open the this robot.txt file in edit
past your code, click save and exit out of it.
done.

User avatar
Newbie

Posts

Joined
Sat Apr 09, 2016 1:25 am
Location - Australia
Who is online

Users browsing this forum: No registered users and 63 guests