Post by kg0925 » Tue Sep 25, 2018 1:09 pm

please anyone help me with robots.txt file. i don't know much about it. before i was not using robot file in my website root. but after adding it my website traffic rank dropped. may be some useful links blocked by robot file.
so please suggest well tested and seo friendly best robot file for my store. thanks.

New member

Posts

Joined
Wed Mar 29, 2017 11:09 pm

Post by khnaz35 » Tue Sep 25, 2018 1:25 pm

kg0925 wrote:
Tue Sep 25, 2018 1:09 pm
please anyone help me with robots.txt file. i don't know much about it. before i was not using robot file in my website root. but after adding it my website traffic rank dropped. may be some useful links blocked by robot file.
so please suggest well tested and seo friendly best robot file for my store. thanks.
Share your robots.txt here

Urgent Questions shoot here: khnaz35@gmail.com
Enjoy nature ;) :) :-*


User avatar
Active Member

Posts

Joined
Mon Aug 27, 2018 11:30 pm
Location - Malaysia

Post by kg0925 » Tue Sep 25, 2018 2:35 pm

khnaz35 wrote:
Tue Sep 25, 2018 1:25 pm
kg0925 wrote:
Tue Sep 25, 2018 1:09 pm
please anyone help me with robots.txt file. i don't know much about it. before i was not using robot file in my website root. but after adding it my website traffic rank dropped. may be some useful links blocked by robot file.
so please suggest well tested and seo friendly best robot file for my store. thanks.
Share your robots.txt here
User-agent: *
Disallow: /*&limit
Disallow: /*?limit
Disallow: /*?sort
Disallow: /*&sort
Disallow: /*?order
Disallow: /*&order
Disallow: /*?price
Disallow: /*&price
Disallow: /*?brand_tabletpc
Disallow: /*&brand_tabletpc
Disallow: /*?color_default
Disallow: /*&color_default
Disallow: /*?filter_tag
Disallow: /*&filter_tag
Disallow: /*?mode
Disallow: /*&mode
Disallow: /*?cat
Disallow: /*&cat
Disallow: /*?dir
Disallow: /*&dir
Disallow: /*?color
Disallow: /*&color
Disallow: /*?product_id
Disallow: /*&product_id
Disallow: /*?minprice
Disallow: /*&minprice
Disallow: /*?maxprice
Disallow: /*&maxprice
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search
Disallow: /*?page=1
Disallow: /*&create=1
Disallow: /?route=information/contact
Disallow: /*?route=affiliate/
Disallow: /*?keyword
Disallow: /*?av
Disallow: /admin/
Disallow: /system/
Disallow: /catalog/

New member

Posts

Joined
Wed Mar 29, 2017 11:09 pm

Post by paulfeakins » Tue Sep 25, 2018 10:18 pm

Why are you disallowing indexing of almost all your site lol?

For quick, professional OpenCart support please email info@antropy.co.uk


User avatar
Guru Member

Posts

Joined
Mon Aug 22, 2011 11:01 pm
Location - Reigate, Surrey, United Kingdom

Post by storiqax » Wed Sep 26, 2018 6:05 am

Interest for this one too. What would you recommend regarding robots.txt for better SEO?

User avatar
New member

Posts

Joined
Tue May 29, 2018 10:19 pm

Post by kg0925 » Wed Sep 26, 2018 12:53 pm

paulfeakins wrote:
Tue Sep 25, 2018 10:18 pm
Why are you disallowing indexing of almost all your site lol?
to avoid duplication. can you suggest me better robots file?? as i said i don't know much about it i found this robots file to many blogs as well it suggested by https://marketplace.webkul.com/optimiza ... rketplace/

please help me with it if you know better robots file.

New member

Posts

Joined
Wed Mar 29, 2017 11:09 pm

Post by uksitebuilder » Wed Sep 26, 2018 2:38 pm

Here's a basic robots.txt file for OpenCart

Code: Select all

User-agent: *
Disallow: /*&limit
Disallow: /*?sort
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search

Sitemap: https://www.yoursitemapurlhere

ImageImageImageImage

For Friendly Professional Support - Click Here


User avatar
Guru Member

Posts

Joined
Thu Jun 09, 2011 11:37 pm
Location - United Kindgom

Post by kg0925 » Wed Sep 26, 2018 6:38 pm

uksitebuilder wrote:
Wed Sep 26, 2018 2:38 pm
Here's a basic robots.txt file for OpenCart

Code: Select all

User-agent: *
Disallow: /*&limit
Disallow: /*?sort
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search

Sitemap: https://www.yoursitemapurlhere
is it good for seo??

New member

Posts

Joined
Wed Mar 29, 2017 11:09 pm

Post by khnaz35 » Wed Sep 26, 2018 8:08 pm

kg0925 wrote:
Wed Sep 26, 2018 6:38 pm
uksitebuilder wrote:
Wed Sep 26, 2018 2:38 pm
Here's a basic robots.txt file for OpenCart

Code: Select all

User-agent: *
Disallow: /*&limit
Disallow: /*?sort
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search

Sitemap: https://www.yoursitemapurlhere
is it good for seo??
Yes it is , just dont forget to mention your site map

Urgent Questions shoot here: khnaz35@gmail.com
Enjoy nature ;) :) :-*


User avatar
Active Member

Posts

Joined
Mon Aug 27, 2018 11:30 pm
Location - Malaysia

Post by newbie87 » Sat Sep 07, 2019 4:53 pm

Regarding the robots/txt file,
/Disallow: /admin
Disallow: /catalog
Disallow: /download
Disallow: /mods
Disallow: /system
Disallow: /system_
Disallow: /vendor
Disallow: /vqmod
why does any search engine need to access these folders in the backend?

Newbie

Posts

Joined
Thu May 28, 2015 5:14 am

Post by uksitebuilder » Sun Sep 08, 2019 3:46 pm

robots.txt should list only links that are accessible via a robot that follows links in your site.

Therefore no admin folder or other OC folders should be listed really. Plus listing them gives script kiddies something to attempt to play with.

robots.txt is not always adhered to (depending on the robot that is crawling) Also, you should be careful not to accidentally block robots from valid pages that you do want indexed.

ImageImageImageImage

For Friendly Professional Support - Click Here


User avatar
Guru Member

Posts

Joined
Thu Jun 09, 2011 11:37 pm
Location - United Kindgom

Post by letxobnav » Sun Sep 08, 2019 4:08 pm

why does any search engine need to access these folders in the backend?
robots.txt is about public url's not your folder structure.

Instead of, or in addition to, robots.txt entries, you can add robots meta tags and X-Robots-Tag headers which are more flexible and dynamic.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by stavrosa » Mon Jun 22, 2020 4:27 pm

This is my setup

Code: Select all

User-agent: *
Disallow: /*&limit
Disallow: /*?sort
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search

User-agent: SemrushBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12Bot
Disallow: /

Sitemap: https://YOUR-DOMAIN/index.php?route=information/sitemap

Newbie

Posts

Joined
Wed Sep 23, 2015 8:08 pm
Who is online

Users browsing this forum: jrr and 17 guests