Post by CaptainHaddock » Tue Sep 15, 2015 7:16 am

Hi,
How do I set up a robots.txt for a multistore?
My main domain had a robots.txt. e.g. www.mysite1.com/robots.txt
I have added a multistore. It is causing problems with server overload as the yahoo slurp robot is trying to crawl too much too quickly. According to Yahoo the crawl rate of Yahoo Slurp bot can be controlled by robots.txt.
How do I set up different robots.txt for my mulltistore e.g. www.mysite2.com/robots.txt

Thanks

Opencart 1.5.5.1
VQMOD
Extensions installed:
Journal Theme
SEO Pack Pro
All Products From Subcategory in Parent Category
All Products From Subcategories in Parent Category Filters
Remove Return Links

Bug fixes applied:
Fix for Tag Filters not working (see OpenCart 1.5.5 Bug Thread)


New member

Posts

Joined
Tue Jul 02, 2013 7:01 am


Post by IP_CAM » Tue Sep 15, 2015 11:43 am


Please don't send me OC Forum Personal Messages, just contact: jti@jacob.ch
---
OC 1.5.6.5 LIGHT Test Site: http://www.bigmax.ch/shop/
OC 1.5.6.5 V-PRO Test Site: http://www.jacob.ch/shop/
My Github OC Site: https://github.com/IP-CAM
2'600+ FREE OC Extensions on the World's largest Github OC Repository Archive Site.


User avatar
Legendary Member

Posts

Joined
Tue Mar 04, 2014 1:37 am
Location - Switzerland
Who is online

Users browsing this forum: No registered users and 43 guests