Post by CaptainHaddock » Tue Sep 15, 2015 7:16 am

Hi,
How do I set up a robots.txt for a multistore?
My main domain had a robots.txt. e.g. www.mysite1.com/robots.txt
I have added a multistore. It is causing problems with server overload as the yahoo slurp robot is trying to crawl too much too quickly. According to Yahoo the crawl rate of Yahoo Slurp bot can be controlled by robots.txt.
How do I set up different robots.txt for my mulltistore e.g. www.mysite2.com/robots.txt

Thanks

Active Member

Posts

Joined
Tue Jul 02, 2013 7:01 am


Post by IP_CAM » Tue Sep 15, 2015 11:43 am


My Github OC Site: https://github.com/IP-CAM
5'200 + FREE OC Extensions, on the World's largest private Github OC Repository Archive Site.


User avatar
Legendary Member

Posts

Joined
Tue Mar 04, 2014 1:37 am
Location - Switzerland

Post by laszkris » Tue Apr 12, 2022 1:08 pm

This solution does not work! Websites are dead or do not contain adequate information!
Does anyone have useful information about robots.txt in multiple stores?
Because in my case I have 3domains and what domain name or URL should be entered in robots txt?
Thanks

Newbie

Posts

Joined
Wed Oct 11, 2017 2:51 pm

Post by ADD Creative » Tue Apr 12, 2022 5:58 pm

laszkris wrote:
Tue Apr 12, 2022 1:08 pm
This solution does not work! Websites are dead or do not contain adequate information!
Does anyone have useful information about robots.txt in multiple stores?
Because in my case I have 3domains and what domain name or URL should be entered in robots txt?
Thanks
You could try a rewrite in your htaccess to a separate file.

Code: Select all

RewriteCond %{HTTP_HOST} ^example.com$
RewriteRule ^robots\.txt$ robots2.txt
And make sure robots2.txt is not blocked by any other rules.

www.add-creative.co.uk


Expert Member

Posts

Joined
Sat Jan 14, 2012 1:02 am
Location - United Kingdom
Who is online

Users browsing this forum: No registered users and 75 guests