Post by whitedotsports » Fri Aug 02, 2024 1:33 am

My server usage is exceeded because of Google bot crawling every second. I am running a 7 multistore with 3 main websites and 4 for later use.
Is there any way I can limit crawling to once a day? or I can block the rest of 4 stores from google crawling. Please guide.


Posts

Joined
Mon Jan 21, 2019 10:43 pm

Post by paulfeakins » Fri Aug 02, 2024 7:51 pm

We've found this occur on multiple client sites. It has become very bad recently. We found that, as long as it's a well-behaved crawler, adding `crawl-delay: 10` to your robots.txt file limits the number of requests these crawlers can make to once every 10 seconds.

UK OpenCart Hosting | OpenCart Audits | OpenCart Support - please email info@antropy.co.uk


User avatar
Legendary Member
Online

Posts

Joined
Mon Aug 22, 2011 11:01 pm
Location - London Gatwick, United Kingdom

Post by JNeuhoff » Fri Aug 02, 2024 8:29 pm

Also, it's a good idea to exclude many of the dodgy bots and crawlers, such as semrush and others, via the '.htaccess' file. A good starting point is this list.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member
Online

Posts

Joined
Wed Dec 05, 2007 3:38 am

Who is online

Users browsing this forum: imager and 2 guests