In recent months, Our server has been consuming too much traffic from Bingbot.
How can I make Bingbot crawl only product and category pages to avoid wasting traffic?
This is the discussion I saw:
https://www.reddit.com/r/bing/comments/ ... tl=zh-hant
How should I modify .htaccess?
Please help me!
Depends on what your product and category urls look like as well as what the "wasted traffic" urls look like.HAO wrote: ↑Tue Jun 10, 2025 12:11 pmIn recent months, Our server has been consuming too much traffic from Bingbot.
How can I make Bingbot crawl only product and category pages to avoid wasting traffic?
This is the discussion I saw:
https://www.reddit.com/r/bing/comments/ ... tl=zh-hant
How should I modify .htaccess?
Please help me!
For complex/flexible exclusions it is often better to use php to send robots headers and/or robots meta tags than using robots.txt, especially when you use seo urls.
All I get is the statistics that Bingbot is wasting over 500G of traffic.
I can't tell what type of links Bingbot is capturing, I would like to start by disabling Bingbot from crawling search links.
Is it possible to modify the search link in this way?
Thank you!
I can't tell what type of links Bingbot is capturing, I would like to start by disabling Bingbot from crawling search links.
Code: Select all
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bingbot [NC]
RewriteCond %{REQUEST_URI} ^/search/?$ [NC]
RewriteRule .* - [F,L]Code: Select all
index.php?route=product/search&search=*You could just use:HAO wrote: ↑Tue Jun 10, 2025 1:09 pmAll I get is the statistics that Bingbot is wasting over 500G of traffic.
I can't tell what type of links Bingbot is capturing, I would like to start by disabling Bingbot from crawling search links.
Is it possible to modify the search link in this way?Code: Select all
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} Bingbot [NC] RewriteCond %{REQUEST_URI} ^/search/?$ [NC] RewriteRule .* - [F,L]Thank you!Code: Select all
index.php?route=product/search&search=*
Code: Select all
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bingbot [NC]
RewriteCond %{REQUEST_URI} search= [NC]
RewriteRule .* - [F,L]
But if you want to give a 403 to all these urls, better also add them to robots.txt as bingbot might still come back for those if you do not explicitely tell it that you do not want them to i.e. in robots.txt.
Code: Select all
User-agent: BingBot
Disallow: search=
No one uses Bing, you could block it entirely?
UK OpenCart Hosting | OpenCart Audits | OpenCart Support - please email info@antropy.co.uk
If you set up a profile with Bing Webmaster Tools they have an option there to set times you want them to crawl and rate limiting.
You can not stop them from crawling the pages they want to crawl, but they do have other options to resolve your issues.
You can not stop them from crawling the pages they want to crawl, but they do have other options to resolve your issues.
DISCLAIMER:
You should not modify core files .. if you would like to donate a cup of coffee I will write it in a modification for you.
https://www.youtube.com/watch?v=zXIxDoCRc84
Who is online
Users browsing this forum: lockiedownunder, Semrush [Bot] and 24 guests
