Page 1 of 1
How to properly restrict Bingbot from crawling unnecessary links?
Posted: Tue Jun 10, 2025 12:11 pm
by HAO
In recent months, Our server has been consuming too much traffic from Bingbot.
How can I make Bingbot crawl only product and category pages to avoid wasting traffic?
This is the discussion I saw:
https://www.reddit.com/r/bing/comments/ ... tl=zh-hant
How should I modify .htaccess?
Please help me!
Re: How to properly restrict Bingbot from crawling unnecessary links?
Posted: Tue Jun 10, 2025 12:47 pm
by nonnedelectari
HAO wrote: ↑Tue Jun 10, 2025 12:11 pm
In recent months, Our server has been consuming too much traffic from Bingbot.
How can I make Bingbot crawl only product and category pages to avoid wasting traffic?
This is the discussion I saw:
https://www.reddit.com/r/bing/comments/ ... tl=zh-hant
How should I modify .htaccess?
Please help me!
Depends on what your product and category urls look like as well as what the "wasted traffic" urls look like.
For complex/flexible exclusions it is often better to use php to send robots headers and/or robots meta tags than using robots.txt, especially when you use seo urls.
Re: How to properly restrict Bingbot from crawling unnecessary links?
Posted: Tue Jun 10, 2025 1:09 pm
by HAO
All I get is the statistics that Bingbot is wasting over 500G of traffic.
I can't tell what type of links Bingbot is capturing, I would like to start by disabling Bingbot from crawling search links.
Code: Select all
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bingbot [NC]
RewriteCond %{REQUEST_URI} ^/search/?$ [NC]
RewriteRule .* - [F,L]
Is it possible to modify the search link in this way?
Code: Select all
index.php?route=product/search&search=*
Thank you!
Re: How to properly restrict Bingbot from crawling unnecessary links?
Posted: Tue Jun 10, 2025 1:25 pm
by nonnedelectari
HAO wrote: ↑Tue Jun 10, 2025 1:09 pm
All I get is the statistics that Bingbot is wasting over 500G of traffic.
I can't tell what type of links Bingbot is capturing, I would like to start by disabling Bingbot from crawling search links.
Code: Select all
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bingbot [NC]
RewriteCond %{REQUEST_URI} ^/search/?$ [NC]
RewriteRule .* - [F,L]
Is it possible to modify the search link in this way?
Code: Select all
index.php?route=product/search&search=*
Thank you!
You could just use:
Code: Select all
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bingbot [NC]
RewriteCond %{REQUEST_URI} search= [NC]
RewriteRule .* - [F,L]
as REQUEST_URI contains both path and query string.
But if you want to give a 403 to all these urls, better also add them to robots.txt as bingbot might still come back for those if you do not explicitely tell it that you do not want them to i.e. in robots.txt.
Code: Select all
User-agent: BingBot
Disallow: search=
Re: How to properly restrict Bingbot from crawling unnecessary links?
Posted: Tue Jun 10, 2025 5:33 pm
by HAO
Thanks for your help, I have modified it.
Re: How to properly restrict Bingbot from crawling unnecessary links?
Posted: Tue Jun 10, 2025 6:06 pm
by paulfeakins
HAO wrote: ↑Tue Jun 10, 2025 12:11 pm
In recent months, Our server has been consuming too much traffic from Bingbot.
No one uses Bing, you could block it entirely?
Re: How to properly restrict Bingbot from crawling unnecessary links?
Posted: Wed Jun 11, 2025 2:51 am
by by mona
If you set up a profile with Bing Webmaster Tools they have an option there to set times you want them to crawl and rate limiting.
You can not stop them from crawling the pages they want to crawl, but they do have other options to resolve your issues.