how to stop google from indexing this type of URL
?sort=p.model&order=DESC&limit=75
?sort=p.price&order=ASC&limit=50
By adding this to your robots.txt :
Code: Select all
Disallow: /*?page=1
Disallow: /*&page=1
Disallow: /*?keyword=
Disallow: /*&keyword=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?limit=
Disallow: /*&limit=
Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig
Adding to robots.txt will stop Google from crawling, but will not stop indexing.
If you want to remove pages from the indexed results, you can set noindex. Adding something like following to htaccess would set noindex.
If you want to remove pages from the indexed results, you can set noindex. Adding something like following to htaccess would set noindex.
Code: Select all
<If "%{QUERY_STRING} =~ m#(sort|order|limit|tag|search|sub_category|description|filter)=#i">
Header set X-Robots-Tag "noindex, nofollow"
</If>
Adding disallow statements in robots.txt will prevent bots from even requesting those urls (if they respect your request).
Therefore they will also not see any urls on those pages.
Pages with paths like /account and /checkout are key candidates for that.
The others are better covered by x-robots headers and/or robots meta tags as there you can request bots to not index a certain url but still visit the urls which that url contains.
ADD Creative suggested to use:
Header set X-Robots-Tag "noindex, nofollow"
which does exactly the same as disallow in robots.txt
I would suggest:
Header set X-Robots-Tag "noindex, follow"
which means, don't index this one but do have a look at the urls on it.
Who is online
Users browsing this forum: No registered users and 16 guests
