(ask) file robots.txt and sitemap file

robots.txt Regular sitemap is automatically generated. Look at the footer of the page. Google Sitemap is enabled in Admin at Extensions->Product Feeds.

You can add a sitemap link to the bottom of the file like so:

The following should work.

There is no need to put in an 'Allow' statement

uksitebuilder wrote:There is no need to put in an 'Allow' statement

For what it's worth that's how Google auto-generates them in webmaster tools.

All files by default are allowed access by robots, so there is no reason to to give them permission per se

Could you share your working version for all who come across this

newbie007 wrote:Absolutely solved!!!

Analysis of ROBOTS.TXT file
No errors have been found.
1 Sitemap(s) found
Value: 100 %

Analysis Sitemap Files
Analysis Result: 1 Sitemap(s) found
Value: 100 %

Analysis Sitemap Validation
Analysis Result: No errors in sitemap(s) found.
Value: 100 %

Analysis Meta tag "robots"
Value: 100 %

Thank you rph
Thank you uksitebuilder
Thank you steve

You all GREAT!!

Hi,

How do you check this?
Which tool do you use?

Please advise how to use that robot.txt file?

Thanks

Is there a way to use sitemaps for multistores? Should I put each domain in as a sitemap link at the bottom?

edit .htaccess file

at the top put

AddType x-mapp-php .txt
AddHandler x-mapp-php .txt

Now you can put php in a text file and will be parsed by the PHP parser

edit your sitemap line in your robots.txt

something like

Sitemap: http://<?php echo $_SERVER['HTTP_HOST']; ?>/index.php?route=feed/google_sitemap

Guys, will Google Webmaster automatically find my sitemap once its enabled??

I have added my website to my webmaster tools and enabled my sitemap but when I paste in the sitemap url its invalid.

To be safe, you should submit it to your webmaster tools account

uksitebuilder wrote:To be safe, you should submit it to your webmaster tools account

I tried this, but Google said that its not a valid feed.

http://www.my-website.co.uk/index.php?r ... le_sitemap

what is the full link to your sitemap with correct domain ?

uksitebuilder wrote:what is the full link to your sitemap with correct domain ?

yes that's the full link the the sitemap. I slightly changed my url as to not show everyone my website

then i have no way of knowing what errors you are getting in your sitemap

im unable to send you a PM

im still not having any luck with this using v1.5.1.3

What about sitemaps for multistores? Could you put in multiple sitemap statements in robots.txt and name them storesitemap.xml and have it work?

I have uksb's mod for multistore sitemaps and have set the paths to them in google webmaster tools but when I scan my site with seo tools, they do not find the sitemaps. I know I can save them with the tool linked above, just not sure how to tell seo tools they are there for each site.

uksitebuilder wrote:The following should work.

There is no need to put in an 'Allow' statement

Code: Select all

user-agent: *
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=checkout/
Disallow: /*?route=account/
Disallow: /*?route=product/search
Disallow: /*?route=affiliate/

Sitemap: http://sentraherbal.net/index.php?route=feed/google_sitemap

Is any option to disallow other languages except my default?