Post by Besti » Thu Sep 10, 2020 4:53 am

Hello!

Thank you very much!

So far i checked out what is varnish.. :)) I disabled Cachewall in CPanel and that but with redirect to the previous page disappeared! Thank you once again! As a non-pro i dont know a lot... I was worried about Pagespeed, but evidently turning off this caching did not affect it (still cannot figure out how to turn on those passive listeners for google... lol)
Regarding your other points:
1. Checked what is - http/2 - sadly my hosting cannot provide me that currently...
2. Read about what is SameSite cookies.. Lax, None etc. Have not really understood what to set and where.. :choke: You said to set it for Chrome and it was written that chrome sets it automatically to Lax if nothing is set... Pretty confused at the moment..
3. 400 code for images - cannot find where it comes from...
4. 502 youtube - so far same... and i dont get those mistakes either..

Sitemap, which comes with OC Feed was not accepted by Google.. They required xml format... So could only get them from here https://www.xml-sitemaps.com/.. Still experimenting.. ::)

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by Besti » Thu Sep 10, 2020 9:08 am

As i have updated the Seo links, for example, for manufacturer and successfully submitted a both sitemaps for two languages and added a sitemapindex... I have noticed that the sitemap contains such urls like Manufacturer/product... For every product.. which i had to delete manually.. Is there a way to avoid these? Dummy question maybe.. But still..

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Thu Sep 10, 2020 12:34 pm

You are right, the default sitemap contains a lot of useless links.

The useful part is:

Code: Select all

			foreach ($products as $product) {
which adds the products and images.

however, then you have:

Code: Select all

			foreach ($manufacturers as $manufacturer) {
that adds the manufacturer links (useful) but in there again all products are listed now per manufacturer.

in function getCategories it lists all category links (useful) but again all products per category are listed.
Since the product page uses the bare product url as the canonical url, all those additional product links are useless in the sitemap as they will be discarded, search engines only index the canonical there.


so you can get rid of this:

Code: Select all

				$products = $this->model_catalog_product->getProducts(array('filter_manufacturer_id' => $manufacturer['manufacturer_id']));

				foreach ($products as $product) {
					$output .= '<url>';
					$output .= '  <loc>' . $this->url->link('product/product', 'manufacturer_id=' . $manufacturer['manufacturer_id'] . '&product_id=' . $product['product_id']) . '</loc>';
					$output .= '  <changefreq>weekly</changefreq>';
					$output .= '  <priority>1.0</priority>';
					$output .= '</url>';
				}

and this:

Code: Select all

			$products = $this->model_catalog_product->getProducts(array('filter_category_id' => $result['category_id']));

			foreach ($products as $product) {
				$output .= '<url>';
				$output .= '  <loc>' . $this->url->link('product/product', 'path=' . $new_path . '&product_id=' . $product['product_id']) . '</loc>';
				$output .= '  <changefreq>weekly</changefreq>';
				$output .= '  <priority>1.0</priority>';
				$output .= '</url>';
			}

it only bloats your sitemap with useless urls.

unless of course you change the canonical url generated in the product controller to include the category path for instance.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by letxobnav » Thu Sep 10, 2020 12:48 pm

for your varnish caching of the pages giving the wrong page (the previous one) when using the language selector, you can use a different approach we use on our production site:

1) we created a new controller prep.php in catalog/controller/startup/
That controller contains:

Code: Select all

<?php
class ControllerStartupPrep extends Controller {
	public function index() {
		$set_save_get = true;
		$headers = array_change_key_case(getallheaders(),CASE_LOWER);
		// do not save get from prefetch or ajax calls
		if (array_key_exists('purpose',$headers)) if ($headers['purpose'] == 'prefetch') $set_save_get = false;
		if (array_key_exists('x-requested-with',$headers)) if ($headers['x-requested-with'] == 'XMLHttpRequest') $set_save_get = false;
		if (isset($this->request->get['route'])) {
			// do not save get from selectors
			if ($this->request->get['route'] == 'common/language/language') $set_save_get = false;
			if ($this->request->get['route'] == 'common/currency/currency') $set_save_get = false;
		}
		if ($set_save_get) $this->session->data['save_get'] = $this->request->get;
	}
}
2) then in system/config/catalog.php we add that controller to the startup actions:

it was:

Code: Select all

// Actions
$_['action_pre_action']  = array(
	'startup/session',
	'startup/startup',
	'startup/error',
	'startup/event',
	'startup/maintenance',
	'startup/seo_url'
);

it is now:

Code: Select all

// Actions
$_['action_pre_action']  = array(
	'startup/session',
	'startup/startup',
	'startup/seo_url',
	'startup/prep',
	'startup/error',
	'startup/event',
	'startup/maintenance'
);
that will execute prep.php right after seo_url which we also bumped up a little in the order.
then you can take

Code: Select all

$this->session->data['save_get'] = $this->request->get;
out of the index function of the common/language class

you can give it a try, I don't use external caches so it is hard for me to double check.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Thu Sep 10, 2020 2:19 pm

Besti wrote:
Thu Sep 10, 2020 4:53 am
Hello!

Thank you very much!

So far i checked out what is varnish.. :)) I disabled Cachewall in CPanel and that but with redirect to the previous page disappeared! Thank you once again! As a non-pro i dont know a lot... I was worried about Pagespeed, but evidently turning off this caching did not affect it (still cannot figure out how to turn on those passive listeners for google... lol)
Regarding your other points:
1. Checked what is - http/2 - sadly my hosting cannot provide me that currently...
2. Read about what is SameSite cookies.. Lax, None etc. Have not really understood what to set and where.. :choke: You said to set it for Chrome and it was written that chrome sets it automatically to Lax if nothing is set... Pretty confused at the moment..
3. 400 code for images - cannot find where it comes from...
4. 502 youtube - so far same... and i dont get those mistakes either..

Sitemap, which comes with OC Feed was not accepted by Google.. They required xml format... So could only get them from here https://www.xml-sitemaps.com/.. Still experimenting.. ::)
Thank you for your suggestions!
It would be nice to use the sitemap from OC Feed, but i guess you did not notice my previous comment - Google just refused to accept it... But from your post now at least i understand that it should not be a problem if i have this kind of manufacturer, etc.. duplicates on the website as long as the URL is canonical...
About caching i followed your advice and switched it off.. At least part of it and the issue was solved.
The funny thing is that after questioning my hosting provider further about HTTP/2, they told me that the Cachewall and their server caching, which i dont use now, is not compatible, so they cant provide HTTP/2.. :crazy:
Thank you once again for your time! You are very kind!!! :)

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Thu Sep 10, 2020 3:07 pm

I was worried about Pagespeed, but evidently turning off this caching did not affect it
most of those external caches have little benefit and a lot of issues.
If your pages do not perform without it, you have double trouble as many pages will not get a cache hit which means you still get the normal slow performance but since caches do not work via "no cure no pay" you also pay the extra penalty for the cache overhead.
Only external cache if you cannot handle the traffic (number of concurrent users), not to speed up slow single page requests, no use.
sadly my hosting cannot provide me that currently...
They will have to deploy that soon, gives you a lot of performance benefits.
Read about what is SameSite cookies.. Lax, None etc. Have not really understood what to set and where.
Samesite parameter for cookies is typical Chrome for now but essential for many payment gateways for session cookie access.
For session cookies there are only 2 code changes needed but they are different depending on the php version you have.
Google just refused to accept it...
More people have reported that but just submit it, eventually it will stick.
Or just load the sitemap in your browser, view source, copy, paste and save and submit that.
But the feed link should work, mine does and I have nothing special there.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Thu Sep 10, 2020 3:51 pm

Hello!

Thank you for a quick reply! ;D
Regarding SameSite cookies - i have 7.3 PHP currently... :)
With the sitemap i will try later again with the php, but i guess it will be faster to do it manually ... lol

Concerning multilanguage URLs i started having lots of errors now and Product attributes simply disappear from time to time... ((
Looks like it is related to the changes made...
All errors are in product.php

PHP Notice: Undefined index: attribute_id
Undefined index: product_seo_url
PHP Notice: Undefined index:
PHP Warning: Illegal string offset 'attribute_id'
PHP Warning: Invalid argument supplied for foreach() in

Any ideas what can be done with that?

P.S. Ah.. And export/import of the database through admin gives 500 error.. Not sure if it is related, but still...

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Thu Sep 10, 2020 4:47 pm

Those are notices and warnings, not errors, are you using options? I cannot find any on your site.
In any event those are not url related.

Don't use the admin database tools, very bad, better use a 3rd party tool for that.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Thu Sep 10, 2020 5:00 pm

Hello!

No, no options and there are not planned any... The warnings started appearing after the changes to the urls... Really, strange...
I use additional tools, it is just worriying that there are such errors in admin, which were not there...

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Thu Sep 10, 2020 5:25 pm

just show the full errors and what oc version you have.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Thu Sep 10, 2020 6:44 pm

OC 3.0.3.6 default

2020-09-09 22:28:41 - PHP Warning: Illegal string offset 'attribute_id' in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 178
2020-09-09 22:28:41 - PHP Warning: Illegal string offset 'attribute_id' in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 180
2020-09-09 22:28:41 - PHP Warning: Illegal string offset 'product_attribute_description' in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 182
2020-09-09 22:28:41 - PHP Warning: Invalid argument supplied for foreach() in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 182
2020-09-09 22:30:06 - PHP Warning: unlink(/home/wittomin/oc/storage/cache/cache.catalog.language.1599694205): No such file or directory in /home/wittomin/public_html/system/library/cache/file.php on line 68
2020-09-09 22:30:06 - PHP Warning: unlink(/home/wittomin/oc/storage/cache/cache.catalog.language.1599694205): No such file or directory in /home/wittomin/public_html/system/library/cache/file.php on line 68
2020-09-09 22:34:53 - PHP Notice: Undefined index: product_seo_url in /home/wittomin/oc/storage/modification/admin/controller/catalog/product.php on line 1222
2020-09-09 22:34:53 - PHP Notice: Undefined index: points in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 146
2020-09-09 23:02:48 - PHP Warning: unlink(/home/wittomin/oc/storage/cache/cache.catalog.language.1599696168): No such file or directory in /home/wittomin/public_html/system/library/cache/file.php on line 68
2020-09-09 23:20:29 - PHP Notice: Undefined index: attribute_id in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 178
2020-09-09 23:28:55 - PHP Notice: Undefined index: attribute_id in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 178
2020-09-10 0:03:45 - PHP Warning: unlink(/home/wittomin/oc/storage/cache/cache.catalog.language.1599699825): No such file or directory in /home/wittomin/public_html/system/library/cache/file.php on line 68
2020-09-10 0:24:10 - PHP Notice: Undefined index: attribute_id in /home/wittomin/oc/storage/modification/admin/model/catalog/product.php on line 178

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Thu Sep 10, 2020 8:58 pm

Well, looks like they are all related (except the language cache issue) to a modification you have installed on the admin/model/catalog/product.php file which is where all admin product related functions are defined like saving products.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Thu Sep 10, 2020 9:51 pm

Thank you! I will try to look into this and switch those mods maybe.. I had troubles with product quantity verification, with pay pal on product page a person was able to buy as many items as he types even when the stock was zero, so had to fix it.. But it was like a week or two ago and the warnings started yesterday..

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Fri Sep 11, 2020 7:46 am

I believe these warnings come from when you edit a product in admin via function editProduct.
I do not know what the modification does and I cannot rely on the line numbers but that seems to be the case.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Fri Sep 11, 2020 4:50 pm

Good morning!

The warnings were gone themselves... Evidently after a backup something got broken, but the redirects happen from time to time, when i switch languages - cachewall disabled.

About the canonical urls - google for some reason indexed page like Manufacturer/product.. So there are duplicates.. Not sure the language ones shall be considered as duplicates. Should i write anything in the xml files for each language?

The Samesite cookies i have found in forum only for OC2 (my OC 3.0.3.6, php 7.3) - https://forum.opencart.com/viewtopic.p ... 20#p796020. On github they wanted to update OC for this, but i suppose the Samesite was not still implemented for 3.0.3.6..
Used tips for currency, language.. But could not find setcookie in session.php,so tried adding line to htaccess.. Now, no idea if it works and is it enough for PayPal.. Also for those cookies for GDPR extension are bothering me..
I already took a lot of your time, but maybe you could advice something on this?

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Fri Sep 11, 2020 5:25 pm

Should i write anything in the xml files for each language?
no, the sitemap just lists the urls, it is the page content which makes for duplicates or not, as long as you make sure your page content is also in different language, i.e. the meta data and content is different for the same product. With non-seo urls you only have 1 canonical url per product, with multi-lingual seo urls you have multiple, just make sure those pages are written in those languages.

for the session cookie:
catalog/controller/startup/session.php
replace:

Code: Select all

setcookie($this->config->get('session_name'), $this->session->getId(), ini_get('session.cookie_lifetime'), ini_get('session.cookie_path'), ini_get('session.cookie_domain'));
with:

Code: Select all

setcookie($this->config->get('session_name'), $this->session->getId(), ['expires' => ini_get('session.cookie_lifetime'), 'path' => ini_get('session.cookie_path'), 'domain' => ini_get('session.cookie_domain'), 'samesite' => 'None', 'secure' => true, 'httponly' => true]);
system/framework.php
replace:

Code: Select all

setcookie($config->get('session_name'), $session->getId(), ini_get('session.cookie_lifetime'), ini_get('session.cookie_path'), ini_get('session.cookie_domain'));
with:

Code: Select all

setcookie($config->get('session_name'), $session->getId(), ['expires' => ini_get('session.cookie_lifetime'), 'path' => ini_get('session.cookie_path'), 'domain' => ini_get('session.cookie_domain'), 'samesite' => 'None', 'secure' => true, 'httponly' => true]);

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Fri Sep 11, 2020 5:30 pm

Thank you for a quick reply!

Just tested with Chrome flags on, looked like it all works (PayPal, etc.) I will check furthe with your suggestions.

About the second language there is a small but irritating thing... if you type yourdomain.com/en/ - works perfectly, but if you type yourdomain.com/en (without slash at the end) - you get page not found... Should i add smth for this to htaccess?

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Fri Sep 11, 2020 5:40 pm

you could add to htaccess after RewriteBase /

Code: Select all

RewriteRule ^en$ en/ [R=301,L]

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan

Post by Besti » Fri Sep 11, 2020 6:00 pm

I followed all of your advices and recommendation and i am happy as hell!
It all works perfectly!!!!
Just came to thank you once again for your time! Wish you all the best!!! :-* ;D

New member

Posts

Joined
Sun May 24, 2020 4:27 pm

Post by letxobnav » Fri Sep 11, 2020 6:14 pm

while you are in htaccess better change this:

Code: Select all

RewriteCond %{REQUEST_URI} !.*\.(ico|gif|jpg|jpeg|png|js|css)
to this:

Code: Select all

RewriteCond %{REQUEST_URI} !.*\.(env|ashx|cfg|dat|ico|cur|txt|mp3|webp|svg|ttf|eot|woff|woff2|gif|jpg|JPG|jpeg|JPEG|png|js|cfg|css|pdf|zip|env|tar|sql|gz|tar|exe|rar|arj|cab|iso|rpm|tbz|tgz|old|bak|backup|dump|db|7z|asp|aspx|exp|html|htm)$
you see, what this:

Code: Select all

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !.*\.(ico|gif|jpg|jpeg|png|js|css)
RewriteRule ^([^?]*) index.php?_route_=$1 [L,QSA]
does is when a requested resource does not exist in your file-system (i.e. seo urls), the request is passed as a get parameter to index.php.
But only if it is not a request for a static asset like ico, gif, etc.
This in the assumption that it must be an seo url so it is handled by the seo_url class which queries the database to find get variables for the keywords in that seo url.

but, there are a lot of bots out there which will request many more asset types from your site to see if they can obtain vital data.
requests for database.bak, xxx.zip, yy.dat, etc.
Those files, I assume and hope, you also do not have in your file-system open to the internet.
This means that each of those requests will also trigger a database query, and a subsequent creation of a nice 404 not found page, for nothing.
And make no mistake, those requests will come at 100+ per second.

Our production site uses this:

Code: Select all

RewriteCond %{REQUEST_URI} !.*\..*$
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^?]*) /co/index.php?_route_=$1 [L,QSA]
which means requests for assets not found in our file-system will be passed to index.php but only if the requested url does not contain a dot.
Meaning all xxxx.xxx requests, covering all.
Of course, this means we cannot use a dot in our seo urls.
but all those requests for x.zip and yyy.bak will be blocked with a 404 at web server level, not even invoking php.

Crystal Light Centrum Taiwan
Extensions: MailQueue | SUKHR | VBoces

“Data security is paramount at [...], and we are committed to protecting the privacy of anyone who is associated with our [...]. We’ve made a lot of improvements and will continue to make them.”
When you know your life savings are gone.


User avatar
Expert Member

Posts

Joined
Fri Aug 18, 2017 4:35 pm
Location - Taiwan
Who is online

Users browsing this forum: No registered users and 2 guests