Page 3 of 6

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Mon Mar 14, 2016 10:45 pm
by pm-netti
Qphoria wrote: I've attached the upgrade files.
Just unzip and upload the normal opencart 2.2.0.0 zip package
Then unzip my files and overwrite the 7 files in install/model/upgrade
Then run the upgrade script and it "should" work.
Be sure to follow the instructions after upgrade about user group permissions
Versions 1.4.x and 1.5.0 is not work:

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Mon Mar 14, 2016 11:21 pm
by Qphoria
pm-netti wrote:
Qphoria wrote: I've attached the upgrade files.
Just unzip and upload the normal opencart 2.2.0.0 zip package
Then unzip my files and overwrite the 7 files in install/model/upgrade
Then run the upgrade script and it "should" work.
Be sure to follow the instructions after upgrade about user group permissions
Versions 1.4.x and 1.5.0 is not work:
Yea 1.4.x won't work.

So here's the thing.. the 2.0 and 2.1 upgrade scripts included a more dynamic design that actually parsed through every table and column in the opencart.sql file and made the database match that. That was removed in 2.2.0.0 in favor of a progressive file system using static db calls.

I had this added back for 2.2 initially (and it left it commented out in 1000.php) but it sort of conflicts with the static calls made in the progressive files. Perhaps we need to detect if versions older than 1.5.2 are the source, then use the dynamic system, otherwise use the static ones. So yea.. I will note that the upgrade scripts as defined in 2.2 are only made for coming from 1.5.3.1 or later.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 12:12 am
by labeshops
Great. So going from 1.5.6.4 to 2.2 should not be a problem?

I'm planning to copy everything to a test folder with a copy of my db just in case, disable all extensions, then run the scripts. Assuming this is the proper way?

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 12:21 am
by pm-netti
Qphoria wrote:
pm-netti wrote:
Qphoria wrote: I've attached the upgrade files.
Just unzip and upload the normal opencart 2.2.0.0 zip package
Then unzip my files and overwrite the 7 files in install/model/upgrade
Then run the upgrade script and it "should" work.
Be sure to follow the instructions after upgrade about user group permissions
Versions 1.4.x and 1.5.0 is not work:
Yea 1.4.x won't work.

So here's the thing.. the 2.0 and 2.1 upgrade scripts included a more dynamic design that actually parsed through every table and column in the opencart.sql file and made the database match that. That was removed in 2.2.0.0 in favor of a progressive file system using static db calls.

I had this added back for 2.2 initially (and it left it commented out in 1000.php) but it sort of conflicts with the static calls made in the progressive files. Perhaps we need to detect if versions older than 1.5.2 are the source, then use the dynamic system, otherwise use the static ones. So yea.. I will note that the upgrade scripts as defined in 2.2 are only made for coming from 1.5.3.1 or later.
Old upgrade script can exploit (this method newData, by line 494):
https://github.com/pekka2/OpenCart-Migr ... ucture.php
when pick up that array, can add new columns automaticilly. (ps. that free upgrade script is not in use)

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 2:37 am
by pm-netti
Ps. In file 1001.php lines 186-218 is the associated script, it can exploid and fix necessary.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 4:19 am
by Qphoria
pm-netti wrote:Ps. In file 1001.php lines 186-218 is the associated script, it can exploid and fix necessary.
You mean 1000.php
But there is a problem with that when you hit a table that changes the primary key to a new name.

There are 2 tables
affiliate_activity
customer_activity

that have their primary keys names changed from
activity_id to affiliate_activity_id
and
activity_id to customer_activity_id

When the lines between 186-218 are uncommented, it tries to add the affiliate_activity_id to the table first because it doesn't exist. So now the table has both "affiliate_activity_id" and "activity_id" and currently activity_id is mapped to the primary key and Auto_inc. But now we are inside the column loop of the NEW columns when it removes the auto-inc from all fields in the list. But that old field "activity_id" isn't in the list of fields so it doesn't get called so then you get a crash that says there can only be one auto-inc field and it has to be the primary key.

That is easy enough to fix by doing a secondary loop through the old table fields and removing all auto-inc on any old fields as well.

But now we have a new affiliate_activity_id with a default 0 value for all records. When the code that tries to set the auto-inc on the new column comes along, it crashes because of duplicate key 0 for the primary key.

So one solution is to call the column name change BEFORE running the dynamic script. But that now adds complexity to the overall system and potential for forgetting about that in the future.

But basically for any future primary key name changes, we will have to pre-handle those before calling the dynamic script. And that breaks the design of that script and the attempt at a progressive style upgrade system.

Perhaps we can track the "old" primary key and the "new" one and if it has a different name then call a CHANGE command instead of ADD

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 5:20 am
by Qphoria
Ok I've done just that. I grabbed the extended field info from the old table and match the auto_inc. If the field doesn't exist and is autoinc in the new table, it changes the name from the old table name to the new table name. Now we can get the dynamic script back again. I'll clean it up and zip it. This should at least give us 1.5.0 support.
1.4.x technically will work except there is no code for the product option changes from 1.4.x to 1.5.x so it will lose all options. For 1.4.x you are best using the BETA upgrade script I made a few years ago.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 5:27 am
by pm-netti
Qphoria wrote:Ok I've done just that. I grabbed the extended field info from the old table and match the auto_inc. If the field doesn't exist and is autoinc in the new table, it changes the name from the old table name to the new table name. Now we can get the dynamic script back again. I'll clean it up and zip it. This should at least give us 1.5.0 support.
1.4.x technically will work except there is no code for the product option changes from 1.4.x to 1.5.x so it will lose all options. For 1.4.x you are best using the BETA upgrade script I made a few years ago.
I am my script removed auto_increment test from loop. then Drop eg. column customer_activity_id and rename activity_id to customer_activity_id.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 10:09 am
by Qphoria
Ok This one adds back the dynamic code and more
https://github.com/opencart/opencart/pull/4126

Features:
- Auto update config.php and admin/config.php with
DIR_UPLOAD
DIR_MODIFICATION
DB_PORT
mysql -> mysqli
- Auto convert 1.5.x modules to 2.x format, disabling any non-core modules to prevent errors.
- Auto-Convert Welcome to HTML
- Restored the dynamic update script for individual columns with fix for issue with renaming primary keys
- Added default values for API user with generated hash if not set
- Added default values for processing/complete statuses, if not set, to prevent errors in the admin
- Copied the product name to the new meta_title field, if empty, to prevent errors as it is a required field.

It would work for 1.4.x except for the product option change will not convert over.
Probably best to leave 1.4.x updates to that old BETA script I made a while back as that does have basic option conversion support already:
viewtopic.php?f=19&t=50292

To test:
1. Download OpenCart 2.2.0.0
2. Unzip and upload to your server, overwriting all existing files (yes you should have backed up first)
3. Unzip the files from my attachment to the install/model/upgrade folder, overwrite the default files.
4. Run the upgrade.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 9:07 pm
by sculptex
just tried upgrading my test site from 2.0.3.1 - and I got an error when processing

actually it must be when processing 1003 as the progress bar says 1002 has been applied

(I have nearly 30,000 customers and over 500 products on there)

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

OK

Code: Select all

<br />
<b>Fatal error</b>:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 32 bytes) in <b>/home/****/public_html/system/library/db/mysqli.php</b> on line <b>24</b><br />

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 10:00 pm
by pm-netti
sculptex wrote:just tried upgrading my test site from 2.0.3.1 - and I got an error when processing

actually it must be when processing 1003 as the progress bar says 1002 has been applied

(I have nearly 30,000 customers and over 500 products on there)

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

OK

Code: Select all

<br />
<b>Fatal error</b>:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 32 bytes) in <b>/home/****/public_html/system/library/db/mysqli.php</b> on line <b>24</b><br />
How if change this lines 128-130 in file 1003.php:

Code: Select all

$data = unserialize($result['data']);

				$this->db->query("UPDATE `" . DB_PREFIX . "customer_activity` SET `data` = '" . $this->db->escape(json_encode($data)) . "' WHERE `customer_activity_id` = '" . (int)$result['customer_activity_id'] . "'");
to:

Code: Select all

$this->db->query("UPDATE `" . DB_PREFIX . "customer_activity` SET `data` = '" . $this->db->escape(json_encode(unserialize($result['data']))) . "' WHERE `customer_activity_id` = '" . (int)$result['customer_activity_id'] . "'");

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 11:07 pm
by Qphoria
sculptex wrote:just tried upgrading my test site from 2.0.3.1 - and I got an error when processing

actually it must be when processing 1003 as the progress bar says 1002 has been applied

(I have nearly 30,000 customers and over 500 products on there)

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

OK

Code: Select all

<br />
<b>Fatal error</b>:  Allowed memory size of 67108864 bytes exhausted (tried to allocate 32 bytes) in <b>/home/****/public_html/system/library/db/mysqli.php</b> on line <b>24</b><br />
That is unfortunately a server side issue. and likely happens during the step that processes the orders or the activity because it tries to load it all up in memory. Try editing the php.ini file in the install folder and change 64M to 1024M

We may need to alter that step and load up only a few thousand rows at a time and loop through to cut back on the overall memory.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Tue Mar 15, 2016 11:18 pm
by Qphoria
pm-netti wrote: How if change this lines 128-130 in file 1003.php:

Code: Select all

$data = unserialize($result['data']);

				$this->db->query("UPDATE `" . DB_PREFIX . "customer_activity` SET `data` = '" . $this->db->escape(json_encode($data)) . "' WHERE `customer_activity_id` = '" . (int)$result['customer_activity_id'] . "'");
to:

Code: Select all

$this->db->query("UPDATE `" . DB_PREFIX . "customer_activity` SET `data` = '" . $this->db->escape(json_encode(unserialize($result['data']))) . "' WHERE `customer_activity_id` = '" . (int)$result['customer_activity_id'] . "'");
I guess that could be the same as just changing:
$data = unserialize($result['data']);
to
$result['data'] = unserialize($result['data']);
?
not sure if setting the variable back to itself saves the memory of creating another variable. But you may be right, it might save some memory your way. Still I think the larger problem is the select * trying to grab too much data at once. Just selecting the individual columns that need to be converted would be best. I'll clean it up a bit.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 12:19 am
by sculptex
okay I increased memory allowance by adding

Code: Select all

ini_set('memory_limit', '2048M');
into 1003.php
after

Code: Select all

public function upgrade() {
and upgrade completed successfully.

Now I get

Notice: Error: Could not load template /home/xxx/public_html/catalog/view/theme/common/language! in /home/xxx/public_html/system/modification/system/engine/loader.php on line 45

and similar in admin

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 2:47 am
by Qphoria
The admin shouldn't have that same error as it doesn't use themes. But the error is weird, like it set the default theme to "language". What is the actual admin error?

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 3:38 am
by sculptex
admin error
Notice: Error: Could not load template /home/xxx/public_html/admin/view/template/common/header! in /home/xxx/public_html/system/modification/system/engine/loader.php on line 45

All I did was re-run the upgrade after increasing the memory, would that be the problem? Do I need to restart from scratch?

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 3:58 am
by OSWorX
sculptex wrote:okay I increased memory allowance by adding

Code: Select all

ini_set('memory_limit', '2048M');
How many users are allowed to increase the memory to such an abstruse value??

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 4:12 am
by pm-netti
sculptex wrote:admin error
Notice: Error: Could not load template /home/xxx/public_html/admin/view/template/common/header! in /home/xxx/public_html/system/modification/system/engine/loader.php on line 45

All I did was re-run the upgrade after increasing the memory, would that be the problem? Do I need to restart from scratch?
My estimate: your filesystem is not clean Oc 2.2.
engine/loader.php line 72:

Code: Select all

$output = $template->render($route . '.tpl');
Your missing file extension .tpl from this line?

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 8:50 pm
by sculptex
OSWorX wrote:
sculptex wrote:okay I increased memory allowance by adding

Code: Select all

ini_set('memory_limit', '2048M');
How many users are allowed to increase the memory to such an abstruse value??
LOL - okay if you have your own server I guess!
I used that figure straight from an example I found which I thought was more than adequate. The reason for doing it this way is so many servers do not allow such modifications in php.ini or htaccess anyway and as its only needed for this script it seems the more sensible method.
Of course this figure is an overkill for most and perhaps when I have time I could try various figures until I hit the point it works and maybe these could be coded into the script so <10,000 records may be fine with 256M, up to 30,000 might require 512M etc.

Re: OpenCart v2.2.0.0 Testing - What you need to know

Posted: Wed Mar 16, 2016 8:54 pm
by sculptex
pm-netti wrote:My estimate: your filesystem is not clean Oc 2.2.
engine/loader.php line 72:

Code: Select all

$output = $template->render($route . '.tpl');
Your missing file extension .tpl from this line?
no, its exactly the same.

when I get a chance I'm gonna try the whole process again from a 'clean' 2.0.3.1 with just my customer database imported to isolate it from being anything else as theres a few mods in there.