[WIP-MODULE] SVN Upgrade via Admin

So for the past few days I've been working on finding the best way to opencart updates. I've created a pretty creative automated system that lets you enter your current svn revison and your target revision, and it generates a zip file of only the changed files on-the-fly between the two versions. This means you can pass in your current revision and the revision you want to be at, and it will generate a zip file of just those changes. If someone else has already requested that same difference, it will just reuse it instead of generating it again.

This is where I got creative of course, because it can anywhere from a few secs to several minutes to create a zip between revisions. The setup is, the opencart module is the "client" that makes a call to my linux server. On the linux server I created a RESTful php server script to handle the request and fire off a bash script that runs the svn commands on the server. While it runs, the REST server returns a message that the file is being created and to try again in a few minutes when it is ready. If you try to submit the request again, I have a lock file in place that tells it to keep waiting. It's pretty damn sweet (modest much? ).

A lot of fun stuff went into the bash script (my first time) and it works pretty well. Once the file is ready, you have the option of choosing the get a list of all the files that changed (which generates the list from the zip file on the server). Or you can do the real update.

I've added lots of special codes and tweaks to make it very seamless. I remove all config.php files, I check if the upgrade.sql file has changed and include all the installer files. It even redirects to the upgrade step after extracting the new files on your store if the upgrade files exist as part of that revision update.

Here is a Video Demo of me upgrading my 1.5.1.3 store which is revision 647 to the latest 1.5.2 branch version which is v1.5.2.2 (latest SVN)
http://screencast.com/t/vNb8Nth3

Now as cool as this is... It is probably overkill. Whether or not going to this extreme of building on the fly is worthwhile, as part of a separate manual update method, I still plan on making nightly patches for the latest release version that should be considered stable. And maybe just link with that nightly list from the admin and grab the zip of all files from the release to the latest revision... even if you are already at the second to last revision. It's not as efficient, but simpler.

Still this was a fun project to make and may still see the light of day.

An insane mod!!!
Cheer Q!!!

It sound very cool.. can't wait to try this soon
I like the "list of changed file only" feature.

Wow, impressive work! Why are you going to make nightly patches? Just so people don't have to wait for zips to be generated?

Can't wait to try this out.

Johnathan wrote:Wow, impressive work! Why are you going to make nightly patches? Just so people don't have to wait for zips to be generated?

Can't wait to try this out.

I dunno... maybe for those who want the simpler method.. in my process of playing with bash for the first time, I found it quite easy to make a simple cron job that can build release to current head every night easily.. and it seems to be the thing that people do (like mozilla, Wordpress, etc)

Just as an alternative option, unrelated to the progress of this mod.

Another aspect is baselining. If we know the release version of 1.5.2 was r885 and the latest head is 940 and people started having problems after installing the 940 patch.. when we know what we are working with. With a system like this you could install 885 to 910.. then choose to skip to 915 to Head... skipping 911-914... then what a headache that could be.

Daniel has expressed interest in this but also fears the support nightmares this can potentially cause..
But would still like to add something like this to the core.

So I'm thinking there are 2 routes:

Simple - Create "stable nightlies" based off the release version to the current HEAD of the release Branch.

Pros:

• Only one nightly version that gets generated automatically by cron.
• Presents a common "checkpoint" of where your store is vs where it was.
• Can easily be uploaded manually if the admin update system hits some sort of error (permissions, timeout, etc).
• No real-time interfacing with servers and shell scripts.
• This one is a pro and con... it always gets "all" files from release to latest. So if you did the update from 800 to 900, then a new 901 version came out, it would get all files from 800 to 901. The pro part of this is that it keeps all files in sync. You always know you have the correct files when you update.

Cons:

• The con side of the last pro item, is that getting all the files each time is somewhat inefficient and updates will just get larger and larger. But the pro side outweighs the con side.

Advanced - Using this mod to create "on-demand" patches from custom start and end revisions

Pros:

• Individual customized builds giving you only what you need.

Cons:

• No common "checkpoint" of where your store is vs where it was. It has potential to get out of sync and you have no idea where you are at.
• No manual fallback (unless we set the server directory index to be viewable which is possible but can be confusing to look at with all the different versions)
• Requires that all the pieces be in place at all times.. the server and shell scripts must always be listening and processing on-demand.

So you see, this current version might be more than is necessary. In my opinion, the Pros for the Simple method outweigh all other pros and cons. And will likely be the one that gets put into the core. It wouldn't be much different from what you see now.. just without the ability to pass in the from/to for on-demand builds and just get the latest pre-built nightly that is available on the server.

Actually from the demo, you can't even see the fun that goes on behind the scenes. So really you won't see any real difference from the admin side than what you see here.

I vote for the Simple version.

i2Paq wrote:I vote for the Simple version.

+1

If system keeps updating our files, users might get irritated. Once a week or every fortnight update isn't that bad.

Looking from the pros and cons, most people will suggest the simple method, including me.

If I can suggest, I think it's better to check some file that modificated by user and show notice before update. Suggest them to update manually instead of using the auto update if some files is modificated manually. If necessary, hide the auto update button if you feel it will break the site.

I think simple sounds best and safest course, as long as there is (as per quahar) user control.
Not all user mods can be handled fully by qvmod (fantastic though it is). Some core files and .css/.js occassionally need manually changing.

MarketInSG wrote:

i2Paq wrote:I vote for the Simple version.

+1

If system keeps updating our files, users might get irritated. Once a week or every fortnight update isn't that bad.

The "system" doesn't do anything in either version. You still control when and if you want to update in both versions. Nightly builds will be made available, but you don't have to upgrade until you want to.

chulann wrote:I think simple sounds best and safest course, as long as there is (as per quahar) user control.
Not all user mods can be handled fully by qvmod (fantastic though it is). Some core files and .css/.js occassionally need manually changing.

I've thought about this too. But the only "good" way to check if a file was changed would be an md5 hash on each file and compare it to an md5 of the release version to see if it has changed.. and each nightly patch would need to regenerate the md5 file hashes for the updated files. Then store that md5.lst file on your site and run md5 generation during the update and compare it to the md5 list of files. That is no small process and md5 generation is also quite an intensive process on the server. The more files, the greater the process.

Remember this system is fundamentally no different than you upgrading your site the manual way, and in the manual way you also have no way to check if a file was changed either... so there is no difference. I'm not saying its impossible to do, but it is not in the current plan.

There is nothing I can think of that you cannot do with vQmod.. even if you need to change js or css, you can always create a separate script and use vQmod to include it in the tpl file. Or at the very least, keep a simple text file list of your core changes and a backup copy of them in a subfolder.

People should start treating the core file as encrypted binary files and forget that they are human readable. Once we treat them like binaries, we can stop worrying about upgrading them.

When can we see this feat of awesomeness? This is soooo needed. Since you are the developer, you decide what you want and how you want it. Opinions are like Sphincters, everybody has one...

Cheer!

woww very cool

Im just finishing up the validation checks (write permissions, ZipArchive php extension check) and trying to test all the recovery scenarios (what happens if it times out half way through? what happens if it unzips half the files then finds one that is write protected, etc. Tomorrow my plate is mostly clean so I should be able to finish it up

Good job You're awesome. Would love to test it out on my server

Qphoria wrote:Im just finishing up the validation checks (write permissions, ZipArchive php extension check) and trying to test all the recovery scenarios (what happens if it times out half way through? what happens if it unzips half the files then finds one that is write protected, etc. Tomorrow my plate is mostly clean so I should be able to finish it up

Great news! Excited

+1, like

Sounds good, on a side note would also be good if there was a method to check when a 3rd party extension/module has been updated and shown in admin, a bit like wordpress. Would probably involve having the developers them selves setting a check for this. Anyway can't wait for this module

Hi Q,

can be an option for custom admin directory.