Post by Qphoria » Fri Dec 03, 2010 10:34 pm

I am putting this here just to keep a running history of discussions and articles I've come across about code extensibility and what works and what doesn't. Feel free to add your own opinions and findings as well. Perhaps with all the collaboration of ideas mixed together we can become alchemists of the code to find a working solution. I will keep adding to this list as I find more stuff. I've also got some new ideas that I haven't seen discussed yet.

Patch/Core Modification Scripts
These are topics about using xml or mod files to patch the actual core files
http://www.jamierf.co.uk/2010/02/08/ext ... s-patches/ - A great pros/cons breakdown
http://docs.simplemachines.org/index.php?topic=402.0
http://www.phpbb.com/mods/automod/
http://easymod.sourceforge.net/readme/# ... _how_works
http://area51.phpbb.com/phpBB/viewtopic ... 334#p96899


Hooks/Plugins/Observer:
These are topics about using a hook or plugin system that trigger other registered events when core events occur
http://codeigniter.com/forums/viewthread/67697/
http://edwardfelch.com/blog/2010/06/bui ... ks-system/
http://theturninggate.net/galleries/ext ... b-engines/
http://stackoverflow.com/questions/3356 ... cms-system
http://61924.nl/05-13-2010/extending-php-software.html
http://phpmef.codeplex.com/


DB Based:
http://www.weberdev.com/ViewArticle/Bui ... Menu-Class

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by Qphoria » Sat Dec 04, 2010 12:40 am

My latest idea that I'm working on I've dubbed "On-the-Fly" modifications or maybe "VirtualMod". 8)

Basically the concept is simple. Controllers, modules, models, templates, and languages all get included/required at some point in one of the system/engine files. So we can:
1. Intercept the file
2. Get the contents of the file before php evaluates it in the require/include function
3. Change the contents with str_replace, regex, line insert, etc to make it look how it should look if it was modified
4. Save that file to a tmp file
5. Then pass that newly modified tmp file into the include step.

Now the file loads with the modifications but the actual core file has not be touched.

This can be setup so that a completely separate directory structure (e.g. /otf_mods/) can be made at the root level and inside this directory we would create the same directory structure of the opencart inside the /otf_mods/ folder except in place of the files would be directories that have the file name as the directory name.

Example:
if you had code to modify the product.php file, you would put your file into:
/otf_mods/catalog/controller/product/product.php/my_product_mod.inc

Note that the /product.php/ is a directory name to allow multiple mods to be applied to the same file.
Now if another person had a mod for that same file you could just add that mod as well to the product.php directory.
The code would loop through all the files in the directory that matched the current filename that was being loaded and apply the modifications on-the-fly.

There would need a syntax defined to tell the code where it should go. I'm thinking something like the SMF xml style mod scripts idea I offered here. There also needs to be some method of sorting for which mod goes first. Also some checks in place to ensure that the attempted changes can be achieved. But that would all be handled by the on-the-fly extension function.
Pros:
- Safely run modifications without fear of permanent changes
- Easily test new changes
- No worries about file permissions
- Multiple mods can be made on the same file, something we've struggled over in the past
- Simply remove the modification file to remove a mod, could be linked and scripted to a package manager to be automatic.
- Easily retrofitted. A simple function call from each of the engine files that loads particular pieces.

Cons:
- The more files that are getting modded on the fly, the more it could affect system performance for the time it takes to read/alter/write the tmpfile. Tho I can't see it taking more than a few milliseconds off the clock, even with larger mods.


I have this already working on my dev site and it seems to work quite well. I've added a page load time to my site so I can see how times vary between using on-the-fly modding and not. So far I've not seen any delay. I will probably make this into a mod as it seems quite easy to do, pending any unforeseen issues.

Alternatives to the oft_mods file structure:
A. Another option for maintaining the modifications could be to add them into the database. A simple sql file could be loaded with the mod and what file it applies to it would show up in a manager in the admin.
B. Another option for maintaining them would be a single file with markers could be made for a mod that alters multiple files into a single file and the the code could parse into memory and apply where the mods need to go. Then each file would just be a text file in the /otf_mods/ directory like:
/otf_mods/change_latest_products_query_to_use_rand.mod
/otf_mods/change_product_to_show_sku.mod
etc

So there are options.

Thoughts?

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by Johnathan » Sat Dec 04, 2010 7:13 am

Qphoria wrote:My latest idea that I'm working on I've dubbed "On-the-Fly" modifications or maybe "FlyMod for OpenCart". 8)

Basically the concept is simple. Controllers, modules, models, templates, and languages all get included/required at some point in one of the system/engine files. So we can:
1. Intercept the file
2. Get the contents of the file before php evaluates it in the require/include function
3. Change the contents with str_replace, regex, line insert, etc to make it look how it should look if it was modified
4. Save that file to a tmp file
5. Then pass that newly modified tmp file into the include step.

Now the file loads with the modifications but the actual core file has not be touched.

Thoughts?
This sounds like an awesome idea to me. It's seems relatively easy to implement, and would be immensely helpful to people trying to modify the core to gain extended functionality. I'm in favor of it. :)

Image Image Image Image Image


User avatar
Administrator

Posts

Joined
Fri Dec 18, 2009 3:08 am


Post by Xsecrets » Sat Dec 04, 2010 7:48 am

Well sounds ok I guess, but it would have to come with solid documentation, because something like this will not be nearly as easy to pick up as just jumping into the code.

OpenCart commercial mods and development http://spotonsolutions.net
Layered Navigation
Shipment Tracking
Vehicle Year/Make/Model Filter


Guru Member

Posts

Joined
Sun Oct 25, 2009 3:51 am
Location - FL US

Post by opencartnovice » Sat Dec 04, 2010 7:50 am

Many thanks

New member

Posts

Joined
Thu Oct 07, 2010 5:12 am

Post by Qphoria » Sat Dec 04, 2010 9:45 am

I'm using the SMF method of xml modification which is pretty well defined here:
http://docs.simplemachines.org/index.php?topic=402.0
and here:
http://www.simplemachines.org/community ... c=299670.0


Basically the system is the same as my previous Phase II plan, just instead of doing actual edits, it is only virtual. That's a good name too "VirtualMod".

I've worked on it most of the day. I've got it all in its own class and a simply 1 line call from the other engine files that run includes or requires. Just have to get the main search and replace system finished up and we'll be "in like flynn"!

I've also already got the phase 1 thing working as well. Just need to add db entries for the installed mods to make it more robust, but I like this virtual mod plan more than the actual edit one so I'm pushing forward on this.

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by JAY6390 » Sat Dec 04, 2010 10:32 am

I don't really understand why you would need the temp file tbh

Image


User avatar
Guru Member

Posts

Joined
Wed May 26, 2010 11:47 pm
Location - United Kingdom

Post by Qphoria » Sat Dec 04, 2010 12:16 pm

JAY6390 wrote:I don't really understand why you would need the temp file tbh
requires and includes have filename arguments

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by imaginetech » Sat Dec 04, 2010 1:56 pm

What are the advantages over what Daniel has done with tpl files falling back to the default path if they are not present?

Couldn't this be replicated with core files

Image
www.opencartstore.com
Get OpenCart Templates and OpenCart Modules.


User avatar
Active Member

Posts

Joined
Fri Sep 04, 2009 12:25 pm
Location - Darwin, Australia

Post by Qphoria » Sat Dec 04, 2010 2:09 pm

imaginetech wrote:What are the advantages over what Daniel has done with tpl files falling back to the default path if they are not present?

Couldn't this be replicated with core files
The fallback was actually my design from ChromiumCart. I did indeed have fallback for not only templates but also controller files. Even library files. In the last version I was actually able to keep all mods in a separate directory tree so that mods would actually go into the "mod" subfolder and the core directory would stay untouched. Though my method did it at the controller class "fetch" level so that it only took 3 lines of code in 1 place to do it, and no other files needed to change. I was not (and still not) a huge fan of this per-controller method as it is just redundant and tedious. Daniel said he didn't want the fallback in the framework layer and that it should be at the app layer. But not really sure how it is considered the "App" layer as all apps on this framework would benefit from this at the framework level... but I digress

Unfortunately, the way the naming convention works in 1.x is a bit too forced. I've tried a few times to implement the same method but could only get it partially working. Additionally, the problem with that method is that you are still restricted to 1 mod on a file since that overrides the entire file.

The pursuit of the SMF-style modification method was popular because it was the only way to allow full core changing at any point. I use a similar method in my options plus mod to modify the files. But then it becomes a pain as you also need to create a way to undo all those changes. Even a hook method, which is still a good idea and should be added, is still limited to where the hooks are placed.

I'm focused now on this idea because its the same concept as a full range modify but with a virtual modify instead of a real modify. And still gives us full control to modify the files at any point in the system.

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by SapporoGuy » Sat Dec 04, 2010 7:22 pm

Now you're talking baby :laugh:

I actually don't like the fallback method since I've modified my template so much that even if I go back to the original template it spews all over the place. Too lazy to research all the reasons but 1 major issue is changing main image size for products and changing the logo. This creates a lot of havoc with the current default template.

So, essentially how does the OnFly work? Does it allow you over-ride the core but yet still allow you to keep the core?

@ OnFly = On the Fly mod
virtual mod sounds too second-life ;)

930sc ... because it is fun!


User avatar
Active Member

Posts

Joined
Mon Nov 01, 2010 7:29 pm

Post by JAY6390 » Sat Dec 04, 2010 7:35 pm

Qphoria wrote:
JAY6390 wrote:I don't really understand why you would need the temp file tbh
requires and includes have filename arguments
lol I'm well aware of how a require/include works! What I mean is your process seems a bit weird imo, you're going to load the file, modify, save, load again then execute the code inside it. Why not just load the file, modify, execute? Plus you would need to ensure each temp file would be unique

Image


User avatar
Guru Member

Posts

Joined
Wed May 26, 2010 11:47 pm
Location - United Kingdom

Post by Xsecrets » Sat Dec 04, 2010 10:36 pm

I guess my big question is still will Daniel allow it in the core, otherwise we now have two mod systems which was my big gripe in readymans thread.

OpenCart commercial mods and development http://spotonsolutions.net
Layered Navigation
Shipment Tracking
Vehicle Year/Make/Model Filter


Guru Member

Posts

Joined
Sun Oct 25, 2009 3:51 am
Location - FL US

Post by Qphoria » Sun Dec 05, 2010 4:15 am

JAY6390 wrote:
Qphoria wrote:
JAY6390 wrote:I don't really understand why you would need the temp file tbh
requires and includes have filename arguments
lol I'm well aware of how a require/include works! What I mean is your process seems a bit weird imo, you're going to load the file, modify, save, load again then execute the code inside it. Why not just load the file, modify, execute? Plus you would need to ensure each temp file would be unique
Well i suppose it is due to me trying to retrofit it into the engine without making too many waves as it does require editing of the engine files.

Take the system/engine/controller.php file

There is this code:

Code: Select all

$file = DIR_TEMPLATE . $filename;

if (file_exists($file)) {
	extract($this->data);

	ob_start();

	require($file);

	$content = ob_get_contents();

	ob_end_clean();

	return $content;
} else {
	exit('Error: Could not load template ' . $file . '!');
}

I simply add:

Code: Select all

$file = $this->virtualmod->applyMod($file);
after:

Code: Select all

$file = DIR_TEMPLATE . $filename;
That's it. There are about 5 places where includes or requires are called and I simply insert that line before each one, leaving the rest of the code the same. And since I am using a tmp file I do NOT need to make unique files... once the file is executed in the ob_get_contents(), the tmpfile can be destroyed as it is no longer needed. I could even reuse that same file. Hence the whole point of "on-the-fly".
Xsecrets wrote:I guess my big question is still will Daniel allow it in the core, otherwise we now have two mod systems which was my big gripe in readymans thread.
Whether in the core or not doesn't really matter, this is easily added as a mod as only 3 files have to change and it only 1 line to add in those places. It doesn't change the way other modification designs work, but since it is in the engine, it actually has the final say, which is in reality really cool because even if you already have a some existing custom changes, those will still be fine. This is at a higher layer when it does the modification. You can still use the template fallback and still make extensions the same way. This is really only for manipulating the core by replacing snippets of code with different ones, right before the system evaluates the code. It's "the old switchamaroo" so to speak, and it occurs at the very last second when the code has not be executed yet.

For example, my newsletter module is all standalone as a module. But I have to insert 5 lines of code in the admin/controller/sale/contact.php file to merge my newsletter table with the existing customer table. This would be the perfect way to do that without having to edit any core files. Simply include a "vmod" file with any name I want like "vmod_newsletter_contact.xml" (or something) and then whenever the contact page was called, it would execute this file

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by JAY6390 » Sun Dec 05, 2010 4:27 am

hmm, still think its a bad idea with the temp files personally. I understand why you've said it, but i don't think the right way to go is to patch bits up as you go, we need to have a solid structure in place in the core

Image


User avatar
Guru Member

Posts

Joined
Wed May 26, 2010 11:47 pm
Location - United Kingdom

Post by Qphoria » Sun Dec 05, 2010 5:08 am

There is this or nothing at this point. Even if this is a stepping stone, its better than nothing, especially since its virtual.

But as I've listed a bunch of links in the first thread, they all have pros and cons.
- Hooks are limited to where they are placed
- Actual Modifications can build up overtime and get messy and cause page errors with no clear resolution
- Fallbacks are on a "per-file" basis
- ORM/Kohana is also on a per-file basis or at the very least a per-function.
- Subclasses are a "per-file" basis

This can be done "right-now" as a mod as daniel has not expressed interest in working on any other options. It can modify any file at any point with no permanent repercussions.. It not only works with opencart but the concept can be made to work with any php project as includes and requires are pretty much a major need where more than 1 file is used... there is no other mod/plugin/hook method that i've found that is more extensible yet as flexible and safe as this idea... It can even be set with a simple if/else to not use a temp file and just rewrite the real file so it could be used for actual core updates and use real modifications. It is not limited to format as it comes down to parsing for the needed search/replace/add data... Move over sliced bread!

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am

Post by imaginetech » Sun Dec 05, 2010 7:56 am

Can see your points Q, earlier you said you did some benchmarking or some performance testing. Was there anything noticeable? Any idea at what point it might cause performance issues if you applied x amount of mods and received x amount of server requests?

Image
www.opencartstore.com
Get OpenCart Templates and OpenCart Modules.


User avatar
Active Member

Posts

Joined
Fri Sep 04, 2009 12:25 pm
Location - Darwin, Australia

Post by Johnathan » Sun Dec 05, 2010 8:15 am

Qphoria wrote:There is this or nothing at this point. Even if this is a stepping stone, its better than nothing, especially since its virtual.
I agree. Even if it's not ideal (although personally I agree with Qphoria), it would be much better than how modifications work now.

Since it's only three files needing changes, any chance this will make it into 1.5.0? ;D

Image Image Image Image Image


User avatar
Administrator

Posts

Joined
Fri Dec 18, 2009 3:08 am


Post by SapporoGuy » Sun Dec 05, 2010 4:25 pm

Since it sounds like that it's not 1.5 or 1.4.9 dpendent, when are you planning on releasing this?

I do have 1 major worry and that is will developers go crazy and start polluting a relatively clean cart?
I say this since there are several mods that actually abuse the system rather than conform and embrace the opencart way of doing things.

930sc ... because it is fun!


User avatar
Active Member

Posts

Joined
Mon Nov 01, 2010 7:29 pm

Post by Qphoria » Mon Dec 06, 2010 12:43 am

imaginetech wrote:Can see your points Q, earlier you said you did some benchmarking or some performance testing. Was there anything noticeable? Any idea at what point it might cause performance issues if you applied x amount of mods and received x amount of server requests?
It should all be on the same server request. But I'm still working on a more complex search/replace functionality.Then I will try making multiple multiline mods. At the moment I have 5 simple one line mods and the page load time doesn't show any variance.
Johnathan wrote: I agree. Even if it's not ideal (although personally I agree with Qphoria), it would be much better than how modifications work now.

Since it's only three files needing changes, any chance this will make it into 1.5.0? ;D
It's too new of a concept at this point. As it is only 3 lines it will be easy to add as a mod and then other modders can choose whether or not to use it. Perhaps after some feedback and performance testing is completed it can be added in future core versions.
SapporoGuy wrote:Since it sounds like that it's not 1.5 or 1.4.9 dpendent, when are you planning on releasing this?

I do have 1 major worry and that is will developers go crazy and start polluting a relatively clean cart?
I say this since there are several mods that actually abuse the system rather than conform and embrace the opencart way of doing things.
You can't pollute the cart since there are no actual changes taking place. I'm actually focused on the last idea for mod management I showed above where mods can be added from a single file. The design I'm working with is:

1. Create a new folder called "vmod" in the store root
2. Create a single file for a mod that changes the latest products to use order by rand() in the query.
3. While still deciding the initial format of mod file(xml or something looser), the basic concept is edit/search/replace
4a. open: catalog/model/catalog/product.php
4b. find:

Code: Select all

ORDER BY p.date_added DESC LIMIT " . (int)$limit);
4b. repl:

Code: Select all

ORDER BY rand() DESC LIMIT " . (int)$limit);
5. save the mod file as whatever "latest_prods_use_rand.vmod"

That's it. If there were more changes to make then you would just keep adding them to that same mod file. If you have multiple mods and 2 of them conflict, you can simply rename/remove each file from the folder one by one and hit refresh on your browser to see which mod is causing the issue. That is all it takes to disable the mod.

I had been using the smf xml format in my development as the initial design, but fear that it may be too tedious for developers to want to use. Example of the above in .xml (SMF) format:

Code: Select all

<modification>
	<id>Latest Products Use Rand</id>
	<version>1.0.0</version>
	<author>qphoria</author>
	<file name="catalog/model/catalog/product.php">
		<operation>
			<search position="replace"><![CDATA[
ORDER BY p.date_added DESC LIMIT " . (int)$limit);
			]]></search>
			<add><![CDATA[
ORDER BY rand() DESC LIMIT " . (int)$limit);
			]]></add>
		</operation>
	</file>
</modification>
To add more than 1 file to mod, just add a new <file> element

While the xml method looks somewhat tedious, I do think we could actually make a mod xml generator to make it easy on devs to just paste the file to edit, the string to search and the string to add after/before/replace. I know SMF has one that generates the xml for you from a simple form.

Or we could use a simpler structure like phpbb easymod:

Code: Select all

#
#-----[ OPEN ]------------------------------------------
#
catalog/model/catalog/product.php
#
#-----[ FIND ]------------------------------------------
#
ORDER BY p.date_added DESC LIMIT " . (int)$limit);
#
#-----[ REPLACE WITH ]------------------------------------------
#
ORDER BY rand() DESC LIMIT " . (int)$limit);
#
#-----[ SAVE/CLOSE ALL FILES ]------------------------------------------
#
# EoM
You may notice that is similar to how I tend to format my forum modification examples

Really the format doesn't matter as I said earlier. They all have the same meat and potatoes, just a different parser. Support could be added for any format.

Image


User avatar
Administrator

Posts

Joined
Tue Jul 22, 2008 3:02 am
Who is online

Users browsing this forum: No registered users and 100 guests