System Tools plan

I am thinking about enhancing the Backup/Restore page in the admin to be a full toolset for admins.

The tools I'd like to have there are:

- Database backup/restore - Backup/restore as .sql
- CSV import/export -Allows you to export a single table at a time in CSV format, edit the fields in excel, save (as .csv) and upload back to the server. I made it completely dynamic so that any custom columns or tables will automatically be supported.
- Cleanup/Reset - Reset System Cache, Reset Image Cache, Reset Order Id

Any other ideas?

cool.. five thumbs..
just wondering, can we choose a certain backup? like customer or sales data?

Sounds great. It might be a good idea to have the option of truncating the tables or not so that you can use the import as a template and then upload other products or whatever without having to overwrite the previous ones

Is this going to be part of the core?

JAY6390 wrote:Sounds great. It might be a good idea to have the option of truncating the tables or not so that you can use the import as a template and then upload other products or whatever without having to overwrite the previous ones

Is this going to be part of the core?

yea I was determining if the truncate or append option would work.
Some of it will be added to core I think. I might first put it out as a mod for 1.4.9

A decent import/export will either truncate or simply overwrite everything. Append is good for new items, but useless with edits or deletions, which are as much a part of inventory management as adding new items.

And more and more suppliers are offering their inventories through rss or xml and the fact that cart owners are going to be receiving data in such ways and making it easy to convert to useful results within OC should be considered.

If you are going to get into this, do it completely, not a bit here and there and set it up so there does not need to be all sorts of changes from one version to the next, as it is now with the mod that is offered by JNeuhoff. Not complaining about his, but it does seem to need a change every time there is a new version of OC.

Having the capability of being able to map fields from the source to the csv file would be good, as well.

peteVA wrote:A decent import/export will either truncate or simply overwrite everything. Append is good for new items, but useless with edits or deletions, which are as much a part of inventory management as adding new items.

And more and more suppliers are offering their inventories through rss or xml and the fact that cart owners are going to be receiving data in such ways and making it easy to convert to useful results within OC should be considered.

If you are going to get into this, do it completely, not a bit here and there and set it up so there does not need to be all sorts of changes from one version to the next, as it is now with the mod that is offered by JNeuhoff. Not complaining about his, but it does seem to need a change every time there is a new version of OC.

Having the capability of being able to map fields from the source to the csv file would be good, as well.

Yeah I don't think this is meant to be a complete import/export solution. There are many things to consider in a complete solution. Some of which you mentioned, but there are more, and it is an enormous task all the things you mention and more are very easy for us to think of and to say hey I want this, but actually implementing them in code is extremely difficult. The biggest problem I have with the import/export mod is that it's excel and those files cannot be processed serial meaning that the resource requirements rise very quickly with the amount of data you are importing.

Correct. This is meant to be a simplified universal solution. JN's mod is great because it handles the relationships between products, product description, product options, product option values, etc. But the downfall is that any field change requires it to be updated because it has to name all the tables and columns

This is a completely dynamic, but simple solution. You export into a more readable, excel editable file:
address table example


It uses the column title row as the table mapping system. No matter what order you put them in, as long as the table.column matches, it will import. It is also smart enough to ignore fields that no longer exist or news fields that aren't listed. There are no table names or column names in the code. It simply grabs ALL tables and prints them out. Then when importing it reads the column row and inserts back into the table based on the column address.

The catch is that it only does single tables at a time and its raw data. But for the most part, you should be able to follow along or export 2 tables and visually link them while making changes.

Qphoria wrote:Any other ideas?

Language Files editor from the BO?

And I mean only the language related files.

Also: a server_info.php wich will give you all info about your server.
osCommerce has this and is damn handy sometime:

Server host: dutch53.vdx.nl (85.158.248.53) Database host: localhost (127.0.0.1)
Server besturingssysteem: Linux 2.6.9-42.ELsmp Database: MySQL 4.1.20-log
Server datum: 21.08.2010 19:35:56 Datebase datum: 21.08.2010 19:35:56
Server uptime:
HTTP server: Apache
PHP versie: 4.4.0 (Zend: 1.3.0)

+ a lot more

i2Paq wrote:

Qphoria wrote:Any other ideas?

Language Files editor from the BO?

And I mean only the language related files.

Also: a server_info.php wich will give you all info about your server.
osCommerce has this and is damn handy sometime:

Server host: dutch53.vdx.nl (85.158.248.53) Database host: localhost (127.0.0.1)
Server besturingssysteem: Linux 2.6.9-42.ELsmp Database: MySQL 4.1.20-log
Server datum: 21.08.2010 19:35:56 Datebase datum: 21.08.2010 19:35:56
Server uptime:
HTTP server: Apache
PHP versie: 4.4.0 (Zend: 1.3.0)

+ a lot more

yeah these are not bad ideas. Both of them could pretty easily be swiped from oscommerce.

Qphoria wrote:The catch is that it only does single tables at a time and its raw data. But for the most part, you should be able to follow along or export 2 tables and visually link them while making changes.

I think you are taking waaaay too much for granted. I would guess that most USERS shake and shiver at the thought of doing anything at associated with a database, have no idea what a table is and would have no idea how to combine or open more than one at a time.

The beauty and the growth of shopping carts is due to the fact that USERS do not have to know how to do specific functions or routines, but that it is all done for them. Things that you can do in your sleep, with one hand and a few fingers tied behind your back would cause sleepless nights and nails bitten down to the quick for most USERS.

Personally, while it may be a bit of a timesaver for those in the know, who will have no problem editing the database anyhow, I think that something like you propose will only cause grief for many who do not fully understand and who also don't (regardless of all the warnings) do backups first.

This could easily be a weapon, rather than a tool.

Maybe offer it as an option for those who want to use it, or come up with a complete solution that does integrate all required fields and relationships.

The other part of the problem is that users don't have a clue what it takes to make things as simple as they want them. It could easily take a programmer months of full time hours to build an import/export as smart as you want it to be, and since no one here works full time on opencart you can extend that timeframe quite a bit, and that's just for the one feature.

using excel for uploading products is really good idea, easy to use, especially to upload large amount of items.

Personally, I'm quite happy with JNeuhoff's package. From my standpoint, if it were incorporated with OC in such a way that you did not have to get a new version with each change and not have to worry that the version of the import/export matched the version on OC, I would be quite happy.

BUT, as someone who hosts numerous carts of all types and is heavy in wholesale / dropshipping forums I know there are many people who are using suppliers with feeds, simply because they have the feeds available. There are many who are looking for a "set it and forget it" type situation.

OC has so much going for it, I recommend it whenever I feel comfortable doing so, but when someone mentions RSS or XML I have to tell them OC is not a good choice.

Quite frankly, it seems there are more feeds designed for, or built around, Zen than all others combined.

Again, for me, if it were not for the constantly changing database structure, I'm perfectly happy with JNeuhoff's deal.

peteVA wrote: OC has so much going for it, I recommend it whenever I feel comfortable doing so, but when someone mentions RSS or XML I have to tell them OC is not a good choice.

Quite frankly, it seems there are more feeds designed for, or built around, Zen than all others combined.

can you tell me one single cart that makes it easy to integrate an xml feed? you can get someone to custom build you an xml import for opencart just like you can for any other cart. Every suppliers feed is different, so there is no way to build an import that automatically fits them all. It's not opencarts fault that the distributors build their import scripts for zencart there's nothing we can do about that.

using excel for uploading products is really good idea, easy to use, especially to upload large amount of items.

actually it's really not a good way to upload a large amount of items because excel for php has to parse the whole file to import where csv can be imported one line at a time to allow for much larger files.

Xsecrets wrote:

PeteVA wrote:using excel for uploading products is really good idea, easy to use, especially to upload large amount of items.

actually it's really not a good way to upload a large amount of items because excel for php has to parse the whole file to import where csv can be imported one line at a time to allow for much larger files.

XLS format isn't needful to work with Excel... CSV format can be easily open and save by Excel and all other spreadsheets.

peteVA wrote: I think you are taking waaaay too much for granted. I would guess that most USERS shake and shiver at the thought of doing anything at associated with a database

Then don't use it

Maybe offer it as an option for those who want to use it, or come up with a complete solution that does integrate all required fields and relationships.

If you want the full integrated option, use the Excel Import/Export by JNuehoff. This is designed to be a simple tool. Yes it could be used as a weapon, but that is why you need to backup your database first before doing anything.

I follow this topic with great interest.

PeteVA say that's better to make a tool that can be use by merchants to update, add, delete or save data.
I'm think so. BUT I think that's a good goal and not the first step...
This development is too difficult to think we could make "The ideal tool" that everyone dream to want... as it was said before.

I think the way Qphoria ans Xsecret take is good. Why ?
Because this first step is going to allow to have a basic tool to update or add data. That's right, it isn't perfect because it will be difficult to novices to use it. But it will be simple and generic !
If I understand well (I'm French and it's possible I misunderstand some informations... ) Data could be saved from one version of OC and reload even if fields have been added or deleted...
So, if table's links don't change between two OC's versions that means we can save category and products data from old OC to new OC ! That's great !
And it means too we could to have an import / export system even if we added specific fields to our products, category, customers, etc. That's very cool ! not ?

After, I think (I hope) we'll can easily use import/export functions to develop an advanced import/export extension. We 'll have just to import or export some tables and merge data in one file for export and explode data in each table for import. And the extension will automatically have the advantages of the basic import/export !

The advanced extension will not need to be added to each version of OC as JNeuhoff's extension... Only change that will need update will be if table's link change...

Isn't it a good news ??

Actually, I would prefer csv with a pipe delimiter, but don't expect it any time soon.

And again, I am not complaining one bit, only mentioning what I hear from my hosting clients and see on other forums.

I'm not taking up for anything, simply stating that MANY OC USERS are not active members of this forum, are not coders of any sort and are deathly afraid of databases. Dangling something in front of them that they do not understand could cause a lot of problems for them.

It's easy for the core of a dozen or so contributors here to blow it off, as this sort of thing is as basic as washing their hands to them, but it is a BIG THING to many.

If you truly want OC to be the next big thing, it has to appeal to and be usable by the riff-raff, as well as those who do not mind digging into the code and database.

peteVA wrote: If you truly want OC to be the next big thing, it has to appeal to and be usable by the riff-raff, as well as those who do not mind digging into the code and database.

I'm agree with you...
You know in France it's worth because there is language barrier too... And a lot of module, extensions, tricks and other useful informations are not reachable a lot of people...
However we have a good French team ! A little small but energetic

I truly think OC can become "the next big thing". But we must go step by step...

"CSV" ... "comma" separated values

"PSV"... "pipe" separated values?