User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by Andi » Sun Nov 15, 2009 2:53 am

Hi,

I'm new to OpenCart - but my main reason for converting one of my larger shops to open cart was this very extension!
With 235 products, many with various sizes, etc - I really didn't want to have to manually put them all in. And couldn't find a suitable conversion utility from CubeCart v3 (yes, the old, naff, freebie one!) to OC.

So I took a dump of the MySQL database, then used the data to lay it out within your export/import spreadsheet...

And it /nearly/ works.
At the moment, I get "500 Internal Server Error" after it processes for about 30 seconds.
And if you look in the shop, all the categories are present, as well as all the products with pictures - just no Options.
It seems to stall on the options section.

I'm guessing it may be an issue with my spreadsheet?
I was wondering if you could take a look and see if there's something missing or wrong that might be causing it to stall?
http://dl.dropbox.com/u/82851/backup_ca ... s_ANDI.xls

Any help gratefully received!

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by JNeuhoff » Sun Nov 15, 2009 3:03 am

I just tested your spreadsheet file by importing it to my 1.3.4 test server, and it works fine.

The internal 500 error message may indicate that the problem is elsewhere. Please check your server's error_log and let me know what it contains. it will tell you what the problem is.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by Andi » Sun Nov 15, 2009 3:11 am

Hmmm, its a shared server on Dreamhost - there isn't anything I can change on it.

Perhaps the sql server is lagging, since its a separate server again.
Will try again later.

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by JNeuhoff » Sun Nov 15, 2009 3:29 am

Here is how to access your error log, according to Dreamhost:
Every time somebody visits your web site, our server takes a log of it in either your access.log file or your error.log file (for times when people get errors like 404, etc... These logs are available for you to download at any time via FTP in the directory logs/yourdomain.com/http/ on our server. We rotate the logs every night so that access.log and error.log only contain today's logs, while error.log.0 and access.log.0 contain yesterdays (the last FULL day of logging available). After two days, we delete the log files.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by Andi » Sun Nov 15, 2009 4:12 am

Ah, thank you - you learn something everyday! :)

http://dl.dropbox.com/u/82851/error.log

It's a brand new install of OpenCart 1.3.4 - no mods other than yours.

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by JNeuhoff » Sun Nov 15, 2009 7:58 am

I don't think it has anything to do with the Export/Import module as such.
The error log repeatedly reports this:

Premature end of script headers: index.php

Can you upload a file with <?php phpinfo(); ?> on your server and run it from your web browser so I can see your PHP configuration? Looks like issue with the Zend Optimiser or some wrong PHP settings.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by Andi » Sun Nov 15, 2009 8:07 am

Here you go:
http://www.newtonsonline.com/phpinfo.php

Thank you for your help with this.
I guess if I can't fix the server, I could always get a SQL dump of it uploaded on another (working) server and import into mine.
It is a one-time only update.
Though might be nice to get it working for future reference! :)

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by moggiex » Sun Nov 15, 2009 9:23 am

JNeuhoff wrote:This module is now available for the OpenCart 1.3.4 version.
Howdy,

Brillant module, one of the reasons why i first chose OpenCart, why its not include by default I'l never know .... HINT HINT!

Anyway... I have had issues with imports timing out and was scanning through the code, when I noticed this:

admin/model/tool/export.php

Code: Select all

	function upload( $filename ) {
		set_error_handler('error_handler_for_export',E_ALL);
		$database = Registry::get('db');
		require_once 'Spreadsheet/Excel/Reader.php';
		ini_set("memory_limit","512M");
		ini_set("max_execution_time",180);
Why is the max_execution_time being set?

My imports are massive, they're going to be topping 60,000 lines, currently I'm at 9,000 and struggling, so there is no chance they're going to complete in 180 seconds :)

Also... Size is becoming a nasty issue currently, could an option be added that if it finds no categories, that it doesn't * delete them all ??? ??? ??? ??? ??? The same goes for the products, it cleans first and then adds, nightmare.... Could there be an option added for append or update where the ID matches?

Matt

Code: Select all

It was like that when I found it, honest!


User avatar
Active Member

Posts

Joined
Mon Nov 09, 2009 9:55 pm

Post by Andi » Sun Nov 15, 2009 9:26 am

180 seconds is probably the amount of time it takes from the upload completing and seeing the Error 500 in my case too...
Perhaps it works on your test server as it has more CPU cycles?
Dreamhost shared hosting is a little 'busy'....

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by moggiex » Sun Nov 15, 2009 9:30 am

Andi wrote:180 seconds is probably the amount of time it takes from the upload completing and seeing the Error 500 in my case too...
Perhaps it works on your test server as it has more CPU cycles?
Dreamhost shared hosting is a little 'busy'....
Hmm... That 500 error is normally because you've killed the htaccess files, rename any you have temporarily and try again.

Matt

Code: Select all

It was like that when I found it, honest!


User avatar
Active Member

Posts

Joined
Mon Nov 09, 2009 9:55 pm

Post by Andi » Sun Nov 15, 2009 9:36 am

My Import partially works though - I get categories and products, but it times out before completing options.
Not sure how that links to .htaccess files?
The only one I have is the one in the root, named htaccess.txt which I renamed correctly so it could be used...

Will try removing it though.
And might increase the timeout - just to see.
... in the morning though! :)

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by moggiex » Sun Nov 15, 2009 9:39 am

Andi wrote:My Import partially works though - I get categories and products, but it times out before completing options.
Not sure how that links to .htaccess files?
The only one I have is the one in the root, named htaccess.txt which I renamed correctly so it could be used...

Will try removing it though.
And might increase the timeout - just to see.
... in the morning though! :)
yea, I've always know 500 to be a screw up on my part in the htaccess file(s), a quick google kinda says the same thing http://wordpress.org/support/topic/278609

Tis an odd one m8

Matt

Code: Select all

It was like that when I found it, honest!


User avatar
Active Member

Posts

Joined
Mon Nov 09, 2009 9:55 pm

Post by Andi » Sun Nov 15, 2009 10:38 pm

I've just changed

Code: Select all

ini_set("max_execution_time",180);
to 1800 instead of 180.

Ran fine, and all products imported.
Sorted.

(I didn't touch the .htaccess file, alter the XLS file - nothing! Just altered that one line and tried again).
Hope that helps.

Newbie

Posts

Joined
Sun Nov 15, 2009 2:47 am

Post by moggiex » Sun Nov 15, 2009 10:41 pm

Andi wrote:I've just changed

Code: Select all

ini_set("max_execution_time",180);
to 1800 instead of 180.

Ran fine, and all products imported.
Sorted.

(I didn't touch the .htaccess file, alter the XLS file - nothing! Just altered that one line and tried again).
Hope that helps.
Sweet, wish that solved my issue, the file is so big, on load it basically fails, it looks like it is down the reader.php file itself.

I did find that by commenting out the delete statements for the product load function, meant I could load smaller files on top of each other without it wiping the data every time.

Could really do with 3 check box options adding to the admin page, to confirm if its new data or old for categories, products and options :)

Matt

Code: Select all

It was like that when I found it, honest!


User avatar
Active Member

Posts

Joined
Mon Nov 09, 2009 9:55 pm

Post by JNeuhoff » Sun Nov 15, 2009 11:59 pm

I think trying to upload 60 000 products won't work the Import because the reader.php, which the Import uses. is very memory hungry, about 64MB to 128MB per 1000 products.

I have been thinking about replacing the XLS format with the Open Document Spreadsheet format if I can find suitable reader and writer classes for it, and if it uses significant less memory.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by moggiex » Mon Nov 16, 2009 12:33 am

JNeuhoff wrote:I think trying to upload 60 000 products won't work the Import because the reader.php, which the Import uses. is very memory hungry, about 64MB to 128MB per 1000 products.

I have been thinking about replacing the XLS format with the Open Document Spreadsheet format if I can find suitable reader and writer classes for it, and if it uses significant less memory.
Yea I guessed loading all in one go was taking the pee a bit. However, even allowing php to consume 2Gb of memory, it still does the same for ~9000 records.

I think in this instance, the ability to specify if its an update (thus replace) or an addition (thus add) would be appropriate (I've taken to manually editing the export.php file now lol)

Matt

Code: Select all

It was like that when I found it, honest!


User avatar
Active Member

Posts

Joined
Mon Nov 09, 2009 9:55 pm

Post by cp30 » Mon Nov 16, 2009 12:50 am

JNeuhoff,
I have the 134 version of opencart and the 134 version of your import /export contribution. I am able to import my 10k of line items into the mySQL ok. Where I am having the problem is exporting them to the xls format. I keep getting the 500 error on the export. I didn't really have any problem with this until I added 2k more to my import. Is there any way to adjust the limitations on the export size. I know that excel 2003 could handle upto 65k lines items.

Newbie

Posts

Joined
Wed Oct 28, 2009 9:03 am

Post by JNeuhoff » Mon Nov 16, 2009 1:12 am

I didn't really have any problem with this until I added 2k more to my import. Is there any way to adjust the limitations on the export size. I know that excel 2003 could handle upto 65k lines items.
Try increasing the php setting memory_limit to a higher value.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by JNeuhoff » Mon Nov 16, 2009 1:18 am

I think in this instance, the ability to specify if its an update (thus replace) or an addition (thus add) would be appropriate (I've taken to manually editing the export.php file now lol)
I won't have the time in the near future to do such a change. However, if you manage to add such a change to this module, I'd be willing to incorporate it in future versions.

Export/Import Tool * SpamBot Buster * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Tag Manager * Survey Plus * OpenTwig


User avatar
Guru Member

Posts

Joined
Wed Dec 05, 2007 3:38 am

Who is online

Users browsing this forum: No registered users and 20 guests