Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Fri May 31, 2013 11:27 pm

Hello,

I'm getting the following error message whilst trying to do an import using the import/export extension:

Code: Select all

Fatal error: Out of memory (allocated 92274688) (tried to allocate 13610417 bytes) in .../vqmod/vqmod.php on line 389

Extension:

http://www.opencart.com/index.php?route ... ort%20tool

The file is not particularly big (about 2MB) and is no different to our normaly imports.

A couple of months ago I got the "Allowed memory size of...bytes exhausted..." error message and increased the PHP memory limit from 64M to 128M using the two php.ini files in the root and /admin folder. This worked until today when I got the above message. I tried increasing the PHP memory limit to 512M, then 1024M but this did not work.

I had a look at 1and1 and apparently they have a memory limit of 80MB on their Business packages, link:

http://www.1and1.co.uk/details-performance

This (roughly) makes sense as 92274688 = 88MB. I've contacted 1and1 to see if there is anyway I can get around this without having to get a dedicated server.

What I don't understand is why the memory is not being freed up? Does the memory decrease slightly with each import until it runs out? I doubt this is how it's supposed to work.

My understanding is that this memory limit is used to determine how much memory is allocated to PHP functions when they are executed. Surely once they have finished executing the memory being used will be freed?

Any help or further clarification/explanation on this would be very much appreicated. I'm just trying to get my head round why exactly the PHP memory limit is running out.

Thanks,

Mike

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Re: Fatal error: Out of memory using Import/Export tool

pedro1993

Active Member

Posts

915

Joined

Tue Oct 18, 2011 4:31 am

Location

Glasgow, Scotland

Post by pedro1993 » Fri May 31, 2013 11:37 pm

It may be down to a lot of database queries, however I can't put my finger on it exactly as I don't know a lot about server memory limits. All I know is when a large query or a lot of data is being processed it can fail. I would contact 1and1 and ask them to tell you exactly what the server is doing when you use your export/import tool.

For OpenCart & PHP/MySQL support feel free to PM me
Click here for my extentions
Did I help you? Donate here to show support

pedro1993

Active Member

Posts

915

Joined

Tue Oct 18, 2011 4:31 am

Location - Glasgow, Scotland

Re: Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Sat Jun 01, 2013 12:12 am

pedro1993 wrote:It may be down to a lot of database queries, however I can't put my finger on it exactly as I don't know a lot about server memory limits. All I know is when a large query or a lot of data is being processed it can fail. I would contact 1and1 and ask them to tell you exactly what the server is doing when you use your export/import tool.

Thanks for your response, I'm still waiting to hear back from 1and1 as they're being very slow.

I've got quite a few errors in my error log too, for example:

Code: Select all

2013-05-31 14:23:34 - PHP Warning:  file_get_contents(/.../system/cache/cache.product.total.1.0.1.009ac0ca42f90ff959ac7a8ab0656cfd.1370013814) [<a href='function.file-get-contents'>function.file-get-contents</a>]: failed to open stream: No such file or directory in /.../system/library/cache.php on line 9

There is at least one of these errors every minute. Could this have something to do with it? I've been told on the forums in the past that these are nothing to worry about. Also:

Code: Select all

2013-05-31 14:24:29 - PHP Notice:  Undefined index:  filter_name in /.../vqmod/vqcache/vq2-catalog_model_catalog_product.php on line 242

Thanks,

Mike

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Re: Fatal error: Out of memory using Import/Export tool

rph

Expert Member

Posts

Joined

Fri Jan 08, 2010 5:05 am

Location

Lincoln, Nebraska

Post by rph » Sat Jun 01, 2013 12:14 am

Import/Export is based on PHPExcel which can be very memory intensive.

http://stackoverflow.com/questions/5560 ... -exhausted

The two main options are contacting your host to increase the memory limit like you've done or doing the import on a local machine then restoring that database to your server.

-Ryan

rph

Expert Member

Posts

Joined

Fri Jan 08, 2010 5:05 am

Location - Lincoln, Nebraska

Re: Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Sat Jun 01, 2013 12:33 am

rph wrote:Import/Export is based on PHPExcel which can be very memory intensive.

http://stackoverflow.com/questions/5560 ... -exhausted

The two main options are contacting your host to increase the memory limit like you've done or doing the import on a local machine then restoring that database to your server.

Thanks for your reply.

1and1 have stated the obvious by saying if I want to increase the PHP memory limit I'll need to upgrade to a server package, by this I'm assuming they mean a VPS/dedicated server.

However they could not explain to me why the memory limit is being reached, and could provide no detail on how the memory limit is used and why PHP processes continue to use this memory even after they have been processed. This is what I'd like to know so that I could try to fix it without ungrading with 1and1.

I asked Support if it was standard practice to just keep increasing the memory limit throughout the duration of the website's life, and he said yes despite me having read that this is generally a bad thing to do, as it is not fixing the problem but just delaying it. Is this correct or am I missing the point?

I've also had a look in my server logs and cannot see anything obvious, however this is mainly because I'm not sure what to look for.

I'm at a bit of a loss and any help would be very much appreciated.

Cheers,

Mike

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Re: Fatal error: Out of memory using Import/Export tool

rph

Expert Member

Posts

Joined

Fri Jan 08, 2010 5:05 am

Location

Lincoln, Nebraska

Post by rph » Sat Jun 01, 2013 1:28 am

Large spreadsheets use lots of memory in PHPExcel. (You can find more info in the above link.) Increasing the memory limit may allow you to complete the operation, it may cause it to fail later. It ultimately depends on how much memory the operation needs. If your host won't increase the limit (many won't) run the conversion on a local machine (use AMPPS/WAMP/MAMP/etc.) then export the new database to your store. You can also hire a developer to do it for you.

-Ryan

rph

Expert Member

Posts

Joined

Fri Jan 08, 2010 5:05 am

Location - Lincoln, Nebraska

Re: Fatal error: Out of memory using Import/Export tool

patchworkqueen

New member

Posts

Joined

Sat May 19, 2012 5:54 am

Post by patchworkqueen » Mon Jun 03, 2013 4:04 pm

I'm having exactly the same problem with 123-reg. I haven't been able to upload using a spreadsheet for a year as it's too big now so do everything locally and upload it back. Have complained several times and all I get back is that there is 64MB limit which seems a huge amount to me. So guess it's the amount of images on my site that is causing the problem. Anyone know of a way of reducing the images without impairing the quality and size?

patchworkqueen

New member

Posts

Joined

Sat May 19, 2012 5:54 am

Re: Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Mon Jun 03, 2013 4:13 pm

rph wrote:Large spreadsheets use lots of memory in PHPExcel. (You can find more info in the above link.) Increasing the memory limit may allow you to complete the operation, it may cause it to fail later. It ultimately depends on how much memory the operation needs. If your host won't increase the limit (many won't) run the conversion on a local machine (use AMPPS/WAMP/MAMP/etc.) then export the new database to your store. You can also hire a developer to do it for you.

Hi,

This spreadsheet is no larger than the spreadsheets we've been importing for the last few months, this is why I'm not sure why it is occuring. Is the spreadsheet requiring more and more resource every time it is being uploaded? Is this the correct process?

I'm worried that if we get a dedicated server for this site and increase the PHP limit, then the same error will occur in a few months time when this memory limit 'runs out'?

Thanks,

Mike

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Re: Fatal error: Out of memory using Import/Export tool

rph

Expert Member

Posts

Joined

Fri Jan 08, 2010 5:05 am

Location

Lincoln, Nebraska

Post by rph » Tue Jun 04, 2013 1:27 am

MikeSCC wrote:This spreadsheet is no larger than the spreadsheets we've been importing for the last few months, this is why I'm not sure why it is occuring.

I'm not sure why you'd be importing if that were the case. I'm guessing it's just slightly larger which is enough to go over your limit. If it's the exact same file it's possible other processes are just using more memory as memory_limit applies to all PHP processes.

I'm worried that if we get a dedicated server for this site and increase the PHP limit, then the same error will occur in a few months time when this memory limit 'runs out'?

memory_limit does not run out. You exceed it. If you don't like either of the two supplied solutions then you would have to hire a programmer to rewrite Export/Import or develop a whole new mod for you.

-Ryan

rph

Expert Member

Posts

Joined

Fri Jan 08, 2010 5:05 am

Location - Lincoln, Nebraska

Re: Fatal error: Out of memory using Import/Export tool

grgr

Active Member

Posts

Joined

Mon Mar 28, 2011 4:08 pm

Location

Post by grgr » Tue Jun 04, 2013 1:47 am

MikeSCC wrote:
rph wrote:Large spreadsheets use lots of memory in PHPExcel. (You can find more info in the above link.) Increasing the memory limit may allow you to complete the operation, it may cause it to fail later. It ultimately depends on how much memory the operation needs. If your host won't increase the limit (many won't) run the conversion on a local machine (use AMPPS/WAMP/MAMP/etc.) then export the new database to your store. You can also hire a developer to do it for you.
Hi,

This spreadsheet is no larger than the spreadsheets we've been importing for the last few months, this is why I'm not sure why it is occuring. Is the spreadsheet requiring more and more resource every time it is being uploaded? Is this the correct process?

I'm worried that if we get a dedicated server for this site and increase the PHP limit, then the same error will occur in a few months time when this memory limit 'runs out'?

Thanks,

Mike

You could always move your site to more sensible hosting of course and then you would at least be able to increase the memory limit a bit - depends just how high you need it I guess.

I upped my memory to 512 the other day (using the php.ini file in the admin folder on a shared hosting account) - I've since put it back down as i don't need it that high for daily use.

-

VIEW ALL EXTENSIONS * EXTENSION SUPPORT * WEBSITE * CUSTOM REQUESTS

grgr

Active Member

Posts

Joined

Mon Mar 28, 2011 4:08 pm

Location - UK

Re: Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Tue Jun 04, 2013 4:31 pm

rph wrote:I'm not sure why you'd be importing if that were the case. I'm guessing it's just slightly larger which is enough to go over your limit. If it's the exact same file it's possible other processes are just using more memory as memory_limit applies to all PHP processes.

Yes you're correct, apologies for not being clear. It must be a fraction larger than previous imports due to more content being added, I thought as the difference was so small this would not be the issue.

rph wrote:memory_limit does not run out. You exceed it. If you don't like either of the two supplied solutions then you would have to hire a programmer to rewrite Export/Import or develop a whole new mod for you.

OK thanks for explaining, I now have a better understanding of how the PHP memory limit works, the plan now is to remove all old '0' stock products which we are not likely to get back in stock (which is quite a few hopefully), which will therefore free up a lot of processing power.

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Re: Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Tue Jun 04, 2013 4:35 pm

grgr wrote:You could always move your site to more sensible hosting of course and then you would at least be able to increase the memory limit a bit - depends just how high you need it I guess.

I upped my memory to 512 the other day (using the php.ini file in the admin folder on a shared hosting account) - I've since put it back down as i don't need it that high for daily use.

Thanks for your reply, I've looked at dedicated server packages with 1and1 and they're not too bad price-wise (~£30pm), however I want to avoid having to do this as I'd need to keep is seperate from our current shared hosting due to other websites, and therefore would have to migrate all the website data over which is just a hassle I could do without.

We're going to go through the import file and delete all products we no longer need, which should free up lots of processing power.

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Re: Fatal error: Out of memory using Import/Export tool

grgr

Active Member

Posts

Joined

Mon Mar 28, 2011 4:08 pm

Location

Post by grgr » Tue Jun 04, 2013 7:40 pm

I thought it was going to be a hassle to move my main shop, but they did it for me at no extra cost and hardly noticed it happened.

That said, deleting old products isn't a bad thing either.

-

VIEW ALL EXTENSIONS * EXTENSION SUPPORT * WEBSITE * CUSTOM REQUESTS

grgr

Active Member

Posts

Joined

Mon Mar 28, 2011 4:08 pm

Location - UK

Re: Fatal error: Out of memory using Import/Export tool

Active Member

Posts

Joined

Tue Nov 20, 2012 5:55 pm

Post by MikeSCC » Fri Jun 07, 2013 6:12 pm

Deleting all the old products did the trick and freed up lots of memory usage, so the problem is solved.

It does mean however that we can only have around 1200 products in OC at any one time. It would be ideal if OC came with a good import/export functionality by default so that we do not have to rely on memory-hungry extensions instead.

Thanks for all help received.

Active Member

Posts