Page 1 of 5

What is the largest shop so far, any scaling issues?

Posted: Wed Feb 24, 2010 8:55 pm
by AndyGaskell
Hi Everyone

I've been making OpenCart shops for a while, but they have mostly been 100-500 products, and I've always found it a great platform. I'm currently working on a shop with nearly 20,000 products, and have noticed some small issues come up as a result.

I was wondering what the largest OpenCart shop developed so-far was, and if there were any issues caused by a large number of products. I'd be really keen to hear from the community about their experiences of large shops. I'm really curious if 20,000 products is fairly normal, or if I'm pushing the boundaries here.

I'm using OpenCart 1.4 on a LAMP server, the shop is for Pet products.

Looking forward to your responses.
Andy

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Feb 24, 2010 10:24 pm
by airetechit
Hi,

We are currently testing OpenCart with 9000 Products on a local server, it seems to be running ok... I know this may not be of much help seen as you are after adding 20,000 products.

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Feb 24, 2010 10:42 pm
by AndyGaskell
Hi David

Thanks for the reply, 9,000 products is a fair size shop, same order of magnitude. Glad to hear it's running ok. :)

I've got all the products in now, and have been doing some testing. The only issue I've seen so-far is that you can't select an image for a product in the admin section, I think this is due to there being 16,000 images in image/data folder and it is trying to generate thumbnails of them all.

ta
Andy

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Feb 24, 2010 10:43 pm
by airetechit
Adding images can be a little of a pain with a product database of a couple thousand products... We just use the Import/Export Module and set all the image file names in there.

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Feb 24, 2010 10:58 pm
by AndyGaskell
My rather hacky solution for the images was to change the the field "image" in "admin/view/template/catalog/product_form.tmpl" on line 66, from type hidden to text. This means you can just type "data/myimage.jpg" in the new text box and hit save. I only did this hack/fix today, and I'm sure there is a more elegant solution.

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Feb 25, 2010 11:10 pm
by tintedpixel
airetechit wrote:Hi,

We are currently testing OpenCart with 9000 Products on a local server, it seems to be running ok... I know this may not be of much help seen as you are after adding 20,000 products.
I tried to get this thread started a few weeks ago, I am very glad to see this discussion finally get traction.

Since our original post http://forum.opencart.com/viewtopic.php?f=10&t=10916, we have put a shop into production that contains over 12k items and is handling @ 2,191unique visits per day on a 1GB of RAM VPS. I have seen upwards of 70 visitors one the site at one time with no performance degradation. I realize this isn't a huge amount of traffic, but so far we are very happy with the result.

We did however have to make some modifications like; query cache, image handling - amongst other things.

Has anyone else had any luck with some of the more recent builds? Traffic/Load?

- Andrew

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Feb 25, 2010 11:38 pm
by AndyGaskell
Hi Andrew

Sorry I missed your post, I skimmed all the titles of existing posts for scaling / large shop sort of chat, before I started this thread.

Great to hear about your shop, not only is that a large number of items, but pretty high traffic volumes too.

You mentioned...
tintedpixel wrote:modifications like; query cache, image handling - amongst other things
...and I'd be really keen to hear what you found worked best for these settings.

On...
tintedpixel wrote:luck with some of the more recent builds? Traffic/Load?
...I'm using 1.4, and am finding it great to use, though we're just in testing stage at the moment, plan on going live end of march, so will let you know how it goes under load then.

Andy

Re: What is the largest shop so far, any scaling issues?

Posted: Sat Feb 27, 2010 2:09 pm
by rph
AndyGaskell wrote:The only issue I've seen so-far is that you can't select an image for a product in the admin section, I think this is due to there being 16,000 images in image/data folder and it is trying to generate thumbnails of them all.
Why didn't you use subdirectories?

Re: What is the largest shop so far, any scaling issues?

Posted: Tue Mar 02, 2010 1:27 am
by tintedpixel
rph wrote:
AndyGaskell wrote:The only issue I've seen so-far is that you can't select an image for a product in the admin section, I think this is due to there being 16,000 images in image/data folder and it is trying to generate thumbnails of them all.
Why didn't you use subdirectories?
in our case we did split into sub directories, most products have several images and this made it much easier to manage with imports.

What types of things have you done/seen with open cart as it relates to scaling?

Re: What is the largest shop so far, any scaling issues?

Posted: Tue Mar 02, 2010 3:09 am
by rph
Image subdirectories really was one of the best new features of 1.4.0.

Unfortunately I don't have experience with large sites. Right now I'm in the process of moving a store with 5k products to OpenCart along with a bunch of custom modding. When it's live it will probably get another 1000 to 2000 products added to it which is still way below the numbers we're talking in here.

What I'd do is see if your slow SQL queries are logged on your server (usually in /tmp/mysql_slow_queries). That would be a good place to start checking OpenCart for bottlenecks.

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Mar 03, 2010 11:20 pm
by ibdesign
A project we are working on currently has 56,290 products. However, we are importing products directly from databases provided by our suppliers, not through the admin panel.

The only issue we have run into so far is that there are so many products in certain categories that when you try to use the Related Products field (which is populated by category), the form area to choose the related products never populates. It must be timing out.

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Mar 04, 2010 2:54 pm
by rph
Maybe some more subcategory division is called for?

Re: What is the largest shop so far, any scaling issues?

Posted: Sun Mar 07, 2010 2:49 am
by tintedpixel
ibdesign wrote:A project we are working on currently has 56,290 products.
Very cool, please keep us in the loop on how this shop performs in production environment under load. I would love to see it and hear more about your mods and environment. We are getting great results (so far) as well.

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Mar 10, 2010 1:23 am
by alleikis
I have been building a store, total products was suppose to reach 27,000. But can't seem to get the import/export to work for more than 13,000 items. '

Adjust timing in export.php with no luck.

Are you using the import/export to add products?

Might be my server needs tweeking..

A real challenge.



url: http://gunorders.com

Re: What is the largest shop so far, any scaling issues?

Posted: Wed Mar 10, 2010 7:42 am
by tintedpixel
alleikis wrote:I have been building a store, total products was suppose to reach 27,000. But can't seem to get the import/export to work for more than 13,000 items. '

Adjust timing in export.php with no luck.

Are you using the import/export to add products?

Might be my server needs tweeking..

A real challenge.



url: http://gunorders.com

Have you tried somehting like this?

Code: Select all

ini_set('memory_limit', '100M');
set_time_limit(0);
or for htaccess maybe something like this..

Code: Select all

php_value memory_limit 100M
php_value upload_max_filesize 100M 
php_value post_max_size 100M
php_value max_execution_time 200
php_value max_input_time 200
also are you getting any errors?

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Mar 18, 2010 9:41 am
by caspara
I currently have over 40,000 downloadable products on my Website:

http://www.downloadxs.com/

I am using VPS hosting with KVCHosting (they are great thus far). Unfortunately, my site is quite slow now. In addition to the regular OpenCart processes, I do have a lot of article samples that are dynamically produced with PHP (well over 100,000) on the article product pages along with an "on the fly" watermarking script for the images. I was planning to have around 100,000 products but imagine it would take a dedicated server to accomplish this.

To attempt to decrease database size, I did hard-code many of the alike product descriptions into the product.tpl file, using IF / ELSE statements depending on the product name. Also, I have redefined how the downloads work upon purchase, so that redundant system processes are limited and only one image, for example, can be used for both the product image and downloadable product while still utilizing the encoding (renaming) feature.

After 25,000 or so products, I began having to use phpMyAdmin to populate my site. Although a bit more work than the export / import mod for OpenCart, it is very quick and relatively easy.

I will be refining my site now to run more quickly, perhaps even moving to a quad-core processor server with more RAM at my current host. Also, here is a post that I found to be helpful for the streamlining process:

http://forum.opencart.com/viewtopic.php ... ads#p40061

Chris O0

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Mar 18, 2010 10:07 am
by rph
caspara wrote: In addition to the regular OpenCart processes, I do have a lot of article samples that are dynamically produced with PHP (well over 100,000) on the article product pages along with an "on the fly" watermarking script for the images.
There's some code in OpenCart for watermarking images already. I haven't had a chance to look through it more thoroughly so I don't know how complete it is.

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Mar 18, 2010 10:32 am
by datacon
I have setup a store that has just over 100,000 products. Works like a dream :) We dont have many people on it at once, but for the few browsign at the same time, its fast.

Only problem with this amount of products ive incurred so far, is the OC backup/restore function in the BO, always fails, as the execution time for php is too low, even when i increase it, and the memory size, it still fails. I have to get a backup manually from the web host.

Re: What is the largest shop so far, any scaling issues?

Posted: Thu Mar 18, 2010 4:51 pm
by AndyGaskell
datacon wrote:Only problem with this amount of products ive incurred so far, is the OC backup/restore function in the BO, always fails, as the execution time for php is too low, even when i increase it, and the memory size, it still fails. I have to get a backup manually from the web host.
I had this issue too, but fixed it by just adding...

Code: Select all

# add larger memory and tinfinate time to allow the backup to work
ini_set("memory_limit", "100M");
ini_set("max_execution_time", 0); # 0 = unlimited exec time 
...at the top of admin/model/tool/backup.php. Workis fine for me. This is in OC 1.4.0 by the way.

Re: What is the largest shop so far, any scaling issues?

Posted: Fri Mar 19, 2010 12:56 pm
by datacon
AndyGaskell wrote:I had this issue too, but fixed it by just adding...

Code: Select all

# add larger memory and tinfinate time to allow the backup to work
ini_set("memory_limit", "100M");
ini_set("max_execution_time", 0); # 0 = unlimited exec time 
...at the top of admin/model/tool/backup.php. Workis fine for me. This is in OC 1.4.0 by the way.
Thanks Andy..Added that code, but still get an error..i think the DB is just too big @ 65MB :(

Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 44 bytes) in C:\Inetpub\httpdocs\system\database\mysql.php on line 29