Post by AndyGaskell » Wed Feb 24, 2010 8:55 pm

Hi Everyone

I've been making OpenCart shops for a while, but they have mostly been 100-500 products, and I've always found it a great platform. I'm currently working on a shop with nearly 20,000 products, and have noticed some small issues come up as a result.

I was wondering what the largest OpenCart shop developed so-far was, and if there were any issues caused by a large number of products. I'd be really keen to hear from the community about their experiences of large shops. I'm really curious if 20,000 products is fairly normal, or if I'm pushing the boundaries here.

I'm using OpenCart 1.4 on a LAMP server, the shop is for Pet products.

Looking forward to your responses.
Andy

I'm a web developer at http://www.ssofb.co.uk in Aberdeen, Scotland.
Got a random blog at http://www.andygaskell.blogspot.com/.


User avatar
New member

Posts

Joined
Thu Dec 10, 2009 3:59 am
Location - Aberdeen UK

Post by airetechit » Wed Feb 24, 2010 10:24 pm

Hi,

We are currently testing OpenCart with 9000 Products on a local server, it seems to be running ok... I know this may not be of much help seen as you are after adding 20,000 products.

Best Regards,
David Catherall,
AireTech IT.


New member

Posts

Joined
Fri Feb 05, 2010 1:46 am
Location - Goole, United Kingdom

Post by AndyGaskell » Wed Feb 24, 2010 10:42 pm

Hi David

Thanks for the reply, 9,000 products is a fair size shop, same order of magnitude. Glad to hear it's running ok. :)

I've got all the products in now, and have been doing some testing. The only issue I've seen so-far is that you can't select an image for a product in the admin section, I think this is due to there being 16,000 images in image/data folder and it is trying to generate thumbnails of them all.

ta
Andy

I'm a web developer at http://www.ssofb.co.uk in Aberdeen, Scotland.
Got a random blog at http://www.andygaskell.blogspot.com/.


User avatar
New member

Posts

Joined
Thu Dec 10, 2009 3:59 am
Location - Aberdeen UK

Post by airetechit » Wed Feb 24, 2010 10:43 pm

Adding images can be a little of a pain with a product database of a couple thousand products... We just use the Import/Export Module and set all the image file names in there.

Best Regards,
David Catherall,
AireTech IT.


New member

Posts

Joined
Fri Feb 05, 2010 1:46 am
Location - Goole, United Kingdom

Post by AndyGaskell » Wed Feb 24, 2010 10:58 pm

My rather hacky solution for the images was to change the the field "image" in "admin/view/template/catalog/product_form.tmpl" on line 66, from type hidden to text. This means you can just type "data/myimage.jpg" in the new text box and hit save. I only did this hack/fix today, and I'm sure there is a more elegant solution.

I'm a web developer at http://www.ssofb.co.uk in Aberdeen, Scotland.
Got a random blog at http://www.andygaskell.blogspot.com/.


User avatar
New member

Posts

Joined
Thu Dec 10, 2009 3:59 am
Location - Aberdeen UK

Post by tintedpixel » Thu Feb 25, 2010 11:10 pm

airetechit wrote:Hi,

We are currently testing OpenCart with 9000 Products on a local server, it seems to be running ok... I know this may not be of much help seen as you are after adding 20,000 products.
I tried to get this thread started a few weeks ago, I am very glad to see this discussion finally get traction.

Since our original post http://forum.opencart.com/viewtopic.php?f=10&t=10916, we have put a shop into production that contains over 12k items and is handling @ 2,191unique visits per day on a 1GB of RAM VPS. I have seen upwards of 70 visitors one the site at one time with no performance degradation. I realize this isn't a huge amount of traffic, but so far we are very happy with the result.

We did however have to make some modifications like; query cache, image handling - amongst other things.

Has anyone else had any luck with some of the more recent builds? Traffic/Load?

- Andrew

http://www.tintedpixel.com
Web Centric Creative


User avatar
New member

Posts

Joined
Fri Sep 25, 2009 11:56 pm
Location - Denver, Colorado

Post by AndyGaskell » Thu Feb 25, 2010 11:38 pm

Hi Andrew

Sorry I missed your post, I skimmed all the titles of existing posts for scaling / large shop sort of chat, before I started this thread.

Great to hear about your shop, not only is that a large number of items, but pretty high traffic volumes too.

You mentioned...
tintedpixel wrote:modifications like; query cache, image handling - amongst other things
...and I'd be really keen to hear what you found worked best for these settings.

On...
tintedpixel wrote:luck with some of the more recent builds? Traffic/Load?
...I'm using 1.4, and am finding it great to use, though we're just in testing stage at the moment, plan on going live end of march, so will let you know how it goes under load then.

Andy

I'm a web developer at http://www.ssofb.co.uk in Aberdeen, Scotland.
Got a random blog at http://www.andygaskell.blogspot.com/.


User avatar
New member

Posts

Joined
Thu Dec 10, 2009 3:59 am
Location - Aberdeen UK

Post by rph » Sat Feb 27, 2010 2:09 pm

AndyGaskell wrote:The only issue I've seen so-far is that you can't select an image for a product in the admin section, I think this is due to there being 16,000 images in image/data folder and it is trying to generate thumbnails of them all.
Why didn't you use subdirectories?

-Ryan


rph
Expert Member

Posts

Joined
Fri Jan 08, 2010 5:05 am
Location - Lincoln, Nebraska

Post by tintedpixel » Tue Mar 02, 2010 1:27 am

rph wrote:
AndyGaskell wrote:The only issue I've seen so-far is that you can't select an image for a product in the admin section, I think this is due to there being 16,000 images in image/data folder and it is trying to generate thumbnails of them all.
Why didn't you use subdirectories?
in our case we did split into sub directories, most products have several images and this made it much easier to manage with imports.

What types of things have you done/seen with open cart as it relates to scaling?

http://www.tintedpixel.com
Web Centric Creative


User avatar
New member

Posts

Joined
Fri Sep 25, 2009 11:56 pm
Location - Denver, Colorado

Post by rph » Tue Mar 02, 2010 3:09 am

Image subdirectories really was one of the best new features of 1.4.0.

Unfortunately I don't have experience with large sites. Right now I'm in the process of moving a store with 5k products to OpenCart along with a bunch of custom modding. When it's live it will probably get another 1000 to 2000 products added to it which is still way below the numbers we're talking in here.

What I'd do is see if your slow SQL queries are logged on your server (usually in /tmp/mysql_slow_queries). That would be a good place to start checking OpenCart for bottlenecks.

-Ryan


rph
Expert Member

Posts

Joined
Fri Jan 08, 2010 5:05 am
Location - Lincoln, Nebraska

Post by ibdesign » Wed Mar 03, 2010 11:20 pm

A project we are working on currently has 56,290 products. However, we are importing products directly from databases provided by our suppliers, not through the admin panel.

The only issue we have run into so far is that there are so many products in certain categories that when you try to use the Related Products field (which is populated by category), the form area to choose the related products never populates. It must be timing out.

Newbie

Posts

Joined
Wed Feb 17, 2010 6:35 am

Post by rph » Thu Mar 04, 2010 2:54 pm

Maybe some more subcategory division is called for?

-Ryan


rph
Expert Member

Posts

Joined
Fri Jan 08, 2010 5:05 am
Location - Lincoln, Nebraska

Post by tintedpixel » Sun Mar 07, 2010 2:49 am

ibdesign wrote:A project we are working on currently has 56,290 products.
Very cool, please keep us in the loop on how this shop performs in production environment under load. I would love to see it and hear more about your mods and environment. We are getting great results (so far) as well.

http://www.tintedpixel.com
Web Centric Creative


User avatar
New member

Posts

Joined
Fri Sep 25, 2009 11:56 pm
Location - Denver, Colorado

Post by alleikis » Wed Mar 10, 2010 1:23 am

I have been building a store, total products was suppose to reach 27,000. But can't seem to get the import/export to work for more than 13,000 items. '

Adjust timing in export.php with no luck.

Are you using the import/export to add products?

Might be my server needs tweeking..

A real challenge.



url: http://gunorders.com

Newbie

Posts

Joined
Sat Mar 06, 2010 7:50 pm
Location - Spring Branch, TX

Post by tintedpixel » Wed Mar 10, 2010 7:42 am

alleikis wrote:I have been building a store, total products was suppose to reach 27,000. But can't seem to get the import/export to work for more than 13,000 items. '

Adjust timing in export.php with no luck.

Are you using the import/export to add products?

Might be my server needs tweeking..

A real challenge.



url: http://gunorders.com

Have you tried somehting like this?

Code: Select all

ini_set('memory_limit', '100M');
set_time_limit(0);
or for htaccess maybe something like this..

Code: Select all

php_value memory_limit 100M
php_value upload_max_filesize 100M 
php_value post_max_size 100M
php_value max_execution_time 200
php_value max_input_time 200
also are you getting any errors?

http://www.tintedpixel.com
Web Centric Creative


User avatar
New member

Posts

Joined
Fri Sep 25, 2009 11:56 pm
Location - Denver, Colorado

Post by caspara » Thu Mar 18, 2010 9:41 am

I currently have over 40,000 downloadable products on my Website:

http://www.downloadxs.com/

I am using VPS hosting with KVCHosting (they are great thus far). Unfortunately, my site is quite slow now. In addition to the regular OpenCart processes, I do have a lot of article samples that are dynamically produced with PHP (well over 100,000) on the article product pages along with an "on the fly" watermarking script for the images. I was planning to have around 100,000 products but imagine it would take a dedicated server to accomplish this.

To attempt to decrease database size, I did hard-code many of the alike product descriptions into the product.tpl file, using IF / ELSE statements depending on the product name. Also, I have redefined how the downloads work upon purchase, so that redundant system processes are limited and only one image, for example, can be used for both the product image and downloadable product while still utilizing the encoding (renaming) feature.

After 25,000 or so products, I began having to use phpMyAdmin to populate my site. Although a bit more work than the export / import mod for OpenCart, it is very quick and relatively easy.

I will be refining my site now to run more quickly, perhaps even moving to a quad-core processor server with more RAM at my current host. Also, here is a post that I found to be helpful for the streamlining process:

http://forum.opencart.com/viewtopic.php ... ads#p40061

Chris O0

DownloadXS.com : Stock Photos & Vintage Illustrations, Fresh PLR Articles and more.
http://www.DownloadXS.com


Newbie

Posts

Joined
Fri Feb 19, 2010 2:16 am

Post by rph » Thu Mar 18, 2010 10:07 am

caspara wrote: In addition to the regular OpenCart processes, I do have a lot of article samples that are dynamically produced with PHP (well over 100,000) on the article product pages along with an "on the fly" watermarking script for the images.
There's some code in OpenCart for watermarking images already. I haven't had a chance to look through it more thoroughly so I don't know how complete it is.

-Ryan


rph
Expert Member

Posts

Joined
Fri Jan 08, 2010 5:05 am
Location - Lincoln, Nebraska

Post by datacon » Thu Mar 18, 2010 10:32 am

I have setup a store that has just over 100,000 products. Works like a dream :) We dont have many people on it at once, but for the few browsign at the same time, its fast.

Only problem with this amount of products ive incurred so far, is the OC backup/restore function in the BO, always fails, as the execution time for php is too low, even when i increase it, and the memory size, it still fails. I have to get a backup manually from the web host.

New member

Posts

Joined
Tue Nov 17, 2009 11:10 am

Post by AndyGaskell » Thu Mar 18, 2010 4:51 pm

datacon wrote:Only problem with this amount of products ive incurred so far, is the OC backup/restore function in the BO, always fails, as the execution time for php is too low, even when i increase it, and the memory size, it still fails. I have to get a backup manually from the web host.
I had this issue too, but fixed it by just adding...

Code: Select all

# add larger memory and tinfinate time to allow the backup to work
ini_set("memory_limit", "100M");
ini_set("max_execution_time", 0); # 0 = unlimited exec time 
...at the top of admin/model/tool/backup.php. Workis fine for me. This is in OC 1.4.0 by the way.

I'm a web developer at http://www.ssofb.co.uk in Aberdeen, Scotland.
Got a random blog at http://www.andygaskell.blogspot.com/.


User avatar
New member

Posts

Joined
Thu Dec 10, 2009 3:59 am
Location - Aberdeen UK

Post by datacon » Fri Mar 19, 2010 12:56 pm

AndyGaskell wrote:I had this issue too, but fixed it by just adding...

Code: Select all

# add larger memory and tinfinate time to allow the backup to work
ini_set("memory_limit", "100M");
ini_set("max_execution_time", 0); # 0 = unlimited exec time 
...at the top of admin/model/tool/backup.php. Workis fine for me. This is in OC 1.4.0 by the way.
Thanks Andy..Added that code, but still get an error..i think the DB is just too big @ 65MB :(

Fatal error: Allowed memory size of 104857600 bytes exhausted (tried to allocate 44 bytes) in C:\Inetpub\httpdocs\system\database\mysql.php on line 29

New member

Posts

Joined
Tue Nov 17, 2009 11:10 am
Who is online

Users browsing this forum: No registered users and 19 guests