Centralised Logging

Hi All,

I'm in the process of moving hosts for my OpenCart store from shared hosting over to Amazon Web Services. The intention is that I'll have multiple application servers (auto scaling in time), a single database server and using an S3 bucket to host all of the images.

I've successfully written the mechanism that handles the S3 bucket, so images and downloads all get stored and served from a single S3 bucket, (including the resized image files). I've also written a module so that the site is using Redis caching so I can just plug that into the Amazon ElastiCache when I go live. The bucket list is also cached and a Lambda function in place to handle external new files which drops the speed of serving up a page considerably.

The last bit to sort is some form of centralised log server and I'm at a bit of a loss - I've more ideas than firm solutions currently.

Although it's really powerful, the Amazon Cloud Watch service appears more suited to system errors and the viewing interface isn't that friendly. Either that or I've missed something!

I could modify log->write to pump all messages into a database table (or even separate database), my concern is if there's a database error then nothing will be logged.

I could move the log into an S3 bucket file. The downside of this is that you can't "lock" an S3 file, so would need to either log each message singularly or risk loosing messages if they're written at the same time, plus S3 PUT's cost money.

So has anybody else come across an alternative centralised logging methods or any thoughts on what could work?

Pete

HI, i am doing this (as you did), and i'm focusing to centralize all my log's on one external site called Loggly.com and it seems pretty accurate of what are we looking for. also they run all their structure with AWS so it will easy to accomplish, i've had this done at the moment, but see how it goes for you!

About the Redis Module, could you tell us more about it? what's the experience against Memcache and if you can share your module or steps to accomplish this with us ?. Thanks and good luck !

Didn't realise I'd had a reply to this

I did consider Loggly and other similar logging services, but we didn't really want any additional expense or to push the logs off onto a third party as there could be occasions when we might log customer data or similar and then would have issues with Data Protection or if credit card details were logged then we'd have PCI compliance issues. My solution was actually to pump the logs into a new database table and mod the log page, it's pretty crude in principal but works really well, plus it filters based on each log "file name" so really easy to jump between payment, SQL and error logs (plus import).

Redis has a small edge over Memcache in a few areas when I looked at it, it's slightly better at handling large caches and long keys and has a more object orientated api. It also offers a decent timeout method rather than simply filling up the memory and garbage collection. I've attached the mod I created for Redis caching and a few other speed improvements*. It is more work to set up if you're planning on running Redis your own server, however I chose to use the AWS Redis cache which comes preconfigured and optimised for running Redis.


* Other improvements include, adding gzcompress as the page compression method, full Redis implementation (including settings in the backend), caching urls, caching getTotalProducts result, cache getProductLayoutId, cache getCategory result, cache getCategories result, cache language id's to code (including getLanguage response), cache the tax rates, cache route layouts, cache getExtensions result

Overall the improvements reduces the number of database queries that OpenCart performs on each page query. Your mileage may vary however but it should be pretty obvious if you want to drop any out.

PeteA wrote:Didn't realise I'd had a reply to this

I did consider Loggly and other similar logging services, but we didn't really want any additional expense or to push the logs off onto a third party as there could be occasions when we might log customer data or similar and then would have issues with Data Protection or if credit card details were logged then we'd have PCI compliance issues. My solution was actually to pump the logs into a new database table and mod the log page, it's pretty crude in principal but works really well, plus it filters based on each log "file name" so really easy to jump between payment, SQL and error logs (plus import).

Redis has a small edge over Memcache in a few areas when I looked at it, it's slightly better at handling large caches and long keys and has a more object orientated api. It also offers a decent timeout method rather than simply filling up the memory and garbage collection. I've attached the mod I created for Redis caching and a few other speed improvements*. It is more work to set up if you're planning on running Redis your own server, however I chose to use the AWS Redis cache which comes preconfigured and optimised for running Redis.


* Other improvements include, adding gzcompress as the page compression method, full Redis implementation (including settings in the backend), caching urls, caching getTotalProducts result, cache getProductLayoutId, cache getCategory result, cache getCategories result, cache language id's to code (including getLanguage response), cache the tax rates, cache route layouts, cache getExtensions result

Overall the improvements reduces the number of database queries that OpenCart performs on each page query. Your mileage may vary however but it should be pretty obvious if you want to drop any out.

HI Pete, Id didn't know either when you replied. sorry for the long delay and thanks for the script.
Is this xml file still working for the updated oc v currently or you have made some modifications already?.

I assume as well need to be localized into the xml folder on vqmod and what else do I need to know beside to create cluster on aws and connect it to ec2 ?.

Kind regards.