Security common sense.

If you are a startup its unlikely some dedicated hacker will try to hack your box

1. File permissions. Directories 740 files 640 - writeable directories like cache images 770. Chown root:www-data.

This way www data user can only read files.

2. Using Ossec to report on any files modifications in html folder. Many host api allow to send shut down command. Perhaps create active Ossec rule to do it? How?

3. Firewall. close all incoming and outgoing apart incoming 80 and 443 and rest as per need. Redirect all traffic to ssl.

4. Set auto update for security rules + enable reboot with time frame now.

5. Using Modsecurity or similar to null SQL injections.

6. Ask module supplier how he security audits his code.

7. Store full logs remotely. Ideally incrementally copied as per every change.

8. Secure nginx and php config. Disable risky php functions.

9. Back up site files and db daily.

In case if all security failed - identify which module caused issue. Apply patch or virtual patch with Modsecurity. Reinstall OS, Opencart, then verify via diff all modules files versus original one, if there are only your changes, upload them from backup. Restore database. Yes some latest customers may have a lost order or two. Simply process them manually and add to DB later.

Perhaps somehow can post a howto on identifying borked piece of code.

For finding SQL statements that are missing escaping and therefore could be a risk depending from where the data came from. You could use a regex search with the following.
It won't pick everything up (if the statements are written in a different format) and will find a lot of false positives (values could be already escaped), but it can be a start.

There are some modules that are made with backdoors. What would be most common things to check for?

Having said that if all public vulnerabilities are patched exploit wont be able escalate to root unless its a private exploit

In fact.

SQL Injection, XSS and Path Traversal.

It seems most open carts running on up to date linux are doing fine even without Modsecurity?
As it takes time to configure it properly for XSS and even then.
Even if badly configured code allowing to traverse and display config.php set sql access to local host only. And set an email alert that will instantly notify when someone read config.php file

And yes if site is large different measures are required to prevent some kernel exploits and likes.