I was finally able to get callback from paypal.
I changed only two things - disabled password protection on directory and enabled SSL option in the admin.
Also - I continued playing with separating order placing from paypal callback.
From confirmation page - opencart goes directly to paypal. After paypal processes payment - it sends callback.
In the original implemantation - callback is what triggers order processing. So - order is created, cart is cleared, success pages is displayed.
Now, if order processing and callback is separated (by copying cod process function to paypal) - then process function is trigerred after user clicks "return to merchant" from paypal.
While callback function is trigerred when paypal calls back.
So - if user delays returning to the page - process can be called after callback.
Logically - you would want to create order first with "pending" status and then in callback - update it to processing. If callback never comes - you can just update status to procesing manually through admin.
So we need a function that creates and inserts order, and another - small one - which just updates order status.
order->process function already exists. It creates order and clears it from order_data table. However, order_data stores all working knowledge about the order - so we need to keep it until callback (I think). Just have to comment out that part of the $order->process function - or make it conditional.
so only simple function somewhat like this is needed (to be trigerred from paypal callback):
function process_paid($order_status_id){
if ($this->data) {
//get order id
$sql = "select order_id from `order` where reference = '?'";
$order_id = $this->database->getRow($this->database->parse($sql, $this->reference));
$sql = "update `order` set order_status_id = '?' where order_id = '?'";
$this->database->query($this->database->parse($sql, $order_status_id, $order_id));
$sql = "update order_history set order_status_id = '?' where order_id = '?'";
$this->database->query($this->database->parse($sql, $order_status_id, $order_id));
//kill order_data
$sql = "delete from order_data where reference = '?'";
$this->database->query($this->database->parse($sql, $this->reference));
}
}
However - if callback comes - and user did not click "return" button - then callback tries to update order which does not exist yet.
Logical solution would be to create order just before going to paypal - with "pending" (or "not paid") status. Then you can be sure that it exists no matter if user clicks "return" or not. So callback can always find it.
If user never pays - you can just cancel the order manually. Its better than not getting order at all.
Then problem remains - to clear order_data if callback never comes. I guess it could be done when manually changing order status.