Ok, I think I figured the biatch.
This is my testing scenario
Product price is 13.984. Currency is set to be displayed with 3 decimal digits.
Product price shows as 13.984, I add 3 products and have
13.984 * 3 = 41.952 in product row
With original code, I get PP errors about invalid amounts.
After rounding L_AMT0, SHIPPINGAMT, TAXAMT, AMT to 2 digits, I get to PP login page.
Totals shown there are off by 0.012, as expected
So, it's now completely obvious - if we use more then 2 decimal places for prices, this is inevitable, because there will always be this error in rounding.
The only solution is to remember the error after rounding and then correct the final price using shipping or tax or whatever.
13.984 rounds to 13.98, we remember +0.004 (+ is indicating we have to add to final price because there are situations when it will round to higher number (13.987 will round to 13.99), and then we have to remember -0.003 and deduct it later in order not to rip off customer).
Then we have 13.98 * 3 = 41.94 as the item total and 3 * +0.004 = +0.012 remainder
We then round the remainder to two digits so +0.012 becomes +0.01.
Finally we add 0.01 as shipping or tax for correction, thus total will be 41.94 + 0.01 = 41.95.
There is still an error when rouding remainder, but that error is limited to 10th parts of a cent, which is tolerable.