You are right, logging is the minimum we can do to trace an error or simply check an application runs well.
For example, we log critical transactions' results regarding payments, even the successful ones. We also have home-made logs systems and admin panels so the customer can check on their own what transaction was successful or not.
Those logs contain unique numbers in the form "123-1234" to be easily memorized and looked up. Our customers will call us with this number and we can double check and investigate more quickly.
For such transactions, we log everything, even the data we send (POST data in general).
This could help identify a malfunction on our side or on the third-party side.
We, of course, log the received data "as is" with no treatments, plus a parsed version.
Because it's easier to read, or it could help identifying if our parser failed on something...
Lastly, data retention. It depends on how much activity you have.
The last system we made, each log file was monthly rotated, because we don't expect much trafic, but daily logs could be necessary with heavy trafic.
We archive those logs externally after one or 2 months. Keep about one year of logs.
That's our basics.
On top of that, we rely on MySQL logs and Apache logs as well.