Log files you create are only as good as the information you can glean from them. Also, the way you log can change pretty dramatically depending on the size of your application, application/system/business requirements, decentralization, and many other factors.
Log Management
If you were going to be logging within a small non-distributed, one-off application I'd probably suggest the route of logging directly to a file on your server. Otherwise, I'd suggest the use of a log management solution like Loggly, or even rolling your own logging layer with Fluentd, along with other tools.
Honestly Loggly and Fluentd are just the tip of the iceberg here. There are a plethora of tools one could use to listen for, normalize, manage, and analyze logs.
Another good, full-featured tool to look at is https://github.com/Graylog2/graylog2-server. It's another open-sourced alternative that is easy to deploy, manage and use.
What To Log
No one can tell you how, or what you should log within your own application. The best anyone can do is guide you in the right direction, purely because of application/system/business requirements.
Sit down and think about the information you're processing in your application, if something goes wrong, what would you need in-order to figure out what went wrong? It will probably take some time, but you'll figure out where your sweet spot is for logging.
Just make sure you don't log any sensitive data. That's just asking for trouble later down the road.
Articles Worth Reading
Logs are so, so, so useful. Imagine a complex task needing to run at the server, and it fails for some reason. The user can't tell you what they did because often times, they don't know. So, you log stuff. Logging can be as simple as "trace 1", "trace 2", "trace 3", etc. after each line of code where you need to track down a bug. You see "trace 1" in the log, and "trace 2", but "trace 3" is missing. Why? Narrows things down for you. That's just one use.
One key area I have logging is in database activity. I have a single database class in my PHP application that wraps database calls so I can have all db activity filter through there. I wrap all database calls in a try/catch, and the catch will do two things: (1) log the database failure, the SQL call that was used and the array of parameters sent to the SQL call so I can reproduce the exact SQL that failed, and (2) send me a PushBullet message using their API, telling me there was an error along with a backtrace. (Plug for PushBullet here... great use case, IMO).
I use logging extensively. I would not write an application without it. I wrote my own logging system as I feel others, like log4j, are too bloated for my needs. And since my application is in PHP, if there's a production issue, I can just add some logging to the production code (yes, bad practice, but I do it anyway) to see what the failures are there that I can't reproduce in a dev environment.
The logger gives you the ability to define different levels of importance of the logged messages and the ability to use different sink for the output - the console, a file, etc.
Also it's easy to enable or disable only some type of message when using a logger - for example you don't want to see every debug message in production.
We can also use logger for profiling i.e; it can become a good tool to profile sections of a program, for instance by logging the start and end of an operation.
Creating a good log file:
Dana Ross
Building the web since 1996. Full-stack developer, feral & abused cat socializer, tech history buff. Director of Engineering at 10up.
There are different kinds of logs you may want/need for your application. Your language runtime and operating system are probably already creating error logs for issues encountered in the code or runtime environment – things like referencing a PHP variable that was never defined. Your software may want to log additional things to the system error log or a separate log when it recognizes an invalid state, like bad or missing data.
Other logs are informational, and track activity for the purpose of debugging or statistics. Your web server's access log is a great example. You can track activity, memory usage, third-party API request timings or timeouts…anything that would help your developers improve the software or help any other stakeholders make informed decisions about how it's used.
Another important type of log is an audit log, sometimes called a compliance log. Some contracts or regulations may call for detailed logging of certain activities, such as logins and credit card transactions. Your stakeholders may want to track admin activity so they know who's responsible if someone does something they're not supposed to.
One important thing to keep in mind when logging – especially compliance logs: don't include personally identifying information (PII) in logs, because your log files probably aren't stored with the same security standards as the rest of your data.
Logging doesn't come free. It's an I/O operation. So, don't write more information than you need, and regularly review what your software is logging to make sure it's still relevant. You may have started logging something in response to an incident five years ago, and that information really isn't relevant anymore.
If your software is running on multiple servers, consider aggregating your logs into a central store like Splunk that lets you view and evaluate activity across all your hardware.