Right now we haven't happily run such a complete solution for our current use-case: containers running in ECS. Relying extensively on AWS, our current plan is to wait for the updated AMI with Docker 1.9 that feature the Cloudwatch Logs connection to the docker logs.
Since we don't really require right now since we're soft-launching our product, we estimated that we'd better spend our energy on more meaningful topics, rather than adding another layer of complexity to get the logs from Docker and push it to Cloudwatch... not that complex, but another agent or container, another thing to maintain, ... Another solution would have been to use our own AMI, but so far we used the ECS optimized AMIs provided by Amazon and were happy. Doing our own AMI would be easy (we tested the Docker 1.9 to Cloudwatch Logs flow succesfully), but it implies that we maintain that new AMI... we could do it, but with the updated AMI coming in a few weeks (maybe before), the updated freedom for a few weeks would have come with the burden of maintaining this after the gain would have been lost...
Once the logs are in Cloudwatch Logs (to collect the many sources), we investigated a few options, using lambdas and/or Kinesis to then push logs to ElasticSearch (ELK) and/or Redshift (depending on the source and use of the logs)
I would love to hear more from @JanVladimirMostert about the RabbitMQ (and consumers) part. In our case we would have either system logs (like nginx access and error logs) and custom logs on which we'd like to do complex queries (hence the Redshift option)
Log4J (for Java) and Log4net (for .NET a fork of Log4j).
Both are easy to use and customize, besides having integration with other libraries like Autofac (an IoC lib for .NET).
Although I haven't used any logging tools, I heard for colleagues very positive reviews about Splunk and Loggly. Both have rich features list. Splunk is more enterprise oriented, while Loggly seems more flexible.
I don't really use "3d party logging tools" - Java I use SL4j and Log4j (that's pretty much the standard for Java apps these days), I often configure log4j to forward a copy of all logs to rabbitMQ - a special app then consumes all applications' logs from RabbitMQ and either write it to standard out or to a database in such a way that I can query the logs. Further I make sure to pipe all logs to the front so that Docker can pick them up; when using Container Engine on Google Cloud, Google's Monitoring then picks up these logs. Logs via RabbitMQ is the most flexible, it means zero lock-in and I can design my own alerts and UI for viewing and filtering logs.
Denny Trebbin
Lead Fullstack Developer. Experimenting with bleeding-edge tech. Irregularly DJ. Hobby drone pilot. Amateur photographer.
Using InfluxDB to send log events. But InfluxDB has changed dramatically just recently. Thinking of using prometheus.io instead or another use case for RethinkDB :-)
I don't really use a 3rd party logging framework. Just HTTP post.