Comment by Sébastien Portebois on "Any suggestions for a good logging solution?"

Have you already looked at fluentd or Graylog ? If you already looked at these, could you tell why they don't fit your needs and what else you're looking for? (Graylog as alerts and threshold and many many features) If I didn't get the question correctly and you're more interested in metrics rather than text logs, Prometheus is the open-source tool to look into. (Icinga2 is also worth being mentioned, as well as the InfluxDB/TICK stack which is really great for metrics!)

There are many log services out there, but if you want to keep control on everything in your stack, they would be my first guesses. I never (yet) had the opportunity to test fluentd for real, but it's a Linux Foundation/Cloud Native Computing Foundation project and looks quite solid. Graylog is really nice (or your own ELK stack, but might might require more ops work?)

We tried graylog for a month and it actually became a bottleneck in the system. Besides needing plenty of work to scale, it also requires a lot of work before you can get any real data out of the logs. Graylog itself is also struggling to handle even small traffic spikes, so unless we throw a lot of hardware at it, we will lose logs

Was looking at fluentd yesterday, I can't remember if it can be self-hosted, but since it has components written in Ruby and we'll likely be running it on older kernels on some of the servers, we'll be hitting the deadlock issue mentioned in their FAQ section - adding Ruby to this already complicated stack makes things even harder to maintain.

I have a call with Loggly and Splunk later today, will check out the other ones you mentioned, thanks :-)

How I thought you didn't want your data to go out of your stack. If it's not the case, Loggly do scale nicely indeed.... at least from a tech point of view. We use it and it's nice but the bill is one of our pain point. Another small issue I have with them (but should be fixed in Q2 this year) is that they don't encrypt anything at rest, and they provide S3 archival (automatic push of the logs to a S3 bucket you set), but they don't offer to set the header during these PUT operations to encrypt data in the bucket (it's so cheap!) I worked around this with S3 notifications+lambda to encrypt the data just after they've been added, but that would be so much better if they just could set a simple http header in their requests :-/

Prices might not be accurate (they are a few weeks old), but we looked if we could use something else than Loggly (we currently have a 15 days retention, and a 20Gb/day quota) (I don't have our Loggly price at hand, got these from Slack history, but it was in the 1500-2000$ range IIRC) (we're just starting business, so these numbers are way bellow your terabytes, but on the other hand we don't have the revenues to generate terabytes of logs :-/

25GB/day is $1575/mo on logz.io
and $1299/mo on logentries.io
and $1420 on papertrail
and $1499 on logmatic.io
and > $2000 on LogDNA
and > $2000 on Splunk Cloud

I don'T know at your scale, but if you could afford your logs to go to a third party, then these are all potential solutions you might like to look into.
Loggly derived fields are really nice, and overall it's a great tool.

Search Hashnode