We're looking for a logging solution that will allow us to log
We have to be able to host it ourselves and it must be able to scale relatively well - we're looking at 50 - 100 servers in the future that will be pumping huge volumes of logs to this log solution.
A further bonus would be if it has some intelligence built in which can give us some information and alerts based on the received logs.
Nothing stops me from building this myself, but I have so many other things that I need to build as well and logging is very far down that list - especially if a solution already exists. Any suggestsions?
Maybe you can take a look to the ELK (Elastic, Logstash and Kibana), I think it's the most common solution for logging all types of information without headache. I don't have a lot of experience but other companies used.
You may google "Logging as a service", it's 2017 ftw, there is X as a service today for everything. However, I have no idea why you even need it?
What is a problem with logging system your app and web server gives you, e.g. just default log files? For app logs in PHP I use Laravel's event and exception system, whenever I want to know something critical, like failed admin logins, delete operations executed from admin panel, any app exceptions and errors I instantly receive an email. I suppose it's easy to configure and set up in any app/framework you use depending on your stack. Usually I also have a Logs section in admin panel with a simple datatable UI, it's not harder to implement either.
Talking about the server logs I have nginx default error logs and access logs. Whenever I need to get some info from those files I just write a simple script. You even may parse those files from app itself and display all the results you need in the UI you want.
There is also nothing to scale, every server/instance should store logs in files. nginx #1 has own logs, nginx #2 owns, etc. Usually those simple log files are gzipped time-to-time and old ones removed with a simple script/cron.
Fact is it's something simple and specific logic always will depend on your specific use-cases, that's why I see no other option but writing own bash scripts/cron tasks/app scripts on the top of the default log files yourself
Sébastien Portebois
Software architect at Ubisoft
Have you already looked at fluentd or Graylog ? If you already looked at these, could you tell why they don't fit your needs and what else you're looking for? (Graylog as alerts and threshold and many many features) If I didn't get the question correctly and you're more interested in metrics rather than text logs, Prometheus is the open-source tool to look into. (Icinga2 is also worth being mentioned, as well as the InfluxDB/TICK stack which is really great for metrics!)
There are many log services out there, but if you want to keep control on everything in your stack, they would be my first guesses. I never (yet) had the opportunity to test fluentd for real, but it's a Linux Foundation/Cloud Native Computing Foundation project and looks quite solid. Graylog is really nice (or your own ELK stack, but might might require more ops work?)