Log All The Things

So you have a shiny new server architecture? You have a bunch of new apps and services? All humming together in perfect harmony?

Nothing can possibly go wrong.

JK. Something will go wrong. You will want to capture that somehow.

So log all the things!

The Players

There are a few key pieces of technology at play. The shipper, broker, and indexer. Of course, we will also want some sort of user interface into this madness.

The Broker: Logstah

Gathers, parses and sends logs off for indexing.

elasticsearch.org/overview/logstash/

The Indexer: Elasticsearch

Lucene powered document store.

logstash.net/

The Shipper: Logstash Forwarder

Ships logs off to the Logstash.

elasticsearch/logstash-forwarder

UI: Kibana

Interface to search logs.

elasticsearch.org/overview/kibana/

Bonus Cool: Ansible

Configuration management for making all the above as painless as possible to deploy.

ansible.com/home

Elasticsearch and Logstash

The heroes of our story.

Deploying Elasticsearch

Logstash without Elasticsearch is like me without pizza. It's unnatural.

The Playbook

Deploying Logstash

The Playbook

Configuring Logstash

Logstash can be configured to accept all kinds of inputs, and even filter that data before passing it along.

/files/conf.d

Checking Elasticsearch Health

Elasticsearch exposes a nice restful interface, so checking health is easy.

/files/conf.d

                            
$ curl localhost:9200/_cat/health
                            
                        

                            
$ 1413915720 14:22:00 elasticsearch red 1 1 19 19 0 4 37
                            
                        

Sad Logging Server :(

Looking At Indexes

                            
$ curl localhost:9200/_cat/indices
                            
                        

We can use the above to checkout how big our indexes have grown.

                            
yellow logstash-2014.10.16   5   1   92115607            0     28.1gb         28.1gb                                
                            
                        

Further Reading

• Elasticsearch: Exploring Your Cluster
• Logstash Docs

Logstash Forwarder

♫ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! ♫

https://github.com/elasticsearch/logstash-forwarder

Deploying Logstash Forwarder

The Playbook

Configuring Logstash Forwarder

The forwarder or shipper or "gets logs to server"-er is in charge of getting logs to the server. Each service/app you want to send logs from needs this installed.

Configuration can be handled via a simple json file.

Kibana

Kibana is our view into our logs. It gives us a nice HTML UI for searching and viewing logs.

Deploying Kibana

The Playbook

Dashboards

Kibana supports several dashboards out of the box, but there is one built for logstash stored at app/dashboards/

Just rename it to default.json