Log All The Things

Nothing can possibly go wrong.

JK. Something will go wrong. You will want to capture that somehow.

So log all the things!

The Broker: Logstah

Gathers, parses and sends logs off for indexing.

elasticsearch.org/overview/logstash/

The Indexer: Elasticsearch

Lucene powered document store.

logstash.net/

The Shipper: Logstash Forwarder

Ships logs off to the Logstash.

elasticsearch/logstash-forwarder

UI: Kibana

Interface to search logs.

elasticsearch.org/overview/kibana/

Bonus Cool: Ansible

Configuration management for making all the above as painless as possible to deploy.

ansible.com/home

Deploying Elasticsearch

Logstash without Elasticsearch is like me without pizza. It's unnatural.

The Playbook

Deploying Logstash

The Playbook

Configuring Logstash

Logstash can be configured to accept all kinds of inputs, and even filter that data before passing it along.

/files/conf.d

Checking Elasticsearch Health

Elasticsearch exposes a nice restful interface, so checking health is easy.

/files/conf.d

                            $ curl localhost:9200/_cat/health
                        

                            $ 1413915720 14:22:00 elasticsearch red 1 1 19 19 0 4 37
                        

Sad Logging Server :(

Looking At Indexes

                            $ curl localhost:9200/_cat/indices
                        

We can use the above to checkout how big our indexes have grown.

                            yellow logstash-2014.10.16   5   1   92115607            0     28.1gb         28.1gb
                        

Further Reading

• Elasticsearch: Exploring Your Cluster
• Logstash Docs

Deploying Logstash Forwarder

The Playbook

Configuring Logstash Forwarder

The forwarder or shipper or "gets logs to server"-er is in charge of getting logs to the server. Each service/app you want to send logs from needs this installed.

Configuration can be handled via a simple json file.

Deploying Kibana

The Playbook

Dashboards

Kibana supports several dashboards out of the box, but there is one built for logstash stored at app/dashboards/

Just rename it to default.json