git submodule deinit redisio
git submodule deinit vim
git submodule deinit windows
git submodule deinit perl
git submodule deinit hostsfile
git submodule deinit dmg
git submodule deinit mysql
git submodule deinit ant

git rm redisio
git rm vim
git rm windows
git rm perl
git rm hostsfile
git rm dmg
git rm mysql
git rm ant


git submodule rm redisio
git rm redisio
git rm vim
git rm windows
git rm perl
git rm hostsfile
git rm dmg
git rm mysql
git rm ant
git submodule deinit postfix
git rm postfix
git submodule deinit php
git rm php
git rm nagios
git submodule deinit nagios
git submodule deinit xfs
git submodule rm memcached
git rm memcached
git rm apache2
git rm java
git rm -f java
git rm xml
git rm ntp
cd database
git rm database
git rm build-essential
git rm erlang

A complete Logstash stack on AWS OpsWorks

UPDATE: I wrote a blog post that will help you set this up.

This is a bunch of cookbooks that will allow you to run a complete Logstash setup on a scalable AWS OpsWorks stack. At Springest we use it to ship from 250 to about 1k log entries per second, depending on the RPM on our 18 servers.

A screenshot of Kibana3, included in the cookbooks.

The stack's layers

A screenshot of the layers in AWS OpsWorks

• A RabbitMQ cluster layer (Ubuntu) – Agents wil ship their logs to this cluster and the Logstash server cluster uses it as an input source.
• An ElasticSearch cluster layer (Amazon Linux) – All log messages are stored and indexed here.
• A LogStash cluster layer (Ubuntu) – Takes the messages from the RabbitMQ fanout and puts them into ElasticSearch.
• A Kibana server (Ubuntu) - An Angular.js interface on top of ElasticSearch to search, graph etc.

Setting up your stack

• Set git://github.com/Springest/opsworks-logstash.git as a repository URL for your custom cookbooks.
• Use the following Chef custom JSON:

{
    "chef_environment": "production",
    "elasticsearch": {
        "cluster": {
            "name": "logstash"
        },
        "basic_auth": {
            "user": "<es username here>",
            "password": "<es password here>"
        }
    },
    "rabbitmq": {
        "cluster": true,
        "erlang_cookie": "<any random alphanumeric string will do>"
    },
    "rabbitmq_cluster": {
        "user": "<rabbitmq username here>",
        "password": "<rabbitmq password here>"
    },
    "kibana": {
        "webserver": "nginx",
        "webserver_hostname": "<your kibana domain, eg: kibana.example.com>",
        "web_user": "<kibana username here>",
        "web_password": "<kibana password here>",
        "es_port": "9200",
        "es_role": "elasticsearch",
        "es_server": "<es loadbalancer address here>",
        "es_user": "<es username here>",
        "es_password": "<es password here>"
    },
    "logstash": {
        "elasticsearch_cluster": "logstash",
        "server": {
            "install_rabbitmq": false,
            "enable_embedded_es": false,
            "elasticsearch_role": "elasticsearch",
            "inputs": [
                {
                    "rabbitmq": {
                        "exchange": "rawlogs",
                        "user": "<rabbitmq username here>",
                        "password": "<rabbitmq password here>",
                        "host": "<IP or LB address of your rabbitmq server/cluster>",
                        "type": "all"
                    }
                }
            ]
        }
    }
}

Configure Custom Chef recipes

ElasticSearch

Kibana

Logstash

RabbitMQ

Contribute

• Fork the project
• Create a feature branch
• Fix the code
• Create a pull request

Thanks to

All of these people have made our lives a little better by writing great software.

The contributors at these projects:

• ElasticSearch for their awesome search engine.
• Kibana for a beautiful graphing tool.
• LogStash for bringing our logs together.
• RabbitMQ for a robust message broker.

And the people who wrote these cookbooks:

• @lusis for his work on the Kibana cookbook and the LogStash cookbook
• @karmi for the original, and @ehlertij, @tkawa, and @interu for their efforts on the ElasticSearch cookbook for OpsWorks.
• @brianbianco for his Redis cookbook
• @sethvargo for the hostsfile cookbook
• @ddux for the rbenv cookbook
• @bryanwb for the Yumrepo cookbook
• @guilhem for his efforts on the Htpaswd cookbook

And the guys at Opscode and their contributors for Chef and cookbooks like:

• RabbitMQ
• Java
• Apt
• Erlang
• Nginx
• NTP
• Runit
• Ant
• Apache2
• Ark
• AWS
• Build-essential
• Chef handler
• Git
• Ohai
• OpenSSL
• Perl
• Python
• Windows
• xfs
• xml
• yum
• rsyslog

And last but not least:

• Amazon AWS