HowTo/InstallGrayLog2

graylog2

Installation pre-requis

L'installation se fera sur une RHEL/CentOS 6.2 minimal.

EPEL

yum install 

Base packages

yum -y install libyaml java-1.6.0-openjdk erlang screen telnet rubygems openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel httpd-devel vim ruby-devel

RabbitMQ

yum -y install http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server-2.8.1-1.noarch.rpm
service rabbitmq-server start
chkconfig rabbitmq-server on

[optional, only if server won't start] edit vi /etc/init.d/rabbitmq-server ...

       #$CONTROL wait $PID_FILE >/dev/null 2>&1
       #RETVAL=$?
       RETVAL=0

... iptables

iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 80 -j ACCEPT
iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 5672 -j ACCEPT
service iptables save

Mongo

This is the database that will store the metadata.

Install the repo: vim /etc/yum.repos.d/10gen.repo

[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
yum -y install mongo-10gen mongo-10gen-server
/etc/mongod.conf
nohttpinterface = true
auth = true
port = 27017
service mongod start
chkconfig mongod on
Taper mongo puis:
use admin
db.addUser('admin', 'iopiop')
db.auth('admin', 'iopiop')
use graylog2
db.addUser('eadam', 'passwd')

Elasticsearch

This is the distributed database that will store the messages.

Download and install elasticsearch from github:

cd
wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz
tar xzvf elasticsearch-0.18.7.tar.gz 
mv elasticsearch-0.18.7 /usr/local/bin/elasticsearch
curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv *servicewrapper*/service /usr/local/bin/elasticsearch/bin/
cd /usr/local/bin/elasticsearch
./bin/service/elasticsearch install
/etc/init.d/elasticsearch start
chkconfig elasticsearch on

Installation de graylog2 serveur+web

Server

wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
tar xzvf graylog2-server-0.9.6.tar.gz
mv graylog2-server-0.9.6 /opt/graylog2
cd /opt/graylog2
cp graylog2.conf.example /etc/graylog2.conf

Mettre a jour la configuration permettant la connexion a mongodb:

vim /etc/graylog2.conf

...
# MongoDB Configuration
mongodb_useauth = true
mongodb_user = <user>
mongodb_password = <password>
mongodb_host = localhost
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017
...

Finalement, démarrer le serveur

Copier le fichier graylog2-server attaché en bas de cette page dans /etc/init.d/ puis exécuter:

chmod +x /etc/init.d/graylog2-server
chkconfig --add graylog2-server
service graylog2-server start

Web interface

Installation

wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
tar xzvf graylog2-web-interface-0.9.6.tar.gz
mv graylog2-web-interface-0.9.6 /opt/graylog2-web-interface
chown apache:apache -R /opt/graylog2-web-interface
chcon -R system_u:object_r:httpd_sys_content_t:s0 /opt/graylog2-web-interface
vim /opt/graylog2-web-interface/config/mongoid.yml
vim /opt/graylog2-web-interface/config/general.yml
vim /opt/graylog2-web-interface/config/email.yml
cd /opt/graylog2-web-interface
gem install bundler
bundle install
setsebool -P httpd_can_network_connect_db on
setsebool -P httpd_can_network_connect on

Passenger

gem install passenger
passenger-install-apache2-module
vim /etc/httpd/conf.d/passenger.conf
LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11
PassengerRuby /usr/bin/ruby

Rajouter a la fin de /etc/httpd/conf/httpd.conf

NameVirtualHost *:80

<VirtualHost *:80>
        ServerName carina4.winsoft-international.com
        DocumentRoot /opt/graylog2-web-interface/public
        <Location />
                Allow from all
                Options -MultiViews
        </Location>
        ErrorLog /var/log/httpd/graylog_error.log
        LogLevel warn
        CustomLog /var/log/httpd/graylog_access.log combined
</VirtualHost>

Redémarrer apache

service httpd restart

logstash

logstash permet de lire les fichiers de log, de les envoyer au travers de queue (rabbitmq ici) et de les 'parser' pour les renvoyer dans graylog2 sous format GELF Une instance de logstash s'exécute sur chaque serveurs envoyant les logs (les clients), et une instance s'exécute sur le serveur centralisant les logs (le serveur). Download

mkdir /opt/logstash wget -O /opt/logstash/logstash.jar http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar mkdir /etc/logstash Sur les serveurs 'clients'

vim /etc/logstash/shipper.conf input {

  file {
     type => "syslog"
     path => [ "/var/log/messages", "/var/log/syslog" ]
  }
  file {
     type => "apache-access"
     path => "/var/log/apache2/access*.log"
  }
  file {
     type => "apache-error"
     path => "/var/log/apache2/error*.log"
  }

} output {

  # Ship events to the amqp fanout exchange named 'rawlogs"
  amqp {
     host => "carina4.winsoft-international.com"
     port => "5672"
     #debug => "true"
     exchange_type => "fanout"
     name => "rawlogs"
  }

} Puis exécuter l'agent dans un screen: cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/shipper.conf Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-shipper'. Sur le serveur de log

vim /etc/logstash/reader.conf input {

       amqp {
               type => "all"
               host => "127.0.0.1"
               exchange => "rawlogs"
               name => "rawlogs_consumer"
       }

} filter {

       grok {
               type => "syslog"
               pattern => "%{SYSLOGLINE}"
       }
       grok {
               type => "apache-access"
               pattern => "%{COMBINEDAPACHELOG}"
       }
       date {
               type => "syslog"
               timestamp => "MMM d HH:mm:ss"
               timestamp => "MMM dd HH:mm:ss"
               timestamp8601 => ISO8601
       }
       date {
               type => "apache-access"
               timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
       }

} output {

       gelf {
     host => "localhost"
     facility => "%{@type}"
  }

} Puis exécuter l'agent dans un screen: cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/reader.conf Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-reader'. UI

If everything went well, you can connect to the web interface of graylog2 through the normal http port 80. Some analytics commands

Get the sorted distinct count of 404 requests all.distribution({_request}, _response=404)