HowTo/InstallGrayLog2

graylog2

Installation pre-requis

L'installation se fera a partir d'une RHEL/CentOS 6.2 minimal. EPEL

Installer EPEL yum -y install libyaml java-1.6.0-openjdk erlang screen telnet rubygems openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel httpd-devel vim ruby-devel RabbitMQ?

yum -y install http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server-2.8.1-1.noarch.rpm service rabbitmq-server start chkconfig rabbitmq-server on [optional, only if server won't start] edit vi /etc/init.d/rabbitmq-server ...

       #$CONTROL wait $PID_FILE >/dev/null 2>&1
       #RETVAL=$?
       RETVAL=0

... iptables

iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 80 -j ACCEPT iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 5672 -j ACCEPT service iptables save Mongo

This is the database that will store the metadata. Install the repo: vim /etc/yum.repos.d/10gen.repo [10gen] name=10gen Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 yum -y install mongo-10gen mongo-10gen-server /etc/mongod.conf nohttpinterface = true auth = true port = 27017 service mongod start chkconfig mongod on Taper mongo puis: use admin db.addUser('admin', 'iopiop') db.auth('admin', 'iopiop') use graylog2 db.addUser('eadam', 'passwd') Elasticsearch

This is the distributed database that will store the messages. Download and install elasticsearch from github: cd wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz tar xzvf elasticsearch-0.18.7.tar.gz mv elasticsearch-0.18.7 /usr/local/bin/elasticsearch curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz mv *servicewrapper*/service /usr/local/bin/elasticsearch/bin/ cd /usr/local/bin/elasticsearch ./bin/service/elasticsearch install /etc/init.d/elasticsearch start chkconfig elasticsearch on Installation de graylog2 serveur+web

Server

wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz tar xzvf graylog2-server-0.9.6.tar.gz mv graylog2-server-0.9.6 /opt/graylog2 cd /opt/graylog2 cp graylog2.conf.example /etc/graylog2.conf Mettre a jour la configuration permettant la connexion a mongodb: vim /etc/graylog2.conf ...

1. MongoDB Configuration

mongodb_useauth = true mongodb_user = <user> mongodb_password = <password> mongodb_host = localhost

1. mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019

mongodb_database = graylog2 mongodb_port = 27017 ... Finalement, démarrer le serveur Copier le fichier graylog2-server attaché en bas de cette page dans /etc/init.d/ puis exécuter: chmod +x /etc/init.d/graylog2-server chkconfig --add graylog2-server service graylog2-server start Web interface

Installation

wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz tar xzvf graylog2-web-interface-0.9.6.tar.gz mv graylog2-web-interface-0.9.6 /opt/graylog2-web-interface chown apache:apache -R /opt/graylog2-web-interface chcon -R system_u:object_r:httpd_sys_content_t:s0 /opt/graylog2-web-interface vim /opt/graylog2-web-interface/config/mongoid.yml vim /opt/graylog2-web-interface/config/general.yml vim /opt/graylog2-web-interface/config/email.yml cd /opt/graylog2-web-interface gem install bundler bundle install setsebool -P httpd_can_network_connect_db on setsebool -P httpd_can_network_connect on Passenger

gem install passenger passenger-install-apache2-module vim /etc/httpd/conf.d/passenger.conf LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11 PassengerRuby /usr/bin/ruby Rajouter a la fin de /etc/httpd/conf/httpd.conf NameVirtualHost *:80

<VirtualHost *:80>

       ServerName carina4.winsoft-international.com
       DocumentRoot /opt/graylog2-web-interface/public
       <Location />
               Allow from all
               Options -MultiViews
       </Location>
       ErrorLog /var/log/httpd/graylog_error.log
       LogLevel warn
       CustomLog /var/log/httpd/graylog_access.log combined

</VirtualHost> Redémarrer apache service httpd restart logstash

logstash permet de lire les fichiers de log, de les envoyer au travers de queue (rabbitmq ici) et de les 'parser' pour les renvoyer dans graylog2 sous format GELF Une instance de logstash s'exécute sur chaque serveurs envoyant les logs (les clients), et une instance s'exécute sur le serveur centralisant les logs (le serveur). Download

mkdir /opt/logstash wget -O /opt/logstash/logstash.jar http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar mkdir /etc/logstash Sur les serveurs 'clients'

vim /etc/logstash/shipper.conf input {

  file {
     type => "syslog"
     path => [ "/var/log/messages", "/var/log/syslog" ]
  }
  file {
     type => "apache-access"
     path => "/var/log/apache2/access*.log"
  }
  file {
     type => "apache-error"
     path => "/var/log/apache2/error*.log"
  }

} output {

  # Ship events to the amqp fanout exchange named 'rawlogs"
  amqp {
     host => "carina4.winsoft-international.com"
     port => "5672"
     #debug => "true"
     exchange_type => "fanout"
     name => "rawlogs"
  }

} Puis exécuter l'agent dans un screen: cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/shipper.conf Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-shipper'. Sur le serveur de log

vim /etc/logstash/reader.conf input {

       amqp {
               type => "all"
               host => "127.0.0.1"
               exchange => "rawlogs"
               name => "rawlogs_consumer"
       }

} filter {

       grok {
               type => "syslog"
               pattern => "%{SYSLOGLINE}"
       }
       grok {
               type => "apache-access"
               pattern => "%{COMBINEDAPACHELOG}"
       }
       date {
               type => "syslog"
               timestamp => "MMM d HH:mm:ss"
               timestamp => "MMM dd HH:mm:ss"
               timestamp8601 => ISO8601
       }
       date {
               type => "apache-access"
               timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
       }

} output {

       gelf {
     host => "localhost"
     facility => "%{@type}"
  }

} Puis exécuter l'agent dans un screen: cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/reader.conf Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-reader'. UI

If everything went well, you can connect to the web interface of graylog2 through the normal http port 80. Some analytics commands

Get the sorted distinct count of 404 requests all.distribution({_request}, _response=404)