HowTo/InstallGrayLog2
Contents
graylog2
Installation pre-requis
L'installation se fera sur une RHEL/CentOS 6.2 minimal.
EPEL
yum install -y http://mirrors.ircam.fr/pub/fedora/epel/6/x86_64/epel-release-6-7.noarch.rpm
Base packages
yum -y install libyaml java-1.6.0-openjdk erlang screen telnet rubygems openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel httpd-devel vim ruby-devel
RabbitMQ
yum -y install http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server-2.8.1-1.noarch.rpm service rabbitmq-server start chkconfig rabbitmq-server on
[optional, only if server won't start] edit vi /etc/init.d/rabbitmq-server ...
#$CONTROL wait $PID_FILE >/dev/null 2>&1
#RETVAL=$?
RETVAL=0
... iptables
iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 80 -j ACCEPT iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 5672 -j ACCEPT service iptables save
Mongo
This is the database that will store the metadata.
Install the repo: vim /etc/yum.repos.d/10gen.repo
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
yum -y install mongo-10gen mongo-10gen-server
/etc/mongod.conf
nohttpinterface = true
auth = true
port = 27017
service mongod start
chkconfig mongod on
Taper mongo puis:
use admin
db.addUser('admin', 'iopiop')
db.auth('admin', 'iopiop')
use graylog2
db.addUser('eadam', 'passwd')
Elasticsearch
This is the distributed database that will store the messages.
Download and install elasticsearch from github:
cd wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz tar xzvf elasticsearch-0.18.7.tar.gz mv elasticsearch-0.18.7 /usr/local/bin/elasticsearch curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz mv *servicewrapper*/service /usr/local/bin/elasticsearch/bin/ cd /usr/local/bin/elasticsearch ./bin/service/elasticsearch install /etc/init.d/elasticsearch start chkconfig elasticsearch on
Installation de graylog2 serveur+web
Server
wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz tar xzvf graylog2-server-0.9.6.tar.gz mv graylog2-server-0.9.6 /opt/graylog2 cd /opt/graylog2 cp graylog2.conf.example /etc/graylog2.conf
Mettre a jour la configuration permettant la connexion a mongodb:
vim /etc/graylog2.conf
... # MongoDB Configuration mongodb_useauth = true mongodb_user = <user> mongodb_password = <password> mongodb_host = localhost #mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019 mongodb_database = graylog2 mongodb_port = 27017 ...
Finalement, démarrer le serveur
Copier le fichier graylog2-server attaché en bas de cette page dans /etc/init.d/ puis exécuter:
chmod +x /etc/init.d/graylog2-server chkconfig --add graylog2-server service graylog2-server start
Web interface
Installation
wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz tar xzvf graylog2-web-interface-0.9.6.tar.gz mv graylog2-web-interface-0.9.6 /opt/graylog2-web-interface chown apache:apache -R /opt/graylog2-web-interface chcon -R system_u:object_r:httpd_sys_content_t:s0 /opt/graylog2-web-interface vim /opt/graylog2-web-interface/config/mongoid.yml vim /opt/graylog2-web-interface/config/general.yml vim /opt/graylog2-web-interface/config/email.yml cd /opt/graylog2-web-interface gem install bundler bundle install setsebool -P httpd_can_network_connect_db on setsebool -P httpd_can_network_connect on
Passenger
gem install passenger passenger-install-apache2-module vim /etc/httpd/conf.d/passenger.conf LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11 PassengerRuby /usr/bin/ruby
Rajouter a la fin de /etc/httpd/conf/httpd.conf
NameVirtualHost *:80
<VirtualHost *:80>
ServerName carina4.winsoft-international.com
DocumentRoot /opt/graylog2-web-interface/public
<Location />
Allow from all
Options -MultiViews
</Location>
ErrorLog /var/log/httpd/graylog_error.log
LogLevel warn
CustomLog /var/log/httpd/graylog_access.log combined
</VirtualHost>
Redémarrer apache
service httpd restart
logstash
logstash permet de lire les fichiers de log, de les envoyer au travers de queue (rabbitmq ici) et de les 'parser' pour les renvoyer dans graylog2 sous format GELF Une instance de logstash s'exécute sur chaque serveurs envoyant les logs (les clients), et une instance s'exécute sur le serveur centralisant les logs (le serveur).
Download
mkdir /opt/logstash wget -O /opt/logstash/logstash.jar http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar mkdir /etc/logstash
Sur les serveurs 'clients'
vim /etc/logstash/shipper.conf
input {
file {
type => "syslog"
path => [ "/var/log/messages", "/var/log/syslog" ]
}
file {
type => "apache-access"
path => "/var/log/apache2/access*.log"
}
file {
type => "apache-error"
path => "/var/log/apache2/error*.log"
}
}
output {
# Ship events to the amqp fanout exchange named 'rawlogs"
amqp {
host => "carina4.winsoft-international.com"
port => "5672"
#debug => "true"
exchange_type => "fanout"
name => "rawlogs"
}
}
Puis exécuter l'agent dans un screen:
cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/shipper.conf
Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-shipper'.
Sur le serveur de log
vim /etc/logstash/reader.conf
input {
amqp {
type => "all"
host => "127.0.0.1"
exchange => "rawlogs"
name => "rawlogs_consumer"
}
}
filter {
grok {
type => "syslog"
pattern => "%{SYSLOGLINE}"
}
grok {
type => "apache-access"
pattern => "%{COMBINEDAPACHELOG}"
}
date {
type => "syslog"
timestamp => "MMM d HH:mm:ss"
timestamp => "MMM dd HH:mm:ss"
timestamp8601 => ISO8601
}
date {
type => "apache-access"
timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
}
}
output {
gelf {
host => "localhost"
facility => "%{@type}"
}
}
Puis exécuter l'agent dans un screen:
cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/reader.conf
Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-reader'.
UI
If everything went well, you can connect to the web interface of graylog2 through the normal http port 80. Some analytics commands
Get the sorted distinct count of 404 requests
all.distribution({_request}, _response=404)
