HowTo/InstallGrayLog2 - Revision history

Eadam: /* EPEL */

2012-06-24T11:53:04Z

EPEL

Eadam: /* Passenger */

2012-06-24T11:51:39Z

Passenger

Eadam at 11:50, 24 June 2012

2012-06-24T11:50:20Z

Eadam: Created page with "graylog2 Installation pre-requis L'installation se fera a partir d'une RHEL/CentOS 6.2 minimal. EPEL Installer EPEL yum -y install libyaml java-1.6.0-openjdk erlang screen ..."

2012-04-15T09:38:14Z

Created page with "graylog2 Installation pre-requis L'installation se fera a partir d'une RHEL/CentOS 6.2 minimal. EPEL Installer EPEL yum -y install libyaml java-1.6.0-openjdk erlang screen ..."

New page

graylog2

Installation pre-requis

L'installation se fera a partir d'une RHEL/CentOS 6.2 minimal.
EPEL

Installer EPEL
yum -y install libyaml java-1.6.0-openjdk erlang screen telnet rubygems openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel httpd-devel vim ruby-devel
RabbitMQ?

yum -y install http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server-2.8.1-1.noarch.rpm
service rabbitmq-server start
chkconfig rabbitmq-server on
[optional, only if server won't start] edit vi /etc/init.d/rabbitmq-server
...
#$CONTROL wait $PID_FILE >/dev/null 2>&1
#RETVAL=$?
RETVAL=0
...
iptables

iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 80 -j ACCEPT
iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 5672 -j ACCEPT
service iptables save
Mongo

This is the database that will store the metadata.
Install the repo: vim /etc/yum.repos.d/10gen.repo
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
yum -y install mongo-10gen mongo-10gen-server
/etc/mongod.conf
nohttpinterface = true
auth = true
port = 27017
service mongod start
chkconfig mongod on
Taper mongo puis:
use admin
db.addUser('admin', 'iopiop')
db.auth('admin', 'iopiop')
use graylog2
db.addUser('eadam', 'passwd')
Elasticsearch

This is the distributed database that will store the messages.
Download and install elasticsearch from github:
cd
wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz
tar xzvf elasticsearch-0.18.7.tar.gz
mv elasticsearch-0.18.7 /usr/local/bin/elasticsearch
curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv *servicewrapper*/service /usr/local/bin/elasticsearch/bin/
cd /usr/local/bin/elasticsearch
./bin/service/elasticsearch install
/etc/init.d/elasticsearch start
chkconfig elasticsearch on
Installation de graylog2 serveur+web

Server

wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
tar xzvf graylog2-server-0.9.6.tar.gz
mv graylog2-server-0.9.6 /opt/graylog2
cd /opt/graylog2
cp graylog2.conf.example /etc/graylog2.conf
Mettre a jour la configuration permettant la connexion a mongodb:
vim /etc/graylog2.conf
...
# MongoDB Configuration
mongodb_useauth = true
mongodb_user = <user>
mongodb_password = <password>
mongodb_host = localhost
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017
...
Finalement, démarrer le serveur
Copier le fichier graylog2-server attaché en bas de cette page dans /etc/init.d/ puis exécuter:
chmod +x /etc/init.d/graylog2-server
chkconfig --add graylog2-server
service graylog2-server start
Web interface

Installation

wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
tar xzvf graylog2-web-interface-0.9.6.tar.gz
mv graylog2-web-interface-0.9.6 /opt/graylog2-web-interface
chown apache:apache -R /opt/graylog2-web-interface
chcon -R system_u:object_r:httpd_sys_content_t:s0 /opt/graylog2-web-interface
vim /opt/graylog2-web-interface/config/mongoid.yml
vim /opt/graylog2-web-interface/config/general.yml
vim /opt/graylog2-web-interface/config/email.yml
cd /opt/graylog2-web-interface
gem install bundler
bundle install
setsebool -P httpd_can_network_connect_db on
setsebool -P httpd_can_network_connect on
Passenger

gem install passenger
passenger-install-apache2-module
vim /etc/httpd/conf.d/passenger.conf
LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11
PassengerRuby /usr/bin/ruby
Rajouter a la fin de /etc/httpd/conf/httpd.conf
NameVirtualHost *:80

<VirtualHost *:80>
ServerName carina4.winsoft-international.com
DocumentRoot /opt/graylog2-web-interface/public
<Location />
Allow from all
Options -MultiViews
</Location>
ErrorLog /var/log/httpd/graylog_error.log
LogLevel warn
CustomLog /var/log/httpd/graylog_access.log combined
</VirtualHost>
Redémarrer apache
service httpd restart
logstash

logstash permet de lire les fichiers de log, de les envoyer au travers de queue (rabbitmq ici) et de les 'parser' pour les renvoyer dans graylog2 sous format GELF
Une instance de logstash s'exécute sur chaque serveurs envoyant les logs (les clients), et une instance s'exécute sur le serveur centralisant les logs (le serveur).
Download

mkdir /opt/logstash
wget -O /opt/logstash/logstash.jar http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
mkdir /etc/logstash
Sur les serveurs 'clients'

vim /etc/logstash/shipper.conf
input {
file {
type => "syslog"
path => [ "/var/log/messages", "/var/log/syslog" ]
}
file {
type => "apache-access"
path => "/var/log/apache2/access*.log"
}
file {
type => "apache-error"
path => "/var/log/apache2/error*.log"
}
}
output {
# Ship events to the amqp fanout exchange named 'rawlogs"
amqp {
host => "carina4.winsoft-international.com"
port => "5672"
#debug => "true"
exchange_type => "fanout"
name => "rawlogs"
}
}
Puis exécuter l'agent dans un screen:
cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/shipper.conf
Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-shipper'.
Sur le serveur de log

vim /etc/logstash/reader.conf
input {
amqp {
type => "all"
host => "127.0.0.1"
exchange => "rawlogs"
name => "rawlogs_consumer"
}
}
filter {
grok {
type => "syslog"
pattern => "%{SYSLOGLINE}"
}
grok {
type => "apache-access"
pattern => "%{COMBINEDAPACHELOG}"
}
date {
type => "syslog"
timestamp => "MMM d HH:mm:ss"
timestamp => "MMM dd HH:mm:ss"
timestamp8601 => ISO8601
}
date {
type => "apache-access"
timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
}
}
output {
gelf {
host => "localhost"
facility => "%{@type}"
}
}
Puis exécuter l'agent dans un screen:
cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/reader.conf
Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-reader'.
UI

If everything went well, you can connect to the web interface of graylog2 through the normal http port 80.
Some analytics commands

Get the sorted distinct count of 404 requests
all.distribution({_request}, _response=404)

@@ Line 7: / Line 7: @@
 ==== EPEL ====
-  yum install
+  yum install -y http://mirrors.ircam.fr/pub/fedora/epel/6/x86_64/epel-release-6-7.noarch.rpm
 ==== Base packages ====

@@ Line 161: / Line 161: @@
   service httpd restart
-logstash
+==== logstash ====
 logstash permet de lire les fichiers de log, de les envoyer au travers de queue (rabbitmq ici) et de les 'parser' pour les renvoyer dans graylog2 sous format GELF
 Une instance de logstash s'exécute sur chaque serveurs envoyant les logs (les clients), et une instance s'exécute sur le serveur centralisant les logs (le serveur).
 Download
-mkdir /opt/logstash
+ mkdir /opt/logstash
-wget -O /opt/logstash/logstash.jar http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
+ wget -O /opt/logstash/logstash.jar http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
-mkdir /etc/logstash
+ mkdir /etc/logstash
-vim /etc/logstash/shipper.conf
+===== Sur les serveurs 'clients' =====
 input {
     file {
@@ Line 197: / Line 200: @@
+    }
+}
 Puis exécuter l'agent dans un screen:
 cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/shipper.conf
 Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-shipper'.
-vim /etc/logstash/reader.conf
+===== Sur le serveur de log =====
 input {
          amqp {
@@ Line 237: / Line 246: @@
+    }
+}
 Puis exécuter l'agent dans un screen:
 cd /opt/logstash; java -jar logstash.jar agent -f /etc/logstash/reader.conf
 Ou le lancer en utilisant le script en bas de cette page '/etc/init.d/logstash-reader'.
 UI
 If everything went well, you can connect to the web interface of graylog2 through the normal http port 80.
@@ Line 246: / Line 260: @@
 Get the sorted distinct count of 404 requests
 all.distribution({_request}, _response=404)

@@ Line 1: / Line 1: @@
-graylog2
+== graylog2 ==
-Installation pre-requis
+=== Installation pre-requis ===
-L'installation se fera a partir d'une RHEL/CentOS 6.2 minimal.
+L'installation se fera sur une RHEL/CentOS 6.2 minimal.
-Installer EPEL
+==== EPEL ====
 yum -y install libyaml java-1.6.0-openjdk erlang screen telnet rubygems openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel httpd-devel vim ruby-devel
-RabbitMQ?
+ yum install
 [optional, only if server won't start] edit vi /etc/init.d/rabbitmq-server
 ...
@@ Line 21: / Line 27: @@
 iptables
-iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 80 -j ACCEPT
+ iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 80 -j ACCEPT
-iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 5672 -j ACCEPT
+ iptables -I INPUT 4 -m tcp -p tcp -m state --state NEW --dport 5672 -j ACCEPT
-service iptables save
+ service iptables save
 Mongo
 This is the database that will store the metadata.
 Install the repo: vim /etc/yum.repos.d/10gen.repo
 [10gen]
 name=10gen Repository
@@ Line 45: / Line 55: @@
 use graylog2
 db.addUser('eadam', 'passwd')
-Elasticsearch
+</pre>
 This is the distributed database that will store the messages.
 Download and install elasticsearch from github:
 cd
 wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz
@@ Line 59: / Line 73: @@
 /etc/init.d/elasticsearch start
 chkconfig elasticsearch on
-Installation de graylog2 serveur+web
+</pre>
-Server
+==== Installation de graylog2 serveur+web ====
 wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
 tar xzvf graylog2-server-0.9.6.tar.gz
@@ Line 68: / Line 84: @@
 cd /opt/graylog2
 cp graylog2.conf.example /etc/graylog2.conf
 Mettre a jour la configuration permettant la connexion a mongodb:
 vim /etc/graylog2.conf
 ...
 # MongoDB Configuration
@@ Line 80: / Line 100: @@
 mongodb_port = 27017
 ...
 Finalement, démarrer le serveur
 Copier le fichier graylog2-server attaché en bas de cette page dans /etc/init.d/ puis exécuter:
 chmod +x /etc/init.d/graylog2-server
-chkconfig --add graylog2-server
+ chmod +x /etc/init.d/graylog2-server
-service graylog2-server start
+ chkconfig --add graylog2-server
-Web interface
+ service graylog2-server start
 Installation
+<pre>
 wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
 tar xzvf graylog2-web-interface-0.9.6.tar.gz
@@ Line 102: / Line 128: @@
 setsebool -P httpd_can_network_connect_db on
 setsebool -P httpd_can_network_connect on
-Passenger
+</pre>
 gem install passenger
 passenger-install-apache2-module
@@ Line 110: / Line 139: @@
 PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11
 PassengerRuby /usr/bin/ruby
 Rajouter a la fin de /etc/httpd/conf/httpd.conf
 NameVirtualHost *:80
@@ Line 124: / Line 156: @@
          CustomLog /var/log/httpd/graylog_access.log combined
 </VirtualHost>
 Redémarrer apache
-service httpd restart
+ service httpd restart
 logstash