A relatively recent while ago Graylog has undergone a huge facelift with the new no more ruby no more passenger web-interface, while also receiving an improved feature-set via the newer server versions (v0.20.x) .
Here’s how to deploy this latest and greatest Graylog on AWS , process which needs some custom ec2-related configuration, as you will understand from this tutorial.
Some pre-notes :
* This installation has been done on Amazon Linux v.2013.09, ec2 type : m1.large .
** I am running all commands as root .
1. Install and configure dependencies:
yum install make wget java-1.7.0-openjdk openjdk-7-jre openssl-devel libyaml-devel httpd git ImageMagick ImageMagick-devel libxml2-devel libxslt-devel gcc-c++ curl-devel httpd-devel apr-devel apr-util-devel -y
Graylog2 is only compatible with Java7, so make sure you are setting the default interpreter accordingly if you have more than one version of Java installed .Issue :
update-alternatives --config java
and select the correct option.
2. Install Mongo (you need at least v2.0):
Enable the mongo repository by adding these lines in /etc/yum.repos.d/mongodb.repo :
[mongodb] name=MongoDB Repository baseurl=https://downloads-distro.mongodb.org/repo/redhat/os/x86_64/ gpgcheck=0 enabled=1
Once done, install mongo :
yum install mongo-10gen mongo-10gen-server
Add mongo to boot:
chkconfig --add mongod chkconfig mongod on
Note that to avoid mongo “could not resolve hostname” type of errors you need to have a persistent hostname setup . In our case, this would mean editing the following files and running the hostname script to apply the change immediately :
/etc/hosts /etc/sysconfig/network hostname your_new_hostname
Start mongo :
service mongod start
Setup graylog2 mongo user and authentication ( authentication is not mandatory, but you can enable it ) :
mongo
use admin
db.addUser('admin', 'adminpass')
db.auth('admin', 'adminpass')
use graylog2
db.addUser('grayloguser', 'graylogpass')
db.auth('grayloguser', 'graylogpass')
exit
Restart mongo :
service mongod restart
3. Install*** ElasticSearch ( you need v0.90.10 specifically ) and the ElasticSearch service wrapper
***In this tutorial everything from now on will be installed in /opt/
Download and extract ES :
cd /opt/ curl https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.1.tar.gz | tar xz
Create a symlink for elasticsearch, to ease management:
ln -s /opt/elasticsearch-0.90.1 /opt/elasticsearch
Install the service wrapper:
curl -k -L https://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz mv *servicewrapper*/service elasticsearch/bin/ rm -Rf *servicewrapper* /opt/elasticsearch/bin/service/elasticsearch install
Next step is to configure ES . Go to /opt/elasticsearch/config/
– cluster name :
cluster.name: graylog2
– ec2discovery : because AWS does not support multicast , you need to explicitly specify the members of the cluster ; do this by un-commenting the following lines and adding your private IP as the host :
discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["your_ip_here"]
– also, if you want to use a different location for the data , this is where you update it , by un-commenting this line and adding your path here :
path.data: /your_path_to_data
Now start the service :
service elasticsearch start
4. Install Graylog2-Server
Download and extract the package :
cd /opt/ wget https://github.com/Graylog2/graylog2-server/releases/download/0.20.1/graylog2-server-0.20.1.tgz tar xvfz graylog2-server-0.20.1.tgz cd graylog2-server-0.20.1
Use the example configuration file as a reference and alter it to your needs:
cp graylog2.conf.example /etc/graylog2.conf
You should change at least the following /etc/graylog2.conf:
-set only one graylog2-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t. Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.:
is_master = true
-you must set a secret that is used for password encryption and salting here. The server will refuse to start if it’s not set. Generate a secret with for example pwgen -s 96.
password_secret
-a SHA2 hash of a password you will use for your initial login. Set this to a SHA2 hash generated with echo -n yourpassword | shasum -a 256 and you will be able to log in to the web interface with username admin and password yourpassword.
root_password_sha2
-elasticsearch cluster name, same as you configured in the elasticsearch.yml:
elasticsearch_cluster_name = graylog2
-the number of shards for your indices. Since we are running a single node cluster, this should be 1:
elasticsearch_shards = 1
-if you want to use mongo authentication, enable it here and provide the credentials configured at step#2:
mongodb_useauth = true
-last but not least , similar to the ElasticSearch config at step#3 , let’s configure ec2 discovery :
elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = your_ip_here:9300
For the optimal Graylog2 tuning, I strongly recommend you use our guide available here : https://www.emind.co/how-to/graylog2-tuning-guide . We apply these extra configs to all our Graylogs , including to older versions of it.
Now that graylog2 server is in place, let’s start it in debug and test it works :
java -jar graylog2-server.jar --debug
If Graylog successfully manages to connect to Elasticsearch, you should see a line like this :
2013-10-01 12:13:22,382 DEBUG: org.elasticsearch.transport.netty - [graylog2-server] connected to node [[Unuscione, Angelo][thN_gIBkQDm2ab7k-2Zaaw][inet[/10.37.160.227:9300]]]
If you see this line in the logs, it means that your Graylog is ready to go :
2013-10-01 12:13:53,149 INFO : org.graylog2.Core - Graylog2 up and running.
Once tests succeed , let’s add a startup script :
cd /etc/init.d ln -s /opt/graylog2-server-0.20.1/bin/graylog2ctl graylog2-server
We need to enable chkconfig in this script to be able to add it at boot, so open it for editing and loop in these lines at the beginning of the script :
#!/bin/bash #------------------------- #chkconfig: 2345 90 60 # #-------------------------
To conclude, we are adding the graylog server to boot :
chkconfig --add graylog2-server chkconfig graylog2-server on
Fire up the graylog server :
service graylog2-server start
5. Install Graylog2-Web-Interface
Download and extract the package :
cd /opt/ wget https://github.com/Graylog2/graylog2-web-interface/releases/download/0.20.1/graylog2-web-interface-0.20.1.tgz tar xvfz graylog2-web-interface-0.20.1.tgz cd graylog2-web-interface-0.20.1/
Configure the web interface in /opt/graylog2-web-interface-0.20.1/conf/graylog2-web-interface.conf . Take care at least of these 2 variables :
-the list of graylog2-server nodes the web interface will try to use. We are running single node in this tutorial, but you can configure multiple, separated by commas:
graylog2-server.uris="https://127.0.0.1:12900/"
-a secret for encryption. Use a long, randomly generated string here. (for example generated using pwgen -s 96 ) :
application.secret=""
Test the web interface :
bin/graylog2-web-interface Play server process ID is 5723 [info] play - Application started (Prod) [info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
The web interface will listen on port 9000.Try it out in your browser, go to https://your_ip_here:9000.
Log in with username admin and the password you configured at root_password_sha2 in the graylog2.conf of your graylog2-server.
Last step is to give yourself a control script for the interface as well. Previous to this, let’s setup a log appender for the web interface logs (which by default go to STDOUT). To do so, create /etc/graylog2-web-interface-log.xml , and in it put this configuration (which of course you can adapt to your preference) :
<configuration>
<!--
<appender name="STDOUT">
<encoder>
<pattern>%date %-5level [%thread] - [%logger]- %msg%n</pattern>
</encoder>
</appender>
-->
<appender name="ROLLING_FILE">
<file>/var/log/graylog2/web/graylog2-web-interface.log</file>
<rollingPolicy>
<FileNamePattern>/var/log/graylog2/web/graylog2-web-interface.log.%d{yyyy-MM-dd}.%i.log.gz</FileNamePattern>
<MaxHistory>30</MaxHistory>
<timeBasedFileNamingAndTriggeringPolicy>
<maxFileSize>100MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
<encoder>
<pattern>%date [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="INFO">
<!--<appender-ref ref="STDOUT" />-->
<appender-ref ref="ROLLING_FILE" />
</root>
</configuration>
And finally the init script – in /etc/init.d create a file called graylog2-web . In it put this to be able to control the interface :
#!/bin/bash
#
#-------------------------
#chkconfig: 2345 90 60
#
#-------------------------
CMD=$1
NOHUP='which nohup'
GRAYLOG2WEB_DIR=/opt/graylog2-web-interface-0.20.1
PID_FILE=RUNNING_PID
LOGGER_CONFIG_FILE=/etc/graylog2-web-interface-log.xml
start() {
echo "Starting graylog2-web-interface ..."
cd "$GRAYLOG2WEB_DIR"
$NOHUP bin/graylog2-web-interface -Dlogger.file=$LOGGER_CONFIG_FILE &
}
stop() {
PID=$(get_pid)
echo "Stopping graylog2-web-interface ($PID) ..."
if kill $PID; then
echo "graylog2-web-interface has been stopped "
rm -rf ${GRAYLOG2WEB_DIR}/${PID_FILE}
fi
}
restart() {
echo "Restarting graylog2-web-interface ..."
stop
start
}
status() {
pid=$(get_pid)
if [ ! -z $pid ]; then
if pid_running $pid; then
echo "graylog2-web-interface running as pid $pid"
return 0
else
echo "Stale pid file with $pid - removing..."
rm -rf ${GRAYLOG2WEB_DIR}/${PID_FILE}
fi
fi
echo "graylog2-web-interface not running"
}
get_pid() {
PID=""
if [ -f "${GRAYLOG2WEB_DIR}/${PID_FILE}" ]; then
PID=$(cat "${GRAYLOG2WEB_DIR}/${PID_FILE}")
fi
if [ -z $PID ]; then
PID=$(ps aux | grep java | grep graylog2-web-interface | awk '{print $2} ')
fi
echo ${PID}
}
pid_running() {
kill -0 $1 2> /dev/null
}
case "$CMD" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
*)
echo "Usage $0 {start|stop|restart|status}"
RETVAL=1
esac
Add the web interface to boot as well :
chkconfig --add graylog2-web chkconfig graylog2-web on
And start it :
service graylog2-web start
This is it. The installation is all done.
For Graylog to be able to receive your messages , you need to login to the web interface , select your graylog2-server node there and click on Manage inputs. Configure Inputs to your needs , most common ones would be GELF UDP and SYSLOG UDP :
Happy logging!