I just announced the new Spring Boot 2 material, coming in REST With Spring:

>> CHECK OUT THE COURSE

1. Introduction

Graylog is a log aggregation service. Simply put, it’s capable of collecting millions of log messages from multiple sources and displaying them in a single interface.

And, it also provides a number of other features such as real-time alerts, dashboards with graphs and charts, and much more.

In this tutorial, we’ll see how to set up a Graylog server and send log messages to it from a Spring Boot application.

2. Setting up Graylog

There are several ways to install and run Graylog. In this tutorial, we’ll discuss the two quickest ways: Docker and Amazon Web Services.

2.1. Docker

The following commands will download all the required Docker images and start a container for each service:

$ docker run --name mongo -d mongo:3
$ docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
    -e ES_JAVA_OPTS="-Xms2g -Xmx4g" \
    -e "discovery.type=single-node" -e "xpack.security.enabled=false" \
    -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
    -d docker.elastic.co/elasticsearch/elasticsearch:5.6.11
$ docker run --name graylog --link mongo --link elasticsearch \
    -p 9000:9000 -p 12201:12201 -p 514:514 -p 5555:5555 \
    -e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api" \
    -d graylog/graylog:2.4.6-1

The Graylog dashboard is now available using the URL http://localhost:9000/ and the default username and password are both admin.

While the Docker setup is the simplest, it does require a substantial amount of memory. It also doesn’t work on Docker for Mac, so may not be suitable for all platforms.

2.2. Amazon Web Services

The next easiest option for setting up Graylog for testing is Amazon Web Services. Graylog provides an official AMI that includes all the required dependencies, although it does require some additional configuration after installation.

We can quickly deploy an EC2 instance with the Graylog AMI by clicking here and selecting a region. Graylog recommends using an instance with at least 4GB memory.

After the instance has started, we need to SSH into the host and make a few changes. The following commands will configure the Graylog service for us:

$ sudo graylog-ctl enforce-ssl
$ sudo graylog-ctl set-external-ip https://<EC2 PUBLIC IP>:443/api/
$ sudo graylog-ctl reconfigure

We also need to update the security group that was created with the EC2 instance to allow network traffic on specific ports. The graphic below shows the ports and protocols that need to be enabled:

The Graylog dashboard is now available using the URL https://<EC2 PUBLIC IP>/ and the default username and password are both admin.

2.3. Other Graylog Installations

Aside from Docker and AWS, there are also Graylog packages for various operating systems. With this approach, we also have to set up an ElasticSearch and MongoDB service.

For this reason, Docker and AWS are much easier to set up, especially for development and testing purposes.

3. Sending In Log Messages

With Graylog up and running, we must now configure our Spring Boot application to send log messages to the Graylog server.

Any Java logging framework can support sending messages to a Graylog server using the GELF protocol.

3.1. Log4J

At this time the only officially supported logging framework is Log4J. Graylog provides an appender, which is available on Maven central.

We can enable it by adding the following Maven dependency to any pom.xml file:

<dependency>
    <groupId>org.graylog2</groupId>
    <artifactId>gelfj</artifactId>
    <version>1.1.16</version>
</dependency>

We also must exclude the logging starter module anywhere we use a Spring Boot starter module:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Now we can define a new appender in our log4j.xml file:

<appender name="graylog" class="org.graylog2.log.GelfAppender">
    <param name="graylogHost" value="<GRAYLOG IP>"/>
    <param name="originHost" value="localhost"/>
    <param name="graylogPort" value="12201"/>
    <param name="extractStacktrace" value="true"/>
    <param name="addExtendedInformation" value="true"/>
    <param name="facility" value="log4j"/>
    <param name="Threshold" value="INFO"/>
    <param name="additionalFields" value="{'environment': 'DEV', 'application': 'GraylogDemoApplication'}"/>
</appender>

This will configure all log messages with INFO level or higher to go to the Graylog appender, which in turn sends the log message to the Graylog server.

3.2. Other Logging Frameworks

The Graylog marketplace has additional libraries that support a variety of other logging frameworks such as Logback, Log4J2, and more. Just beware that these libraries are not maintained by Graylog. Some of them are abandoned, and others have little or no documentation.

Caution should be used when relying on these 3rd party libraries.

3.3. Graylog Collector Sidecar

Another option for log collection is the Graylog Collector Sidecar. The sidecar is a process that runs along a file collector, sending log file contents to a Graylog server.

The Sidecar is a great option for applications where changing log configuration files isn’t possible. And because it reads log files directly from disk, it can also be used to integrate log messages from any platform and programming language.

4. Viewing Messages in Graylog

We can use the Graylog dashboard to confirm successful delivery of our log messages. Using the filter source:localhost will show the log messages from our sample log4j config above:

5. Conclusion

Graylog is just one of many log aggregation services. It can quickly search millions of log messages, visualize log data in real-time, and send alerts when certain conditions are true.

Integrating Graylog into a Spring Boot application only requires a few lines of configuration and without any new code.

Code samples, as always, can be found on GitHub.

I just announced the new Spring Boot 2 material, coming in REST With Spring:

>> CHECK OUT THE LESSONS

4
Leave a Reply

avatar
2 Comment threads
2 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
3 Comment authors
Loredana CrusoveanuBiniSam Mizanin Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Sam Mizanin
Guest
Sam Mizanin

It’s a good article which shows configuration but not the problems. What if the graylog server goes down when the application is up and running? Where will the log messages be written in such a case? If it’s written directly using the gelf appender then we would loose the logs and will be in a greater problem if there is a critical issue that needs to be debugged. Can you please include these points as well as one of the sections? Developers can do the configurations and do a POC very easily but pointing out intricate problems and solutions to… Read more »

Loredana Crusoveanu
Editor

Hey Sam,

Thanks for the feedback. For a more robust setup in production, there are several options. You could have multiple clustered Graylog servers behind a load balancer (http://docs.graylog.org/en/latest/pages/architecture.html#complex-multi-node), or use Kafka to add an extra layer of message queueing (http://docs.graylog.org/en/2.4/pages/sending_data.html#using-apache-kafka-as-transport-queue). This type of architectural decisions isn’t strictly related to Graylog which is why they weren’t part of this write-up. But we’ll add updating this article with more solutions as suggested to our content calendar.

Cheers.

Bini
Guest
Bini

Great tutorial. Thanks.

Can you explain why the docker option doesn’t work on Mac and if there is a workaround?

Loredana Crusoveanu
Editor

Hey Bini,

Not sure what the cause is – it may be related to having insufficient memory. It’s worth trying on a more powerful machine if you have one.