How to setup Fluentd to Log to S3 with Rails

2014-11-25

logging , production , fluentd

Logging is important. When you’re running a fully fledge production application server, you do not have the luxury of monitoring your rails console all the time.

In this example, I will go through with you the process of setting up Fluentd with your Rails App to log data and to upload them onto S3 for persistent archival purposes.

Pre-requsite

Running Trusty (14.04)
Ruby 2.1.2 installed
An AWS Account with your AWS Credentials (key and token) ready

Step 1: Download Td-Agent

Wait, isn’t this a Fluentd tutorial? What’s td-agent? Is this a bait and switch?

Nope you’re not! Td-Agent IS Fluentd. Just that its a slightly more stable Fluentd binary that is pre-packaged as an rpm/deb which makes it more convenient to deploy on servers.

Got it? Ok lets download!

# root not needed
$ curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

Thats it. You’re done. See you next week.

Hehe, seriously, you’re going to be out of here real shortly.

Step 2: Install the Fluentd-S3-Plugin

$ sudo gem install fluent-plugin-s3

Step 3: The Configuration

Add a relevant match like the following:

Replace logthis with a specific tag that you’d like to use to differenciate your different logs.

A good rule of thumb is to use task-based names. ie. “debugging”, “exceptions” etc…

# /etc/td-agent/td-agent.conf
<match logthis.**>
   type s3
   s3_region ap-southeast-1 

   aws_key_id __<INSERT AWS KEY>__
   aws_sec_key __<INSERT AWS SECRET KEY>__
   s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
   s3_bucket <INSERT BUCKET NAME>
   path logs/ # you can change this to anything you like
   buffer_path /var/log/fluent/s3 # this is the directory where your fluent logs
reside before being pushed out

   time_slice_format %Y%m%d%H%M
   time_slice_wait 1m
   utc
   buffer_chunk_limit 256m
</match>

Step 4: Creating the relevant buffer_path directory

And of course giving the right permissions.

$ cd /var/log
/var/log $ mkdir fluent
/var/log $ chown -Rf td-agent:td-agent fluent/

Step 5: Start Fluentd

Start td-agent and check if there’s any failure. If there is, tail -f it and restart.

$ service td-agent start
$ tail /var/log/td-agent/td-agent.log

If you see this:

# /var/log/td-agent/td-agent.log
...
2014-11-11 03:18:58 +0000 [info]: adding source type="forward"
2014-11-11 03:18:58 +0000 [info]: adding source type="http"
2014-11-11 03:18:58 +0000 [info]: adding source type="debug_agent"
2014-11-11 03:18:58 +0000 [info]: adding match pattern="debug.**" type="stdout"
2014-11-11 03:18:58 +0000 [info]: adding match pattern="logthis.**" type="s3"
2014-11-11 03:18:59 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2014-11-11 03:18:59 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230"
object="Engine"

You’re ok.

Step 6: …

For the purpose of this example, we’re going to use fluent-logger

  # if you don't have it already
  $ gem install fluent-logger

  $ irb
  >> require 'fluent-logger'
  => true
  >> logger = Fluent::Logger::FluentLogger.open(nil, :host=>'localhost', :port=>24224)
  >> logger.post('logthis.hey', {:message => "Hello!"})

In your /var/log/fluent directory, you should see a new log file, like:

/var/log/fluent $ ls
s3.201411110327.b5078cdb47b2e167b.log

Tail it and you should see your message in there.

/var/log/fluent $ tail  s3.201411110327.b5078cdb47b2e167b.log
2014-11-11T03:27:19Z    logthis.hey       {"message":"Hello!"}

Woo! At this stage, give yourself a little pat.

Step 7: Profit(?)

Now, wait for 2 minutes and you should see the log appearing in your assigned S3 Bucket’s path from your td-agent.conf file.

# /etc/td-agent/td-agent.conf
<match ...>
   type s3
   path logs/ # <-- 
</match>

If you don’t, check your /var/log/td-agent/td-agent.log if you’re having any errors.

Congratulations! You’re now running a fully archived logging system!