Powered by GitBook

Data Collection

Data Import from Java Applications

The 'fluent-logger-java' library is used to post records from Java applications to Fluentd.

This article explains how to use the fluent-logger-java library.

Prerequisites

  • Basic knowledge of Java
  • Basic knowledge of Fluentd
  • Java 6 or higher

Installing Fluentd

Please refer to the following documents to install fluentd.

Modifying the Config File

Next, please configure Fluentd to use the forward Input plugin as its data source.

<source>
  type forward
  port 24224
</source>
<match fluentd.test.**>
  type stdout
</match>

Please restart your agent once these lines are in place.

# for rpm/deb only
$ sudo /etc/init.d/td-agent restart

Using fluent-logger-java

First, please add the following lines to pom.xml. The logger's revision information can be found in CHANGES.txt.

<dependencies>
  ...
  <dependency>
    <groupId>org.fluentd</groupId>
    <artifactId>fluent-logger</artifactId>
    <version>${logger.version}</version>
  </dependency>
  ...
</dependencies>

Next, please insert the following lines into your application. Further information regarding the API can be found here.

import java.util.HashMap;
import java.util.Map;
import org.fluentd.logger.FluentLogger;

public class Main {
    private static FluentLogger LOG = FluentLogger.getLogger("fluentd.test");

    public void doApplicationLogic() {
        // ...
        Map<String, String> data = new HashMap<String, String>();
        data.put("from", "userA");
        data.put("to", "userB");
        LOG.log("follow", data);
        // ...
    }
}

Executing the script will send the logs to Fluentd.

$ java -jar test.jar

The logs should be output to /var/log/td-agent/td-agent.log or stdout of the Fluentd process via the stdout Output plugin.

Production Deployments

Output Plugins

Various output plugins are available for writing records to other destinations:

High-Availablability Configurations of Fluentd

For high-traffic websites (more than 5 application nodes), we recommend using a high availability configuration of td-agent. This will improve data transfer reliability and query performance.

Monitoring

Monitoring Fluentd itself is also important. The article below describes general monitoring methods for td-agent.