Using the Logstash collector version 1.0

Use the Logstash collector feature in Liberty to collect log and other events from your Liberty servers and send them to a remote Logstash server. The collected events can be used for log analysis and troubleshooting purposes.

Before you begin

You can install and configure Kibana 4.x by following the instructions from elastic.co.

Procedure

  1. Set up Logstash V2.x by following the instructions from elastic.co.
  2. Create or acquire certificate and key pair files for SSL for Logstash. The following example is the command for openSSL that can be used for generating a certificate and key pair. Customize the number of days the keys are valid as required.
    openssl req -x509 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -days 365 -nodes
  3. For Logstash V2.x and Elasticsearch users, copy the sample into a liberty_logstash_template.json file. See the repository for a sample Logstash index template. Customize the _ttl defaults as required to indicate the number of milliseconds to keep records of each event type.
  4. For Logstash V2.x and Elasticsearch users, copy the sample into a liberty_logstash.conf file. See the repository for sample Logstash filters. Customize lumberjack ssl_certificate path, ssl_key path, and port number as required. Customize Elasticsearch hosts and template path as required.
  5. Complete the following steps for each of the Liberty servers that you want to collect events from:
    1. Acquire or create a keystore for the Liberty server. To create a self-signed certificate, use the following command. Customize the server name, password, and subject as required.
      d:\wlp\bin\securityUtility createSSLCertificate --server=myServerName --password="Liberty" --subject=CN=myHostname,OU=defaultServer,O=ibm,C=us
    2. Import the logstash.crt file from step 2 into the trust.jks file in your server. Customize the wlp_install_dir and server name as required. When prompted for a password, use the certificate password from step 5a.
      d:\java\bin\keytool -import -noprompt -alias logstash -file logstash.crt -keystore wlp_install_dir\usr\servers\myServerName\resources\security\trust.jks -storepass Liberty
    3. Run the following command to install the logstashcollector-1.0 feature:
      d:\wlp\bin\installUtility install logstashcollector-1.0
    4. Configure Logstash collector in the server.xml file in Liberty by adding the following content. Customize the logstashCollector list of sources, host name, and port as required.

      [17.0.0.1 and later]You can add custom tags to decorate events that are sent from the server. The following example has two tag elements in the logstashCollector element.

      <featureManager>
          <feature>logstashCollector-1.0</feature>
      </featureManager>
      
      <keyStore id="defaultKeyStore" password="Liberty" />
      <keyStore id="defaultTrustStore" location="trust.jks" password="Liberty" />
      <ssl id="mySSLConfig" trustStoreRef="defaultTrustStore" keyStoreRef="defaultKeyStore" />
      
      <logstashCollector 
      	source="message,trace,garbageCollection,ffdc,accessLog" 
      	hostName="localhost" 
      	port="5043" 
      	sslRef="mySSLConfig">
      
          [17.0.0.1 and later]<tag>serverRackA5</tag>
          [17.0.0.1 and later]<tag>billingAppTeam</tag>
      </logstashCollector>
      Note: Trace and access logs are high volume logs and require more network, CPU, and storage resources to collect.
  6. For users of Elasticsearch and Kibana V4.x, import Kibana dashboards as follows:
    1. Save Kibana dashboard, visualization, and search JSON files to a directory on your local file system. See the repository for samples.
    2. In Kibana, configure the index pattern. Click Settings > Indices and set Index name to logstash-* and Time-field name to datetime.
    3. Click Settings > Objects > Import to import the dashboards, visualizations, and searches into Kibana. When prompted, provide the path to the files you saved in the previous step.

    The next time that you go to Kibana from any browser, you can load the imported dashboards by using the load icon.

Results

You configured your Liberty servers to send events to your Logstash server and can now view your events in the Liberty dashboard by using Kibana.


Icon that indicates the type of topic Task topic

File name: twlp_analytics_logstash.html