Logstash and Bluemix log collectors

Liberty generates various events at runtime, such as log events, trace events, first failure data capture (FFDC) events, access log events, and garbage collection events. It is helpful to consolidate events from all servers so the events can be searched, filtered, and analyzed, particularly when you are managing many servers, or when you are running servers on different platforms (for example on dedicated hardware and in the cloud). It can also be helpful to store events on a separate server in cases where you use Liberty in environments that lack persistent file storage for problem determination data.

Liberty provides the Logstash and Bluemix log collector features to help you remotely consolidate events. The collectors capture in-flight events, break them into fields, and securely forward the events to the configured Logstash or Bluemix log collection server. The collectors can be configured to send one or more of the following types of data:

  • Message log events
  • Trace log events
  • HTTP access log events (from the Liberty server)
  • FFDC log events
  • Garbage collection events (available for IBM JDKs only)

Logstash collector

The Logstash collector feature (logstashCollector-1.0) sends events to a Logstash server that you provide.

Logstash can be used with the Elasticsearch search server and Kibana dashboard, all of which you provide, set up, and manage, to give a consolidated view of logs or other events from across your enterprise. You don't need to set up separate processes or agents on the Liberty server computer for the Logstash collector to forward events.

Event structure

The collectors send each event as a set of field name-value pairs. Each different type of event has its own set of fields. Knowing which fields each event has is useful when you create your own Kibana dashboards.

The following fields are common and present in all events:
Table 1. Common event types
logstashCollector-1.0 Field Name Field Description
type A string that identifies the type of event
datetime Time at which the event occurred
hostName Host name of the server that was the source of the event

For more information about the rules to set the host name, see Using Logstash and Bluemix log collectors with Liberty running in Docker.

wlpUserDir User directory of the server that was the source of the event, for example, D:\wlp\usr
serverName Server name of the server that was the source of the event

For more information about the rules to set the server name, see Using Logstash and Bluemix log collectors with Liberty running in Docker

sequence Sequence number of event (useful for sorting records with the same time stamp)
Besides the common fields, each of the event types also has its own unique fields:
  • Message events
    Table 2. Message events (type: "liberty_message":)
    logstashCollector-1.0 Field Name Field Description
    severity Severity indicator (F = Fatal, E = Error, W = Warning, A = Audit, I = Info, O = SystemOut, R = SystemErr)
     

    Severity indicator (FATAL, ERROR, WARNING, AUDIT, INFO, CONFIG, DETAIL)

    messageId Message ID in the log line, which can be used to find out specific types of errors, for example, SRVE0250I
    methodName Method name from log record
    className Class name from log record
    loggerName Logger name from log record
    threadId Thread ID in the log line, for example, 00000015. The thread ID is a string and not a number
    message The message from log record, starting with the message ID
    [16.0.0.4 and later]Note: In addition to the default log and trace framework, the Logstash collector forwards message events when binary logging is enabled.
  • Trace events
    Table 3. Trace events (type: "liberty_trace":)
    logstashCollector-1.0 Field Name Field Description
    severity Severity indicator (1 = Fine, 2 = Finer, 3 = Finest, > = Entry, < = Exit)
      Severity indicator (FINE, FINER, FINEST, ENTRY, EXIT)
    methodName Method name from log record
    className Class name from log record
    loggerName Logger name from log record
    threadId Thread ID in the log line, for example, 00000015. The thread ID is a string and not a number
    message The message from log record
    [16.0.0.4 and later]Note: In addition to the default log and trace framework, the Logstash collector forwards trace events when binary logging is enabled.
  • HTTP access log events
    Table 4. HTTP access log events (type: "liberty_accesslog":)
    logstashCollector-1.0 Field Name Field Description
    uriPath Path information for the requested URL. This path information does not contain the query parameters, for example, /pushworksserver/ push/apps/tags.
    requestMethod HTTP verb, for example, GET
    remoteHost Remote host IP address, for example, 127.0.0.1
    userAgent The userAgent value in the request
    requestProtocol Protocol type, for example, HTTP/1.1
    queryString String representing query string from the HTTP request, for example, color=blue&size=large
    bytesReceived Bytes received in the URL, for example, 94
    responseCode HTTP response code, for example, 200
    elapsedTime Time taken to serve the request, in microseconds
    requestHost Request host IP address, for example, 127.0.0.1
    requestPort Port number of the request
  • FFDC events
    Table 5. FFDC events (type: "liberty_ffdc":)
    logstashCollector-1.0 Field Name Field Description
    className The class that emitted the FFDC entry
    exceptionName The exception that was reported in the FFDC entry
    probeID The unique identifier of the FFDC point within the class
    stackTrace The stack trace of the FFDC incident
    objectDetails The incident details for the FFDC incident
    threadId The thread ID of the FFDC incident
  • Garbage collection events
    Table 6. Garbage collection events (type: "liberty_gc":)
    logstashCollector-1.0 Field Name Field Description
    heap The total heap available
    usedHeap The amount of heap used
    duration The duration for which garbage collection was run, in microseconds
    gcType The type of garbage collection event (for example, Nursery, Global)
    reason The reason for the garbage collection

Using Logstash and Bluemix log collectors with Liberty running in Docker

For running Liberty servers in Docker containers or other virtualized environments, the hostName and serverName are set as follows:

The hostName field is set to the first of the following values that is available:
  • The value of the CONTAINER_HOST environment variable
  • The value of the ${defaultHostName} Liberty configuration variable
  • The canonical host name as reported by the software development kit (JDK)
The serverName field is set to the first of the following values that is available:
  • The value of the CONTAINER_NAME environment variable
  • The value of the ${wlp.server.name} Liberty configuration variable

When Liberty servers run in the Bluemix Container Service, the CONTAINER_HOST and CONTAINER_NAME are already set for you.

When Liberty servers run in Liberty Docker collectives, the CONTAINER_HOST and CONTAINER_NAME are already set for you.

When running Liberty servers in Docker containers or other virtual environments you manage, set the CONTAINER_HOST and CONTAINER_NAME environment variables when you run your image. These environment variables ensure that the Logstash collector or Bluemix log collector tags the records that they send with the appropriate host and container name, which aids in problem determination. If you do not set these environment variables, you might find it difficult to determine which container sent which events when you use a dashboard that shows events from multiple containers. When starting a Docker container, use a command similar to the following:
docker run -d -e LICENSE=accept -e CONTAINER_NAME=yourContainerName -e CONTAINER_HOST=yourContainerHost --name=yourContainerName yourImageName

Icon that indicates the type of topic Concept topic



Timestamp icon Last updated: Monday, 5 December 2016
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=cord&product=was-nd-mp&topic=cwlp_analytics_collectors
File name: cwlp_analytics_collectors.html