Logstash and Bluemix log collectors

Liberty generates various events at runtime, such as log events, trace events, first failure data capture (ffdc) events, access log events, and garbage collection events. It is helpful to consolidate events from all servers so the events can be searched, filtered, and analyzed, particularly when you are managing many servers, or when you are running servers on different platforms (for example on dedicated hardware and in the cloud). It can also be helpful to store events on a separate server in cases where you use Liberty in environments that lack persistent file storage for problem determination data.

Liberty now provides the Logstash and Bluemix log collector features to help you remotely consolidate events. The collectors capture in-flight events, break them into fields, and securely forward the events to the configured Logstash or Bluemix log collection server. The collectors can be configured to send one or more of the following types of data:
  • Message log events
  • Trace log events
  • HTTP access log events (from the Liberty server)
  • FFDC log events
  • Garbage collection events (available for IBM JDKs only)

Logstash collector

The Logstash collector feature (logstashCollector-1.0 or logstashCollector-1.1) sends events to a Logstash server that you provide.

Logstash can be used with the Elasticsearch search server and Kibana dashboard, all of which you provide, set up, and manage, to give a consolidated view of logs or other events from across your enterprise. Note that there are no separate processes or agents to be set up on the Liberty server machine when using the Logstash collector to forward events.

Figure 1. Logstash collector
This diagram shows the flow of events to the Elasticsearch server through the logstash event pipeline.

Event structure

The collectors send each event as a set of field name-value pairs. Each different type of event has its own set of fields. Knowing which fields each event has is useful when creating your own Kibana dashboards.
Note: The logstashCollector-1.0 feature has different field names than the logstashCollector-1.1 and bluemixLogCollector-1.1 features.
The following fields are common and present in all events:
Table 1. Common event types
logstashCollector-1.0 Field Name logstashCollector-1.1 / bluemixLogCollector-1.1 Field Name Field Description
type type a string that identifies the type of event
datetime was_datetime time at which the event occurred
hostName host host name of the server that was the source of the event

For more information about the rules to set the host name, see Using Logstash and Bluemix log collectors with Liberty running in Docker

wlpUserDir was_userDir user directory of the server that was the source of the event, for example, D:\wlp\usr
serverName was_serverName server name of the server that was the source of the event

For more information about the rules to set the server name, see Using Logstash and Bluemix log collectors with Liberty running in Docker

sequence was_sequence sequence number of event (useful for sorting records with the same time stamp)
Besides the common fields, each of the event types also has its own unique fields:
  • Message events
    Table 2. Message events (type: "liberty_message":)
    logstashCollector-1.0 Field Name logstashCollector-1.1 / bluemixLogCollector-1.1 Field Name Field Description
    severity   severity indicator (F = Fatal, E = Error, W = Warning, A = Audit, I = Info, O = SystemOut, R = SystemErr)
      loglevel

    severity indicator (FATAL, ERROR, WARNING, AUDIT, INFO, CONFIG, DETAIL)

    messageId was_messageId message ID in the log line, which can be used to find out specific types of errors, for example, SRVE0250I
    methodName was_methodName method name from log record
    className was_className class name from log record
    loggerName module logger name from log record
    threadId was_threadId thread ID in the log line, for example, 00000015. Note that the thread ID is a string and not a number
    message message the message from log record, starting with the message ID
  • Trace events
    Table 3. Trace events (type: "liberty_trace":)
    logstashCollector-1.0 Field Name logstashCollector-1.1 / bluemixLogCollector-1.1 Field Name Field Description
    severity   severity indicator (1 = Fine, 2 = Finer, 3 = Finest, > = Entry, < = Exit)
      loglevel severity indicator (FINE, FINER, FINEST, ENTRY, EXIT)
    methodName was_methodName method name from log record
    className was_className class name from log record
    loggerName module logger name from log record
    threadId was_threadId thread ID in the log line, for example, 00000015. Note that the thread ID is a string and not a number
    message message the message from log record
  • HTTP access log events
    Table 4. HTTP access log events (type: "liberty_accesslog":)
    logstashCollector-1.0 Field Name logstashCollector-1.1 / bluemixLogCollector-1.1 Field Name Field Description
    uriPath was_uriPath Path information for the requested URL. This does NOT contain the query parameters, for example, /pushworksserver/ push/apps/tags
    requestMethod was_requestMethod HTTP verb used, for example, GET
    remoteHost was_remoteHost remote host IP address, for example, 127.0.0.1
    userAgent was_userAgent userAgent value in the request
    requestProtocol was_requestProtocol Protocol type, for example, HTTP/1.1
    queryString was_queryString

    string representing query string from the HTTP request, for example, color=blue&size=large

    bytesReceived was_bytesReceived bytes received in the URL, for example, 94
    responseCode was_responseCode HTTP response code, for example, 200
    elapsedTime was_elapsedTime time taken to serve the request, in microseconds
    requestHost was_requestHost request host IP address, for example, 127.0.0.1
    requestPort was_requestPort port number of the request
  • FFDC events
    Table 5. FFDC events (type: "liberty_ffdc":)
    logstashCollector-1.0 Field Name logstashCollector-1.1 / bluemixLogCollector-1.1 Field Name Field Description
    className was_className the class that emitted the FFDC entry
    exceptionName was_exceptionName the exception that was reported in the FFDC entry
    probeID was_probeID the unique identifier of the FFDC point within the class
    stackTrace was_stackTrace the stack trace of the FFDC incident
    objectDetails was_objectDetails the incident details for the FFDC incident
    threadId was_threadId the thread ID of the FFDC incident
  • Garbage collection events
    Table 6. Garbage collection events (type: "liberty_gc":)
    logstashCollector-1.0 Field Name logstashCollector-1.1 / bluemixLogCollector-1.1 Field Name Field Description
    heap was_heap the total heap available
    usedHeap was_usedHeap the amount of heap used
    duration was_duration the duration for which garbage collection was run, in microseconds
    gcType was_gcType the type of garbage collection event (for example: Nursery, Global, etc.)
    reason was_reason the reason for the garbage collection

Using Logstash and Bluemix log collectors with Liberty running in Docker

For running Liberty servers in Docker containers or other virtualized environments, the host/hostName and serverName/was_serverName are set as follows:

the host/hostName field is set to the first of the following values that is available:
  • the value of the CONTAINER_HOST environment variable
  • the value of the ${defaultHostName} Liberty configuration variable
  • the canonical hostname as reported by the JDK
the serverName/was_serverName is set to the first of the following values that is available:
  • the value of the CONTAINER_NAME environment variable
  • the value of the ${wlp.server.name} Liberty configuration variable

When running Liberty servers in the Bluemix Container Service the CONTAINER_HOST and CONTAINER_NAME are already set for you.

When running Liberty servers in Liberty Docker Collectives the CONTAINER_HOST and CONTAINER_NAME are already set for you.

When running Liberty servers in Docker containers or other virtual environments you manage, set the CONTAINER_HOST and CONTAINER_NAME environment variables when you run your image. This will ensure that the Logstash collector or Bluemix log collector tags the records that they send with the appropriate host and container name, which aids in problem determination. If you do not set these environment variables, you might find it difficult to determine which container sent which events when using a dashboard that shows events from multiple containers. When starting a Docker container, use a command similar to the following:
docker run -d -e LICENSE=accept -e CONTAINER_NAME=yourContainerName -e CONTAINER_HOST=yourContainerHost --name=yourContainerName yourImageName

Icon that indicates the type of topic Concept topic



Timestamp icon Last updated: Friday, 16 September 2016
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=cord&product=was-nd-dist&topic=cwlp_analytics_collectors
File name: cwlp_analytics_collectors.html