Data transformers

Data transformers convert raw cluster data in the relational database into a format usable for reporting and analysis.

About data transformers

The LSF and license data is logged in the relational database in a raw format. At regular intervals, the data transformer converts this data to a usable format.

Logging levels

There are logging levels that determine the detail of messages that the data transformers record in the log files. In decreasing level of detail, these levels are ALL (all messages), TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF (no messages).

By default, the data transformers log messages of INFO level or higher (that is, all INFO, WARN, ERROR, and FATAL messages).

The data transformer log files are located in the datatransformer subdirectory of your Platform Analytics server log directory:

  • UNIX: ANALYTICS_TOP/log/datatransformer

  • Windows: ANALYTICS_TOP\log\datatransformer

Default behavior

Data transformers convert data at a regular 10 minutes intervals. The following is a list of the data transformers and the database tables in which the data transformers generate the data:


Data transformer name

Transformed database tables

ClusterCapacity

RPT_CLUSTER_CAPACITY_RAW

FlexlmLicUsage

RPT_FLEXLM_LICUSAGE_RAW

Hardware

RPT_HARDWARE_RAW

RPT_HARDWARE_DAY

WorkloadAccounting and Resource Usage

RPT_JOBMART_RAW

RPT_JOBMART_DAY

WorkloadStatistics

RPT_WORKLOAD_STATISTICS_RAW


Data transformer interactions

Data transformers convert raw cluster data from the data tables through the server data sources in the relational database into a format usable for reporting and analysis.

The following diagram illustrates the interaction between the data transformers and other components.

Configuration to modify data transformer behavior


Action

Configuration files

Parameter and syntax

Specify the default log level of your data transformer log files.

log4j.properties

File location: ANALYTICS_TOP/conf

log4j.properties

log4j.appender.${datatransformer}=log_level, ${datatransformer}

where

  • log_level is the default log level of your data transformer log files.

Specify the log level of the log file for the specified data transformer.

log4j.logger.transformer.datatransformer_name=log_level

where

  • datatransformer_name is the name of the data transformer.

  • log_level is the log level of your data transformer log file.

For example, to set hardware to ERROR, add the following line to log4j.properties:

log4j.logger.transformer.hardware.loader=ERROR

Specify the log level of the log file for the Extractor or Loader in the ETL flow for the specified data transformer.

log4j.logger.transformer.datatransformer_name.component=log_level

where

  • datatransformer_name is the name of the data transformer.

  • component is the ETL flow component. Use extractor to specify the Extractor and use loader to specify the Loader in the ETL flow.

  • log_level is the log level of your data transformer Extractor or Loader log files.

For example, to set the Loader in WorkloadAccounting to WARN, add the following line to log4j.properties:

log4j.logger.transformer.WorkloadAccounting.loader=WARN


The data transformer only logs messages of the same or lower level of detail as log_level. Therefore, if you change the log level to ERROR, the data transformer will only log ERROR and FATAL messages.

Data transformer actions

Data transformers are installed as scheduled tasks. Change the schedule of data transformer services as you would for scheduled tasks (see Scheduled tasks).