Managing pipelines
Pipelines are the heart of the system. They are where the processing
takes place: where entities are resolved, where relationships are detected,
and where alerts are generated. Pipelines are the primary way that data is
loaded into the entity database. Managing pipelines is an ongoing operational
task that involves configuring pipelines, starting and stopping pipelines,
monitoring pipelines, and routing messages from pipelines to other pipelines,
nodes, or external systems.
- Starting pipelines
Before a pipeline can receive and process data, it must be started.
It is common to run multiple pipelines to increase data throughput or process
different types of source data. Use these steps to start a pipeline or re-start
a pipeline that is down.
- Stopping pipelines
Stopping a pipeline means changing its status from active and open
for processing data to inactive and closed to incoming data. You can manually
stop one pipeline at a time. Use these instructions to stop a pipeline after
you make changes to the system configuration (then re-start the pipeline for
the configuration changes to take effect), if you are installing a hot fix
or an upgrade release, or if you are making configuration changes to the pipeline
node that hosts the pipeline.
- Configuring pipelines
When a pipeline starts up, it checks for a pipeline configuration
file to get its initial startup variables and configuration information necessary
to process incoming data. By default, when a pipeline is installed on the
pipeline node, the system also installs a default pipeline configuration file,
named pipeline.ini, that can be used by all pipelines on that pipeline node.
But some sections of this default file must be configured specifically to
the pipelines running on the pipeline node. so the pipeline has the proper
connections and access to the entity database. Use these instructions to configure
the pipeline configuration file.
- Pipelines registration
Before you can monitor status or route results for pipelines, you
must first register the pipelines in the Configuration Console. Registering
pipelines is not the same as installing or configuring a pipeline; it means
adding the pipeline to the Pipelines tab in the Configuration
Console.
- Configuring routing rules
Routing rules allow you to route the results of pipeline processing
or acquisition program to a database, a pipeline, or an external system. You
configure routing rules in the Configuration Console on the Routing
Rules tab, but you can only route from pipelines or acquisition
programs that have been registered with the application monitor. You can either
configure a new routing rule from scratch or based on an existing routing
rule.
- Deleting routing rules
Once a routing rule is configured, it cannot be edited; if you
need to correct or update information, you must delete the old routing rule
and configure a new one. You might also need to delete a routing rule that
is no longer needed or used. You can delete one or more configured routing
rules from the Routing Rules tab in the Configuration
Console.
- Pipeline status and statistics
Monitoring the status, statistics, and performance of a pipeline
is important to keeping it running, balancing data loads, and spotting potential
problems before they occur. The application monitor works with an SNMP agent
running on the pipeline node to communicate the status and statistics of pipelines,
which are then displayed in the Configuration Console.