IBM® Tivoli® Storage Manager (TSM) supports a wide variety of hardware mass storage devices that can be used as the fixed content system in support of a P8 fixed storage area.
The Content Engine stores the content of Documents that are checked into a TSM Fixed Storage Area as archive objects on the TSM server. Each Content Transfer object of the Document is stored as a separate named object on the TSM server. Retention is applied to the named TSM objects based on the TSM management class associated with the Fixed Storage Area. Content Engine allows either an Event-Based or a Chronological management class to be associated with a TSM Fixed Storage Area.
TivoliFixedContentDevice represents a single TSM server instance. The configuration data contains the IP address, port number of the TSM server, and a client node name (the Content Engine in this case) and password. This data is written to an options file, which is used to connect to the TSM device. See the Options files section for more information on the TSM options files.
FixedStorageArea represents the staging area for the TSM device. Content is always uploaded to this staging area before being migrated to the TSM device.
The Fixed Content Provider (FCP) pool contains a set of FCPs for a given Fixed Content Device.
For information on Fixed Content Device properties for Tivoli Storage Manager, see the Fixed Content Devices properties (General tab).
Content Engine supports static retention, which is applied when a document is checked in and which prevents deletion until after the retention period expires. When connected to a Tivoli Storage Manager server, CE can use either scheme that is available on the TSM server: Event-Based Retention, or Chronological Retention coupled with the Hold/Release capability. CE leverages the behavior of the TSM management class to control when a document can be deleted. Unless Chronological Retention is explicitly required, you should use Event-Based Retention for the following reasons:
Content Engine always remains in control of the document deletion process. So even though TSM servers support the automatic deletion of objects after their retention period has expired, CE will not allow it until after the retention period has expired AND a user specifically deletes the associated document from the CE. When both conditions are met, CE relinquishes control of the TSM object, allowing TSM to automatically delete it.
To control retention while suppressing automatic deletion, Content Engine uses the specific parameters of the two retention schemes available in TSM as shown in the following table.
Parameter and purpose | Value for chronological retention | Value for event-based retention |
---|---|---|
RETINIT Determines when to initiate the retention period defined in the RETVER attribute |
CREATION (start counting when the object is archived) |
EVENT (start counting when the event is triggered) |
RETVER Number of days to retain the archived object after retention is initiated |
1 to 30,000 days, or NOLIMIT | 0 to 30,000 days |
RETMIN Minimum number of days to retain the archived object |
Not applicable | 0 to 30,000 days |
An EBR management class has two retention settings that combine to protect a TSM object. The RETMIN parameter defines the minimum retention period for the object from creation time. The RETVER parameter defines the retention from the time of activation. As long as the EBR "activate" event is not issued, the object is protected from deletion, no matter what the value is for RETMIN. When the EBR "activate" event is issued, the retention period defined in RETVER is applied from that moment. It might or it might not overlap with RETMIN. If RETVER=0, for example, and the "activate" event is issued during the RETMIN period, the object continues to be protected until RETMIN expires.
Content Engine uses the behavior of the EBR management class to control when the TSM object can be deleted. If you set in TSM an EBR class with RETMIN=365 and REVER=0, then you can create in CE a TSM Fixed Storage Area that uses the EBR management class. When a document is checked into this Fixed Storage Area, CE calculates the delete date (today + 365 days) and writes it to the ContentRetentionDate attribute of the document. When a user wants to delete the document more than one year later, the CE delete call sends an "activate" event to the TSM server, and because RETVER=0, the retention expires immediately, which allows the object to be deleted automatically by the TSM background task.
When you create a TSM management class to support event-based retention, use the following parameters:
RETINIT=EVENT RETVER=0 RETMIN=<actual retention period>
NOTE The minimum value that can be assigned to RETVER in an EBR Management Class is 0, and no hold is placed by CE on the TSM objects binding to such a class. Setting RETVER=0 and RETMIN=0 provides a way to create a Fixed Storage Area with no storage-level retention while protecting the document from tampering and deletion. The CE application remains in full control of retention and deletion. This is the preferred setting for Records Management use cases.
A chronological management class uses the RETVER parameter to protect a TSM object. The RETVER parameter defines the number of days that the object should be retained starting from the date that the object is stored on the TSM server.
TSM automatically deletes objects when their retention period (RETVER) has expired. To enable CE to retain control of the objects with chronological retention, the CE places a hold on the object after creation. Later, when the document is finally eligible for deletion, after the RETVER has expired and a user deletes the associated document, the CE releases the hold and allows the TSM automatic deletion process to delete the object.
NOTE Placing a hold on an object requires a TSM retention protection license, so the chronological retention scheme cannot be used without this license. CE filters chronological management classes out of the CmTivoliManagementClassSet if the retention protection flag in the Fixed Content Device is OFF.
When you create a TSM management class to support chronological retention, use the following parameters:
RETINIT=CREATION RETVER=<actual retention period>
NOTE The minimum value that can be assigned to RETVER in a Chronological Management Class for CE to be able to place a hold and prevent automatic deletion is 1 day. Content Engine automatically filters out all Chronological Management Classes with RETVER=0. Therefore, it is not possible to create a TSM Fixed Storage Area with no retention, based on a Chronological Retention Management Class.
CE creates or updates the TSM options files whenever a Fixed Content Device is created or altered. The shared options files define connection information for one or more TSM servers, and must be accessible to every CE instance in the domain.
NOTE The shared TSM options files are in the directory referenced by Configuration Files Share property of the the TSM Fixed Content Device. These files are critical to the operation of CE and must not be deleted. They should also be included in all system backups and disaster recovery procedures.
On UNIX servers, the DSMI Directory property of the TSM FCD must be set to the location of the TSM client software directory that contains the dms.sys file (for example, /opt/tivoli/tsm/client/ba/bin). This directory should also be included in all system backups and disaster recovery procedures.
On Windows servers, the DSMI Directory property must not be set.
Each UNIX server where the Content Engine is running must have the TSM client software installed. Within the TSM client installation folders there is a dsm.sys file, which is used by all TSM clients on that server, including the Content Engine, to connect to TSM.
The CE maintains a shared version of the dsm.sys file in the folder defined by the Configuration Files Share property of the TSM Fixed Content Device. The shared file is named IBMFileNetP8.dsm.sys. Each CE instance merges the changes it makes (and changes that other CE instances make) to the shared dsm.sys file into the local dsm.sys file on the server, which keeps all local dsm.sys files in sync with the CE shared dsm.sys file.
Note that it might be necessary to mount the shared options file folder with the uid=(username) option (where username is the user account that CE is running as). Without this option, CE could create the shared options files successfully on the mounted folder, but the files might not grant the CE user full control access, and CE would not be able to write the files.
On Windows servers, each TSM Fixed Content Device has a separate options file that is located in the folder referenced by the Configuration Files Share property. The options filename consists of the object identifier of the TSM Fixed Content Device (with the leading and trailing brackets removed) followed by ".opt" file extension. On Windows an additional file is generated (named IBMFileNetP8.opt), which is normally empty.
The Content Engine 4.5.1 release supports writing and retrieving content that is stored on a magnetic tape device in a TSM 6.1 server. Typically, CE writes content to the magnetic storage tier in the TSM server. Later, TSM migrates the content to a tape device in the background. The migration to tape is not apparent to P8 users. Retrieval from tape might also not be apparent, depending on the state of the tape. Because tape is a sequential access medium, response times from reading the tape are somewhat slower.
If the wait is too long, the TSM server cancels the request and sends an exception to the Content Engine, which asks you to retry the retrieval later:
The attempt to read document id [{0}], element [{1}] failed. This may be due to a TSM unmounted tape volume. Please try again later, allowing sufficient time for tape to be mounted.
Content Engine limits the maximum number of users that can be waiting for tape to be mounted by means of the Concurrent Tape Readers configuration parameter for TSM Fixed Content Devices. If the maximum number of users are already waiting for a tape to be mounted, Content Engine denies any new retrieval requests that require a tape to be mounted. Note that Content Engine does not limit the number of users that can be reading from tape, just the number of users that can be waiting for tape to be mounted.
The Concurrent Tape Readers configuration parameter represents the maximum number of concurrent users that are allowed to wait for a tape to be mounted for each CE instance, for each TSM Fixed Content Device. That is, each instance of CE can have up to this number of users waiting for a tape to be mounted for each TSM FCD. The default value of one limits each CE instance to one user waiting for a tape to be mounted for each TSM Fixed Content Device.
If your tape library contains multiple tape drives, the minimum value for this parameter should be equal to the number of drives in the library. You can also increase the value of this parameter if the tape is either normally already mounted (you have a small number of tapes) or if the tape can be mounted quickly. Depending on how your data is collocated on tapes and the performance of your tape library, you can increase the value of this parameter, especially if you get a high number of retrieval failures for documents on tape, and you believe that your tape library should be able to handle more of these requests.
Each waiting user takes up one TSM session and one item in the Fixed Content Provider Pool. The TSM MAXSESSIONS and Content Engine FCP Pool Max In Use parameters might need to be adjusted to account for users who are waiting for a tape to be mounted.
When you set the Concurrent Tape Readers parameter, follow these guidelines:
When you retrieve content that is stored on a TSM server, Content Engine first attempts to obtain the content without waiting for a tape to be mounted. If the content is not stored on tape, it is retrieved and returned to you immediately. If the content is stored on tape, CE determines if it can allow you to wait for the tape to be mounted, based on Concurrent Tape Readers parameter and the current number of waiting users.
If the number of users that are already waiting is equal to or greater than the Concurrent Tape Readers parameter, CE produces a CONTENT_RETRYABLE_READ_FAILURE exception that rejects the read request and asks you to retry the retrieval at a later time. If you are allowed to wait, CE tries to obtain the content. In this case, the tape is either mounted successfully and the content is returned to you, or the TSM server times out the mount request. Two TSM server parameters control the timeout:
These two timeout parameters should be adjusted to prevent users from waiting for an excessively long time for a tape to be mounted.
Because the CE server uses the same TSM client NODE definition for all its connections to the TSM server, two additional options must be specified on this node to allow it to process requests from multiple concurrent CE users.
MAXNUMMP set to 999
KEEPMP set to NO
The MAXNUMMP parameter increases the maximum number of tape mount points allowed for a node to handle multiple concurrent requests from multiple CE servers that all use the same TSM node definition. Technically, the MAXNUMMP value should be the same as the MAXSESSIONS value, but setting it to a high value, such as 999, is sufficient.
The KEEPMP parameter is set to NO so that tape mount points can be released as quickly as possible and can be made available to other sessions that need them.
Finally, for maximum performance, your TSM server should have a disk storage pool in front of the storage pool that is used for tape with an appropriate migration policy to migrate items from the disk pool to the tape pool, as required. Thus, the TSM client NODE used by your FCD should be configured to use this disk pool as its primary storage pool.
Without a disk storage pool, you risk running out of storage space in the CE staging area of the TSM Fixed Storage Area during periods of high content ingestion. Because P8 documents are staged locally in the TSM Fixed Storage Area front-end before being queued up for migration to the TSM server itself, P8 must provide enough local storage to accommodate the accumulation of documents before migration. The slower the migration—as might be expected when writing directly to a sequential storage device such as a tape drive—the higher the storage space requirement in the front-end, and the larger the migration queue in the CE Object Store.
Therefore, to maximize ingestion performance while ensuring that documents can be transferred fast enough from the staging area before hitting quota limits, having CE write to a disk storage pool is preferable to writing directly to tape.
Content Engine can achieve high migration rates by scaling both up and out:
A Content Cache is strongly recommended because it can greatly improve retrieval performance. If the content stored on tape is going to be read multiple times, the Content Cache copies the content from TSM tape to a shared file system (Content Cache Area) on the first retrieval, and all subsequent retrievals read the content from the file system until the content is aged out of the cache.
The Content Cache also has its own inherent timeout mechanism. If a retrieval from tape takes longer than this internal timeout, the content cache returns a message that instructs you to try your retrieval again later. In the meantime, the content cache continues to try to retrieve the tape content so that the content is available in its cache area the next time you make your retrieval request.
Create a Content Cache Area for your site using FileNet Enterprise Manager. Make sure you increase the Maximum Size and Maximum Number of Elements for your Cache Area because the default values for these settings are typically too small. After this cache area is created, make sure you then select it in the Content Cache tab of your Site properties. Note that when specifying a content cache for a site, you must select the Override inherited setting checkbox. Then you can select the cache area to use for your site. See Content cache areas for more details.
Finally, the Fixed Storage Areas in the object store that you have associated to TSM FCDs must be set to Content Caching Allowed, instead of Cross-site Only. This setting allows tape content to be cached for your site while it is being retrieved from the TSM server.