| IJ58836 |
High Importance
|
Negative lookups are queued to gateway after cache drop on client
(show details)
| Symptom |
Unexpected Results |
| Environment |
Linux Only |
| Trigger |
run negative lookup (ls on non-existant file ) from non-gw node and clear cache from non-gw |
| Workaround |
None |
|
6.0.1.1 |
AFM |
| IJ58969 |
Suggested |
When a symlink is accessed on another node, then deleted on a different node and on the first node the Linux struct inode is reused, some memory might not be deallocated.
(show details)
| Symptom |
Unexpected Results/Behavior |
| Environment |
ALL Linux OS environments |
| Trigger |
The scenario is to have a symlink access it on node A. Then delete the symlink on a different node. When a different file is created and accessed on node A, some memory allocated when accessing the symlink |
| Workaround |
There is no workaround. |
|
6.0.1.1 |
All Scale Users |
| IJ58838 |
High Importance
|
mmsmb exportacl list command will list the output in SIDs instead of user/group names when there are more than 6000 unique SIDs in the system.
This is because the command uses rpcclient to resolve SIDs to user/group names, and rpcclient has a limit of 6000 SIDs that it can resolve at once.
When there are more than 6000 unique SIDs, rpcclient fails to resolve them, and mmsmb exportacl list will show the SIDs instead of user/group names.
Warning message is seen in the logs:
mmsmb exportacl list: Execution response: Usage: rpcclient [OPTION...] BINDING-STRING|HOST
(show details)
| Symptom |
Error output/message
Warning message is seen in the logs:
mmsmb exportacl list: Execution response: Usage: rpcclient [OPTION...] BINDING-STRING|HOST |
| Environment |
Linux Only |
| Trigger |
Customer having more than 6000 unique SIDs in the system. |
| Workaround |
Customer can still see the SIDs in the output of mmsmb exportacl list command, but they will not be able to see the user/group names. |
|
6.0.1.1 |
CES |
| IJ58970 |
High Importance
|
In a heavy workload that opens and closes an excessive number of files and directories, a kernel leak may continuously grow over days and result in the depletion of the Windows non-paged pool, requiring a node reboot to restore it to a healthy state.
(show details)
| Symptom |
Unexpected Results/Behavior. |
| Environment |
Windows/x86_64 only |
| Trigger |
Heavy workload that opens and closes an excessive number of files and directories over an extended time. |
| Workaround |
None |
|
6.0.1.1 |
Windows |
| IJ58917 |
High Importance
|
AFM does not failover to another available NFS Server in the export map using afmFailoverMap casing the deadlock
(show details)
| Symptom |
Deadlock |
| Environment |
Linux Only |
| Trigger |
NFS failover with afmFailoverMap |
| Workaround |
None |
|
6.0.1.1 |
AFM |
| IJ58919 |
Medium Importance |
If a node is promoted to quorum node by "mmchnode --quorum" after one or more other cluster nodes are already expelledand "disablePersistExpelList=false" (the default), the expelled node list is not set up on the new quorum node, so if the new quorum node subsequently becomes the leader (cluster manager), the expelled nodes will be allowed to rejoin.
(show details)
| Symptom |
Nodes may unexpectedly rejoin the cluster. |
| Environment |
All |
| Trigger |
Newly-added quorum node and persistent expelled node(s) |
| Workaround |
None |
|
6.0.1.1 |
GPFS core |
| IJ58971 |
Suggested |
Various deficiencies in Multitenant Security RPC (MTSec) Filtering were not resolved in 6.0.1.0 GA (where the feature first appeared).
- mtSecPrintResetInterval and mtSecPrintLimitReset should limit the number of messages output to mmfs.log during an attack or a defect, but the filtering reset limits did not work properly.
- Changed to do rate limited on a per-node basis
- Fix incorrect aborting from bcast_takeover_query & doRfacCleanup
- Removed some duplicated messages
- Improved reporting for some NSD and related MTSec errors to show the offending sender node instead of NODE_NONE
(show details)
| Symptom |
Warning messages in mmfs.log |
| Environment |
All |
| Trigger |
Remote cluster storage clients generating impermissible RPCs. |
| Workaround |
N/A |
|
6.0.1.1 |
GPFS core |
| IJ58840 |
High Importance
|
Under certain workloads, the token manager nodes can experience a contention on the aTokeClassMutex, leading to performance degradation. All token revokes for the same token type on each token server has to acquire the same mutex. The symptom would mmdiag --waiters on the token manager nodes shows many short waiters for aTokenClassMutex.
(show details)
| Symptom |
Performance Impact/Degradation |
| Environment |
ALL Operating System environments |
| Trigger |
This is hit with a workload that triggers many token revokes. Having fewer token manager nodes might exacerbate the problem, since the mutex is local to each token server. |
| Workaround |
Modifying the workload to reduce token contention could avoid this problem. Also adding more manager nodes to spread the token manager load could reduce this problem. |
|
6.0.1.1 |
All Scale Users |
| IJ58920 |
High Importance
|
Revalidation Performance Degradation After a Fileset Is Converted to Primary Mode. This is due to Open requests being sent to the target/secondary.
(show details)
| Symptom |
Performance Impact |
| Environment |
Linux Only |
| Trigger |
This issue occurs when a fileset is converted to an AFM Primary mode fileset. |
| Workaround |
None |
|
6.0.1.1 |
AFM |
| IJ58972 |
Suggested |
When vlans are used on IP over IB devices with Vlans an event will be raised unless vlans are used on a bond.
(show details)
| Symptom |
Error output/message |
| Environment |
ALL Linux OS environments |
| Trigger |
When vlans are used on IP over IB devices with Vlans an event will be raised unless vlans are used on a bond. |
| Workaround |
None |
|
6.0.1.1 |
System Health |
| IJ58921 |
High Importance
|
Revalidation Performance Degradation After a Fileset Is Converted to Primary Mode. This is due to Open requests being sent to the target/secondary.
(show details)
| Symptom |
Performance Impact |
| Environment |
Linux Only |
| Trigger |
This issue occurs when a fileset is converted to an AFM Primary mode fileset. |
| Workaround |
None |
|
6.0.1.1 |
AFM |
| IJ58973 |
High Importance
|
When Call Home is configured to use a custom CA bundle (e.g. via the CURL_CA_BUNDLE environment variable set in the mmsysmon service configuration), upload operations that transmit data fail with the following SSL error:
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
All other Call Home operations succeed, including connectivity tests performed by "mmcallhome test connection" succeed. Only the actual data upload path is affected.
(show details)
| Symptom |
- Error output/message
- Component Level Outage |
| Environment |
ALL Linux OS environments |
| Trigger |
All of the following conditions must be met:
1. IBM Storage Scale 5.2.2.1 or later is installed.
2. A custom CA bundle is configured for the mmsysmon service via the CURL_CA_BUNDLE environment variable (typically in /etc/systemd/system/mmsysmon.service.d/).
3. The environment uses a self-signed or enterprise-issued CA certificate that is not present in the system default CA store.
4. A Call Home upload operation is triggered (automatic daily/weekly gather-and-send, or a manual mmcallhome send). |
| Workaround |
On the node designated as the Call Home server, apply the following manual code change in
/usr/lpp/mmfs/lib/mmsysmon/callhome/Callhome.py:
@@ -173,6 +173,8 @@ class CallhomeJobPrototype(object):
"""Performs an upload attempt and returns its rc and stdout"""
self.logger.info("Running command: %s", " ".join(command)
CURL_ERROR_PREFIX = "curl: ("
+ fullenv = dict(os.environ)
+ fullenv.update({"https_proxy": CallhomeConfig().proxyCurlEnvParameter})
try:
proc = subprocess.Popen(
command,
@@ -181,7 +183,7 @@ class CallhomeJobPrototype(object):
stderr=subprocess.PIPE,
close_fds=True,
encoding="utf-8",
- env={"https_proxy": CallhomeConfig().proxyCurlEnvParameter},
+ env=fullenv,
)
lastTrailingLine = ""
# for preventing the same percentage being written in log file
Then restart the mmsysmon daemon:
mmsysmoncontrol restart |
|
6.0.1.1 |
Callhome |
| IJ58899 |
Medium Importance |
Inode used not decreasing with auto eviction & inode quota enabled
(show details)
| Symptom |
Slow or no progress in release of inodes with auto-eviction |
| Environment |
All OS environments |
| Trigger |
Auto eveiction is configured on fileset, emptyPtrashErrors.xxxx accumulate over time. |
| Workaround |
Manually analyze .ptrash & .afm directory and delete emptyPtrashErrors.xxxx files |
|
6.0.1.1 |
AFM |
| IJ58925 |
High Importance
|
empty directory and symlink mtime is not in sync with remote.
(show details)
| Symptom |
Unexpected Results |
| Environment |
Linux Only |
| Trigger |
run dmapi prefetch for empty dir or symlink. |
| Workaround |
run mmafmlocal rm -rf <emptyDir / symlink>
then run ls <emptyDir / symlink></emptyDir> |
|
6.0.1.1 |
AFM |
| IJ58843 |
Suggested |
mmbackup may incorrectly report successful completion when some files actually failed to backup. This occurs due to three specific issues: (1) The ANS4047E error from TSM (indicating a read error on a file that is skipped) is not included in the error filter list, causing mmbackup to miss this failure condition. (2) When dsmc reports "Total number of objects failed: <n>", mmbackup does not correctly count these failures. (3) mmbackup does not report "some files are missing" when only file updates (metadata changes) fail, even though backup operations for new or changed file content may have succeeded.
(show details)
| Symptom |
Error output/message |
| Environment |
All platforms that support mmbackup |
| Trigger |
This issue affects users running mmbackup operations when any of the following conditions occur:
1) Files encounter read errors during backup, generating ANS4047E errors from TSM. This can happen when:
- Files are being modified during the backup operation
- File permissions change during backup
- Files are deleted after being selected for backup but before being read
- I/O errors occur when reading file data
2) Protect client reports failed objects in its output with "Total number of objects failed: ", but these failures are not properly counted by mmbackup
3) During incremental backup operations where only file metadata needs to be updated (not file content), and these update operations fail |
| Workaround |
There is no workaround to prevent the incorrect success reporting. Users should manually review Protect dsmerror log files after each backup operation to verify that all expected files were successfully backed up, even when mmbackup reports success. Look specifically for ANS4047E errors and "Total number of objects failed" messages in the Protect client logs. |
|
6.0.1.1 |
mmbackup |
| IJ58974 |
High Importance
|
When multiple IOs with overlapping buffers are happening on same strip of a vtrack, the current code gets into a reconstruction deadlock due to a confusion in threads while deciding which buffers they should reconstruct & which buffers they should wait for others to reconstruct and complete the IO. We see long waiters saying "waiting for incompatible vdisk operations" in the dumps.
(show details)
| Symptom |
Hang/Deadlock/Unresponsiveness/Long Waiters |
| Environment |
ALL Operating System environments |
| Trigger |
While the IOs on same strip of same vtrack are in progress, a disk goes down triggering reconstruction in the IO threads. |
| Workaround |
NA |
|
6.0.1.1 |
ESS/GNR |
| IJ58894 |
High Importance
|
Log group resign happens when some error seen on the disk.
(show details)
| Symptom |
Abend/Crash |
| Environment |
Linux Only |
| Trigger |
Simply creating the new recovery group would have this problem and any medium writes with disk failures would end up causing the problem. |
| Workaround |
None |
|
6.0.1.1 |
GNR |
| IJ58983 |
High Importance
|
GPFS can halt due to logAssert(jniFlags & 0x02) or logAssert(id == myClusterId) in clusters where mmexpelnode has been invoked to expel other nodes and the persistent expel feature has not been disabled. This feature can be disabled with
(show details)
| Symptom |
Abend |
| Environment |
All |
| Trigger |
Numerous scenarios including unexpelling nodes or startup/restart/changes in existing nodes |
| Workaround |
The problem cannot occur in clusters where persistent expel feature is disabled. To disable thisfeature, run
mmchconfig "disablePersistExpelList=yes" -i
This must be run in both the local cluster (for any expelled nodes) and any storage clusters of which the expelled nodes were clients. |
|
6.0.1.1 |
GPFS core |
| IJ58839 |
High Importance
|
In large clusters with hundreds of nodes, network delays and inconsistencies in the delivery of QoS statistics and control messages can lead to race conditions. These race conditions may cause in-flight QoS configuration updates or refresh operations to hang while attempting to stop the existing QoS manager. Because the QoS shutdown thread holds filesystem-level mutexes, such a hang can cascade into cluster-wide waiters if other operations attempt to acquire the same filesystem-level locks.
(show details)
| Symptom |
Hang/Deadlock/Unresponsiveness/Long Waiters. |
| Environment |
ALL Operating System environments. |
| Trigger |
Frequent QoS configuration updates and refreshes in a large cluster with network delays, during heavy I/O workload. |
| Workaround |
None |
|
6.0.1.1 |
QoS |
| IJ59364 |
High Importance
|
mmfsckx reports false positive non-critical directory entry corruptions.
(show details)
| Symptom |
False positive output by mmfsckx |
| Environment |
All |
| Trigger |
Running mmfsckx on a busy system |
| Workaround |
Use offline fsck |
|
6.0.1.1 |
mmfsckx |
| IJ59361 |
Suggested |
A being deleted disk isn't removed from the performance disk address leading to FSErrBadDiskAddrIndex. That's because the performance pool isn't repaired when repairing the regular pool failed with E_NOREPLGRP when the file has two regular replicas.
(show details)
| Symptom |
Abend/Crash |
| Environment |
ALL Operating System environments |
| Trigger |
A file in a DAT file system has two regular replicas and the regular pool doesn't have enough failure groups. |
| Workaround |
Make sure there are enough failure groups for the reliable pool, and then run mmrestripefs or mmrestripefile. |
|
6.0.1.1 |
All Scale Users of a DAT file system |
| IJ58981 |
High Importance
|
Lookup of file is evicting file after data is appended to the uncached file.
(show details)
| Symptom |
Unexpected Results |
| Environment |
Linux Only |
| Trigger |
File data block in cache are zero after append operation is replicated to home. |
| Workaround |
None |
|
6.0.1.1 |
AFM |
| IJ59363 |
High Importance
|
GPFS registers a PR key on a disk that doesn't belong to GPFS any more
(show details)
| Symptom |
Unexpected Results/Behavior |
| Environment |
ALL Operating System environments |
| Trigger |
Because the underlying dev name can change on node reboot, CCR could get stale information and reserves key without checking the disks in CCR nsdmap have a valid nsdId. |
| Workaround |
No work around to prevent this problem from happening, one can remove the key after the fact. |
|
6.0.1.1 |
All Scale Users |
| IJ59463 |
HIPER |
Samba can deadlock if multiple SMB clients concurrently open the same files.
(show details)
| Symptom |
Hang/Deadlock/Unresponsiveness/Long Waiters |
| Environment |
ALL Linux OS environments |
| Trigger |
Open the same files multiple times through SMB clients connected to different nodes. The exact scenario is that two SMB clients connect to two different CES nodes and each open a different file. That grants oplocks for each open file.
If then both SMB clients open the file the other SMB client already has open, at the same time, then this results in two oplock break requests. If the oplock breaks overlap and each SMB client is not able to respond to the break request and both stay stuck in the open calls. A bug in the GPFS code prevents breaking out from this scenario. |
| Workaround |
The deadlock occurs due to the breaks happening for granted oplocks across multiple nodes. That provides multiple workarounds:
1) Not using oplocks at all avoid this problem at a possible cost of performance (mmsmb oplocks).
2) If data is only accessed through SMB, the cross-protocol integration can also be disabled (mmsmb gpfs:leases=no), but this is dangerous if the same data is also accessed outside of SMB.
3) Ensure that data for each SMB share is only accessed through one CES node removes the cross-node aspect. The simple approach would be moving all CES IP address to one node, but then that node needs to be able to handle the complete SMB workload. |
|
6.0.1.1 |
SMB |
| IJ58594 |
High Importance
|
Filesystem unmounts with error code 786 and message "encryptionHandled not set for inode" during I/O operations on encrypted filesystems accessed via remote mounts, resulting in filesystem unavailability and I/O failures.
(show details)
| Symptom |
Filesystem panic/unmount (error code 786) |
| Environment |
All (Linux, AIX, Windows)> |
| Trigger |
Encrypted filesystem with remote mount configuration during sustained I/O workload when token revoke operations occur (e.g., another node requesting access, periodic token management, or memory pressure triggering background sync/flush operations). |
| Workaround |
None |
|
6.0.1.0 |
File System Encryption |