A large number of amqrmppa processes spawn very quickly in IBM Websphere MQ system
You see a large number of amqrmppa channel processes that are spawned very quickly in a WebSphere MQ system
Symptom
A
large number of amqrmppa processes are created in a short period of
time. Their parent process is a runmqlsr process. Sometimes there are
multiple runmqlsr processes that were spawned, which has this runmqlsr
process as a parent process as well. The number of spawned processes
might keep increasing and sometimes reaches the maxuproc limit of the
operating system, hence it could cause a queue manager outage.
Cause
The
cause of the large number of amqrmppa and runmqlsr processes is that
these processes are attached to an old set of shared memory segments.
Diagnosing the problem
When
a MQCONN request is received by the runmqlsr process it seems that it
has no active amqrmppa processes listed in its old shared memory
segments and so runmqlsr starts a new amqrmppa process. However it
fails to communicate with the newly started amqrmppa process because the
amqrmppa attaches to the new shared memory segments, and so it instead
starts a thread and writes an error message. The amqrmppa process is
left orphaned waiting for a connect request which will never arrive.
This process is repeated for each MQCONN request, with a new amqrmppa
process started for each client MQCONN.
As the initial trigger for the failure, one scenario could be that the last time the queue manager was ended the system's shared memory was manually cleaned-up using ipcrm, but the listener was missed and was left running. The queue manager was subsequently restarted and had no knowledge of the old listener process.
The recommended workaround solution for this issue is to kill the old runmqlsr process and bring up a new MQ listener process either by "runmqlsr" command " or "start listener" using runmqsc interface like below:
$ runmqlsr -t tcp -p -m
or
$ runmqsc
start listener('listener name')
As the initial trigger for the failure, one scenario could be that the last time the queue manager was ended the system's shared memory was manually cleaned-up using ipcrm, but the listener was missed and was left running. The queue manager was subsequently restarted and had no knowledge of the old listener process.
The recommended workaround solution for this issue is to kill the old runmqlsr process and bring up a new MQ listener process either by "runmqlsr" command " or "start listener" using runmqsc interface like below:
$ runmqlsr -t tcp -p -m
or
$ runmqsc
start listener('listener name')