In large clusters, job querying can grow quickly. If your site sees a lot of high traffic job querying, you can tune LSF to limit the number of job queries that mbatchd can handle. This helps decrease the load on the master host.
If a job information query is sent after the limit has been reached, an error message ("Batch system concurrent query limit exceeded") is displayed and mbatchd keeps retrying, in one second intervals. If the number of job queries later drops below the limit, mbatchd handles the query.
If mbatchd is not using multithreading, the value of MAX_CONCURRENT_JOB_QUERY is always the maximum number of job queries in the cluster.
If mbatchd is using multithreading (defined by the parameter LSB_QUERY_PORT in lsf.conf ), the number of job queries in the cluster can temporarily become higher than the number specified by MAX_CONCURRENT_JOB_QUERY.
This increase in the total number of job queries is possible because the value of MAX_CONCURRENT_JOB_QUERY actually sets the maximum number of queries that can be handled by each child mbatchd that is forked by mbatchd. When the new child mbatchd starts, it handles new queries, but the old child mbatchd continues to run until all the old queries are finished. It is possible that the total number of job queries can be as high as MAX_CONCURRENT_JOB_QUERY multiplied by the number of child daemons forked by mbatchd.
Specifies the maximum number of job queries that can be handled by mbatchd. Valid values are positive integers between 1 and 100. The default value is unlimited.