IllegalArgumentException when starting a QueryBatcher

[ralfhergert]

When we run our integration test-suite especially the tests using the DataMovementManager and a QueryBatcher. The LinkedBlockingQueue used in QueryBatcherImpl#QueryThreadPoolExecutor (QueryBatcherImpl line 1132) is throwing an IllegalArgumentException because the poolSize is calculated to be -1. The line in question is QueryBatcherImpl#1132:

new LinkedBlockingQueue<>((forestsLength * docToUriBatchRatio * 2) +  threadCount)

with forestsLength=1, docToUriBatchRatio=-1, threadCount=1

It is wierd that docToUriBatchRatio is -1. Having a look at QueryBatcherImpl#withBatchSize (line 296), which is called in our case:

public QueryBatcher withBatchSize(int docBatchSize) {
    if (docBatchSize > this.maxUriBatchSize) {
      logger.debug("docBatchSize is beyond maxDocBatchSize, which is {}.", this.maxUriBatchSize);
    }
    if (docBatchSize < 1) {
      throw new IllegalArgumentException("docBatchSize cannot be less than 1");
    }
    requireNotStarted();
    super.withBatchSize(docBatchSize);
    this.docToUriBatchRatio = Math.min(this.maxDocToUriBatchRatio, this.maxUriBatchSize / docBatchSize);
    if (this.docToUriBatchRatio == 0) {
      this.docToUriBatchRatio = 1;
    }
    return this;
  }

it happens in line 305 due to maxDocToUriBatchRatio being -1. This -1 is taken from the forest-response given by the MarkLogic server in DataMovementServices#initConfig (line 64). (The else-branch in line 67 is not visited in our case.)

Solutions:
It feels like the maxDocToUriBatchRatio=-1 given by the MarkLogic server is supposed to mean something like "undefined" or "unlimited". Using the -1 in the calculation for the thread pool size seems oddly wrong. If so intended, we would appreciate a proper error message like "QueryBatcher can not be started ...". Currently just an IllegalArgumentException without any message is thrown by the Java's native LinkedBlockingQueue.

[rjrudin]

Thanks @ralfhergert for the report - which versions of the Java Client and MarkLogic are you using?

[ralfhergert]

Hi @rjrudin we are using the marklogic-java-client in version 8.0.0 and the MarkLogic server in version 11.3.1

[rjrudin]

Thanks - can you also share the code you're using to construct a QueryBatcher, along with the number of forests in your database?

[rjrudin]

@ralfhergert The value of -1 originates from this code in the server:

let $maxDocToUriBatchRatio :=
  floor(($availableHostCount * xdmp:server-max-threads(xdmp:server())) idiv count($forestIds)) - 1

My hunch is you have a large number of forests that ends up exceeding the server-max-threads value.

I'm able to reproduce the error with a test against a database having 3 forests - I adjust the "Threads" value in the Admin UI for the database to 2, and then I get the error.

Either the Java Client or server should definitely handle this so an error isn't thrown, but in the near term - I'm wondering if adjusting your Threads value for the server will get around the problem? Also curious what your Threads count and your forest count is.

[rjrudin]

@ralfhergert We may make a server fix for this, but in the near term, the 8.1.0 Java Client release will adjust the value from -1 to 1 when it receives -1 from the server. Testing has shown that this will avoid the problem in the scenario where the app server has fewer threads than the number of forests in the associated database.