Remote execution issues #22937
Description
Describe the bug
Pants client seems to be hanging after ~50 completed RE requests on one of our larger demos involving ~15k targets.
It looks like remote exection implementation is less stable in pants and the RE client version is an older one. We are using pants 2.28.
Here is the list of workarounds we had to implement to get it to the current state, most of which are also described in https://github.com/gruzewski/pants-remote-exec-issues
In particular, to get it to work with BB we had to do the following:
- downgrade the version of the remote executor from ghcr.io/buildbarn/bb-worker:20250820T081613Z-efef252 to ghcr.io/buildbarn/bb-worker:20250526T162405Z-20574f0
- Needed to remove the "platform" settings from the remote worker configuration. Otherwise, no workers are being picked up by pants client.
- Needed to explicitly set the following environment variables in the remote worker:
environment_variables: { PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" }
Pants version
2.28
OS
Linux, MacOS not tested
Additional info
Our example have been mostly already provided by Scott Venier called pants_scaling. Small modifications have been made to enable remote environment and remote execution. This example is locking up.
Useful info on some of the issues and workarounds - https://github.com/gruzewski/pants-remote-exec-issues