Tags: instructlab/training
Tags
fix(torchrun): Omit empty arguments and correct nproc_per_node type (#… …661) * fix(torchrun): Omit empty arguments and correct nproc_per_node type The command generation logic is updated to dynamically build the torchrun command, excluding arguments that are empty or None. This prevents them from overriding environment variables, ensuring that torchrun can correctly inherit its configuration. An exception is made for integer arguments where 0 is a valid value. Additionally, the nproc_per_node argument type has been changed from int to str to support special values accepted by PyTorch, such as 'auto', 'gpu', and 'cpu'. Reference: https://github.com/pytorch/pytorch/blob/main/torch/distributed/run.py#L77-L88 Signed-off-by: Saad Zaher <szaher@redhat.com> * only dynamically add torchrun args & change rdzv_id type to str Signed-off-by: Saad Zaher <szaher@redhat.com> * fix smoke tests Signed-off-by: Saad Zaher <szaher@redhat.com> * Enable both dtypes str, int for nproc_per_node, rdzv_id Signed-off-by: Saad Zaher <szaher@redhat.com> * Use python3.11 style for pydatnic model Signed-off-by: Saad Zaher <szaher@redhat.com> * add all torchrun args and validate them Signed-off-by: Saad Zaher <szaher@redhat.com> * Remove non-required dependencies Signed-off-by: Saad Zaher <szaher@redhat.com> * update datatypes only Signed-off-by: Saad Zaher <szaher@redhat.com> * replace _ with - when passing torchrun args Signed-off-by: Saad Zaher <szaher@redhat.com> * make nproc_per_node to only accept gpu or int Signed-off-by: Saad Zaher <szaher@redhat.com> * add master_{addr, port} validate args Signed-off-by: Saad Zaher <szaher@redhat.com> * check for not set or empty rdzv endpoint Signed-off-by: Saad Zaher <szaher@redhat.com> * fix formatting error Signed-off-by: Saad Zaher <szaher@redhat.com> * Update src/instructlab/training/config.py Signed-off-by: Saad Zaher <szaher@redhat.com> * Update tests/smoke/test_train.py Signed-off-by: Saad Zaher <szaher@redhat.com> * Update src/instructlab/training/main_ds.py Signed-off-by: Saad Zaher <szaher@redhat.com> * fixes indentation Signed-off-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> * formatting * add standalone as the fallback when neither master_addr nor rdzv_endpoint are provided Signed-off-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> * clarify rdzv-backend arg --------- Signed-off-by: Saad Zaher <szaher@redhat.com> Signed-off-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com>
PreviousNext