Connecting to existing cluster fails to load imageName in code

[NewbieScriptWriter]

Before submitting a bug please check the following:

• [yes] Start a new R session
• [yes] Check your credentials file
• [yes] Install the latest doAzureParallel package
• [yes] Submit a minimal, reproducible example
• [yes] run sessionInfo()

If we connect to an existing cluster then jobs fail to run, likely due to missing "imageName" in code.

{"id":"1","commandLine":"/bin/bash -c "set -e; set -o pipefail; Rscript --no-save --no-environ --no-restore --no-site-file --verbose $AZ_BATCH_JOB_PREP_WORKING_DIR/worker.R 1 10 0 stop > $AZ_BATCH_TASK_ID.txt; wait"","userIdentity":{"autoUser":{"scope":"pool","elevationLevel":"admin"}},"environmentSettings"........removed code for privacy........{"filePattern":"../stdout.txt","destination":{"container":{"path":"stdout/1-........removed code for privacy........constraints":{"maxTaskRetryCount":3},"exitConditions":{"default":{"dependencyAction":"satisfy"}},"containerSettings":{"imageName":{},"containerRunOptions":"--rm"}}

Error in curl::curl_fetch_memory(url, handle = handle) :
Failure when receiving data from the peer

But if we create a new cluster and then connect to the existing cluster, then the code runs fine and the verbose output shows "imageName":"rocker/tidyverse:3.6.3":

{"id":"1","commandLine":"/bin/bash -c "set -e; set -o pipefail; Rscript --no-save --no-environ --no-restore --no-site-file --verbose $AZ_BATCH_JOB_PREP_WORKING_DIR/worker.R 1 1 0 stop > $AZ_BATCH_TASK_ID.txt; wait"","userIdentity":{"autoUser":{"scope":"pool","elevationLevel":"admin"}},"environmentSettings"........removed code for privacy........{"filePattern":"../stdout.txt","destination":{"container":{"path":"stdout/1-........removed code for privacy........constraints":{"maxTaskRetryCount":3},"exitConditions":{"default":{"dependencyAction":"satisfy"}},"containerSettings":{"imageName":"rocker/tidyverse:3.6.3","containerRunOptions":"--rm"}}

Steps we follow to reproduce the issue:

cluster already exists with several idle nodes

#------------------# Load the doAzureParallel library
library(doAzureParallel)
#------------------# Logging on
setVerbose(TRUE)
setHttpTraffic(TRUE)
#------------------# Set your credentials
setCredentials("credentials.json")
#------------------# Get existing cluster
cluster <- getCluster("TestCluster_2020", verbose = TRUE)
#------------------# Register the cluster as your parallel backend
registerDoAzureParallel(cluster)

#------------------# Test simulation inputs
mean_change = 1.001
volatility = 0.01
opening_price = 100

getClosingPrice <- function() {
days <- 1825 # ~ 5 years
movement <- rnorm(days, mean=mean_change, sd=volatility)
path <- cumprod(c(opening_price, movement))
closingPrice <- path[days]
return(closingPrice)
}

#------------------# PARALLEL Test simulation
opt <- list(chunkSize = 10)
start_p <- Sys.time()
closingPrices_p <- foreach(i = 1:10, .combine='c', .options.azure = opt) %dopar% {
replicate(10, getClosingPrice())
}
end_p <- Sys.time()

hist(closingPrices_p)

difftime(end_p, start_p, unit = "min")

So the only way to run the code against an existing cluster is to create a "throw-away" cluster first and then use the existing cluster for execution:

#------------------# Create your cluster in Azure passing, it your cluster config file.
#throw-away cluster#
cluster <- makeCluster("cluster.json")
#------------------# Get existing cluster
cluster <- getCluster("TestCluster_2020", verbose = TRUE)
#------------------# Register the cluster as your parallel backend
registerDoAzureParallel(cluster)

#############################

sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C LC_TIME=English_United Kingdom.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] doAzureParallel_0.8.0 iterators_1.0.12 foreach_1.5.0

loaded via a namespace (and not attached):
[1] compiler_3.6.3 prettyunits_1.1.1 bitops_1.0-6 remotes_2.2.0 tools_3.6.3 testthat_2.3.2 digest_0.6.25 pkgbuild_1.1.0 pkgload_1.1.0 jsonlite_1.7.1 memoise_1.1.0 rlang_0.4.7 cli_2.0.2
[14] rstudioapi_0.11 curl_4.3 withr_2.3.0 httr_1.4.2 desc_1.2.0 fs_1.5.0 devtools_2.3.2 rprojroot_1.3-2 glue_1.4.2 R6_2.4.1 processx_3.4.4 fansi_0.4.1 sessioninfo_1.1.1
[27] callr_3.4.4 magrittr_1.5 backports_1.1.10 ps_1.3.4 codetools_0.2-16 ellipsis_0.3.1 usethis_1.6.3 assertthat_0.2.1 mime_0.9 rAzureBatch_0.7.0 RCurl_1.98-1.2 crayon_1.3.4 rjson_0.2.20