I have been using Azure Batch for a while and have come across a new error in the merge task even on the sample code. this appears to happen with any size of data while running the foreach %dopar%. has anyone else noticed this?
HTTPError: 403 Client Error: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. for url: https://_batchaccount._blob.core.windows.net/job12345?sv=2016-05-31&sr=c&st=2020-07-18T21%3A09%3A30Z&se=2020-07-21T21%3A09%3A30Z&sp=rwcl&sig=signature%3D&include=metadata&restype=container&comp=list&maxresults=1000
Reproduce Example
generateCredentialsConfig("credentials.json")
setCredentials("credentials.json")
generateClusterConfig("cluster.json")
cluster <- makeCluster("cluster.json")
Register your parallel backend
registerDoAzureParallel(cluster)
setVerbose(FALSE)
setAutoDeleteJob(FALSE)
mean_change = 1.001
volatility = 0.01
opening_price = 100
getClosingPrice <- function() {
days <- 1825 # ~ 5 years
movement <- rnorm(days, mean=mean_change, sd=volatility)
path <- cumprod(c(opening_price, movement))
closingPrice <- path[days]
return(closingPrice)
}
Optimize runtime. Chunking allows running multiple iterations on a single R instance.
opt <- list(chunkSize = 10)
start_p <- Sys.time()
closingPrices_p <- foreach(i = 1:1000, .combine='c', .options.azure = opt) %dopar% {
replicate(100000, getClosingPrice())
}
end_p <- Sys.time()
hist(closingPrices_s)