Utility for package exactextractr, which has non-exportable objects

[GatesDupont]

One package with non-exportable objects is exactextractr, which offers the fastest raster extractions.

On your page, https://future.futureverse.org/articles/future-4-non-exportable-objects.html, you asked people report such instances.

[HenrikBengtsson]

Thanks for sharing. I'd love to add this to the list, but I'd probably need your help since I'm to well versed in the spatial field. What I need it a minimal reproducible example that illustrate the problem. Maybe some similar to the example for raster that results in:

f <- future({
  print(dim(r))
  print(dim(s))
})
Error: Detected a non-exportable reference ('externalptr') in one of the
globals ('s' of class 'RasterLayer') used in the future expression

[HenrikBengtsson]

@GatesDupont , I would need your help to come up with a tiny reproducible example that I can document. @dbaston, as the maintainer of exactextractr, would you be able to pitch in here?

[HenrikBengtsson]

@GatesDupont, just a friendly reminder about this one; if you can give me some small example code that allows me to reproduce the problem, I can begin documenting it. Without it, I cannot add any documentation.

[HenrikBengtsson]

@sigmafelix, @kyle-messier, because of your work on https://github.com/ropensci/chopin/, is this small issue something you could help out with? I'm looking for a teeny example for creating a exactextractr object, which I understand cannot be exported to a parallel worker as-is.

[sigmafelix]

@GatesDupont @HenrikBengtsson

externalptr comes from raster or terra dependency. To note, exactextractr::exact_extract() returns one of vector, data.frame, or list object depending on argument values. x argument should be one of RasterBrick, RasterLayer, RasterStack (of raster package, superseded by terra) or RasterLayer (of terra package), and I found some of which cannot be exported to parallel workers under multisession and cluster plans in future. multicore plan can access any of these class objects since it relies on shared memory. It repeats the list of non-exportable objects.

My reproducible examples below demonstrate that SpatRaster in terra with multisession plan failed with a memory reference error. The objects related to raster package did not make errors; presumably, all of RasterLayer or RasterBrick objects were not actually externalptr.

library(future)
library(exactextractr)
library(raster)
library(terra)
library(sf)
library(reprex)
options(sf_use_s2 = FALSE)
ncpoly <- system.file("shape/nc.shp", package = "sf")

# terra cases
ncvect <- terra::vect(ncpoly)
ncsf <- sf::st_as_sf(ncvect)
nclyr_terra <- terra::rast(ncvect, ncols = 500, nrows = 300)
terra::values(nclyr_terra) <- rnorm(terra::ncell(nclyr_terra), 10, 10)

# Multicore case with SpatRaster
future::plan(future::multicore, workers = 2)
initdf <- list()
res <-
    future::future(
        {
            for (i in nrow(ncsf)) {
                initdf <-
                  append(
                    initdf,
                    exactextractr::exact_extract(nclyr_terra, ncsf, fun = "mean")
                  )
            }
            Reduce(rbind, initdf)
        },
        globals = list(nclyr_terra = nclyr_terra, initdf = initdf, ncsf = ncsf),
        seed = TRUE
    )
resval <- future::value(res)
head(resval)
future::plan(future::sequential)


# Multisession case with SpatRaster: error
future::plan(future::multisession, workers = 2)
initdf <- list()
res <-
    future::future(
        {
            for (i in nrow(ncsf)) {
                initdf <-
                  append(
                    initdf,
                    exactextractr::exact_extract(nclyr_terra, ncsf, fun = "mean")
                  )
            }
            Reduce(rbind, initdf)
        },
        globals = list(nclyr_terra = nclyr_terra, initdf = initdf, ncsf = ncsf),
        seed = TRUE
    )
resval <- future::value(res)
# Error in .Call(list(name = "CppField__get", address = <pointer: (nil)>,  : 
#   NULL value passed as symbol address
future::plan(future::sequential)

# Multicore case with RasterLayer
ncpolyras <- raster::raster(nclyr_terra)
future::plan(future::multicore, workers = 2)
initdf <- list()
res <-
    future::future(
        {
            for (i in nrow(ncsf)) {
                initdf <-
                  append(
                    initdf,
                    exactextractr::exact_extract(ncpolyras, ncsf, fun = "mean")
                  )
            }
            Reduce(rbind, initdf)
        },
        globals = list(ncpolyras = ncpolyras, initdf = initdf, ncsf = ncsf),
        seed = TRUE
    )
resval <- future::value(res)
head(resval)
future::plan(future::sequential)

# Multicore case with RasterBrick
raster::inMemory(ncpolyras)
ncpolyras <- raster::brick(ncpolyras)
# TRUE
future::plan(future::multisession, workers = 2)
initdf <- list()
res <-
    future::future(
        {
            for (i in nrow(ncsf)) {
                initdf <-
                  append(
                    initdf,
                    exactextractr::exact_extract(ncpolyras, ncsf, fun = "mean")
                  )
            }
            Reduce(rbind, initdf)
        },
        globals = list(ncpolyras = ncpolyras, initdf = initdf, ncsf = ncsf),
        seed = TRUE
    )
resval <- future::value(res)
head(resval)
future::plan(future::sequential)


# RasterLayer case: read raster from a file path
temppath <- file.path(tempdir(), "myraster.tif")
raster::writeRaster(ncpolyras, temppath)
ncpolyras2 <- raster::raster(temppath)
raster::inMemory(ncpolyras2)
# FALSE

future::plan(future::multisession, workers = 2)
initdf <- list()
res <-
    future::future(
        {
            for (i in nrow(ncsf)) {
                initdf <-
                  append(
                    initdf,
                    exactextractr::exact_extract(kk, ncsf, fun = "mean")
                  )
            }
            Reduce(rbind, initdf)
        },
        globals = list(kk = ncpolyras2, initdf = initdf, ncsf = ncsf),
        seed = TRUE
    )
resval <- future::value(res)
head(resval)
future::plan(future::sequential)