Internal parallel functions

Experimental parallel functions, intended for internal use now. The goal is to allow 'RAVE' functions to gain the potential benefit from parallel computing, but allow users to control whether to do it.

Usage

with_mirai_parallel(
  expr,
  workers = 0,
  globals = list(),
  serialization_config = list(),
  always = FALSE
)

lapply_jobs(x, fun, ..., .globals = list(), callback = NULL)

Arguments

expr

expression to evaluate with parallel workers

workers

number of workers

globals, .globals

global variables to be serialized

serialization_config

serialization configurations

always

whether always use workers, only considered when number of workers is one; default is false, then run jobs in the main process when only one worker is required

x

a list, vector, array of R objects

fun

function to apply to each element of x

...

additional arguments to be passed to fun

callback

callback function, input is each element of x and should return a string, for progress bar

Examples

# Run without `with_mirai_parallel`
res <- lapply_jobs(1:5, function(x, ...) {
  c(child = Sys.getpid(), ...)
}, main = Sys.getpid())

simplify2array(res)
#>       [,1] [,2] [,3] [,4] [,5]
#> child 7800 7800 7800 7800 7800
#> main  7800 7800 7800 7800 7800

# When wrapped in `with_mirai_parallel`
with_mirai_parallel({
  res <- lapply_jobs(1:5, function(x, ...) {
    c(child = Sys.getpid(), ...)
  }, main = Sys.getpid())
  simplify2array(res)
}, workers = 1)
#>       [,1] [,2] [,3] [,4] [,5]
#> child 7800 7800 7800 7800 7800
#> main  7800 7800 7800 7800 7800

# Comparison
f <- function() {
  system.time({
    lapply_jobs(1:5, function(x, ...) {
      Sys.sleep(1)
      c(child = Sys.getpid(), ...)
    }, main = Sys.getpid())
  })
}

if (FALSE) { # \dontrun{

# Without parallel
f()
#>    user  system elapsed
#>   0.022   0.019   5.010

# with parallel
with_mirai_parallel(f(), worker = 5)
#>    user  system elapsed
#>   0.002   0.001   1.146

} # }