Known issues: https://github.com/PredictiveEcology/SpaDES.core/issues
SpaDES modules can now be R packages. The simplest way to convert a module to a package is using the new function convertToPackage. Benefits of doing this are so that a SpaDES module can benefit from the infrastructure of an R package (e.g., devtools::document, devtools::check, setting up Continuous Integration systems etc.). Any documentation written using #' i.e., roxygen2 will be copied immediately above the function where it was sitting.newModule now correctly places the SpaDES.core package dependency in the reqdPkgs element of the metadata, instead of version. It will put the full Github reference if SpaDES.core was installed directly from GitHub.qs package: either qsave or qread converts data.table objects to list objects. loadSimList has a work around internally to convert these objects back to data.table, if the metadata indicate that the objects should be data.table objects.paramCheckOtherMods. Can be used within a module to assert that a parameter has the same value as the same parameter in other modules. This is therefore a check of a parameter that might be considered a .global and passed within simInit(..., params = list(.globals = list(someParam = "someValue"))), but the user did not do that.spades messaging when e.g., debug = 1 can correctly accommodate nested spades calls, i.e., a SpaDES module calling spades internally.newModule now puts SpaDES.core dependency in the correct reqdPkgs instead of version metadata element.plots instead of .plotInitialTime, Plots will check whether .plotInitialTime is actually set in the module metadata first. Only if it is there, will it evaluate its value. Currently, modules get default values for .plotInitialTime even if the module developer didn’t include it in the module metadata.debug arg of spades is set to an event type that is also in the core modules (e.g., save, load), such as “init”Cache-ing of a simList, when quick is a character vector, errored. Now fixed.moduleMetadata incorrectly dropped defineModuleListItems under certain signatures.moduleCoverage has been rewritten to estimate code coverage of a module using covr package.P now has a replacement method. So, to update a parameter within a module (where it is namespaced, i.e., don’t have to specify the module name): P(sim, "paramName") <- 1. If using this outside a module, then module (3rd argument) will have to be specified.P argument order changed to accommodate the fact that namespacing is used to detect module name: the user does not need to supply module, so it should not be second. This is for the normal P method and the new replace method above: it is now P(sim, param, module); there are attempts to capture errors (i.e., parameter supplied that matches a module, but not a parameter; vice versa) and give a warning for user to change code. This may have little downstream effect as all known cases use the P(sim)$paramName, which will still work fine, instead of P(sim, "paramName").Plots does a better job with RasterStack objects plotted to screen without ggplot2.isFALSE: use base::isFALSE nowPlots can now omit the data argument; just use the named arguments in …defineParameter now allows multi-line desc or multiple strings; paste is no longer needed for long descmoduleCodeFiles a new function that identifies all the code files in a collection of modules.globals functionality is modified. If a user specifies a .globals in the parameters object (passed into simInit), then all identical parameters in all modules will be overridden with these .global valuesdefineParameter, expectsInput and createsOuptut can all now have multi-line desc, without needing to use paste or paste0. Extraneous spaces and carriage returns will all be stripped. This can either be using a single multi-line quote or via multiple lines, each with its own "".simList is no longer “locked” with lockBinding. It is already hidden in sim$.mods, and since sim$.mods can be modified, this was a weak caution against user modification. Further, for moduleCoverage, the module environment needed to be unlocked, which is not allowed by CRAN..inputObjects was cached (via setting useCache = '.inputObjects' parameter), it was “too sensitive”. Changes to any module’s parameters, not just the current module, would cause rerun of .inputObjects. Now it correctly identifies parameter changes only in the current module. THIS WILL CAUSE some existing caches to trigger a rerun once; after this, it will be less sensitiverestartSpades did not correctly deal with objects that did not yet exist prior to the event. Fixed with: 24b9cd12973aa81a9a4923a02225e095fa28f77a.restartSpades was losing the previous completed events list. This has been fixed; it is now kept after restartSpadesdata only (e.g. quickPlot::Plot-like behaviour)simInitAndSpades now has .plots arg to match spadesPlots function can be used like Plot, but with types specified. The devices to save on disk will have some different behaviours to the screen representation, since “wiping” an individual plot on a device doesn’t exist for a file device.Plots function that will produce zero to 4 types of items that are relevant for plotting: 1) Visual on screen, 2) The plot object saved to disk, 3) The raw data that went into the plot and 4) The plot as one or more image files, e.g., .png or .pdf via ggsavespades now accepts an events argument, which will limit the events that are run to those specified in the argument. This seems to be most useful for the init case, e.g., spades(sim, events = "init"). See ?spades.simInit now is prefixed with Sys.time() and "simInit"spades is simplified to take up fewer characters: INFO:: has been removedsimInit now checks for minimum version of SpaDES.core needed in a module and stops if it fails, giving instructions how to upgrade.spatialExtent, as they are not used by the spades algorithmsanyPlotting to test whether plotting of one form or another should occurspades call is now more informative, including module name (by default shortened – can be changed with options("spades.messagingNumCharsModule")))defineParameter can now accept a vector of “class”, so a parameter can be more than one class. Presumably this should generally not be used, but a good reason could be, say, c("numeric", "function"), where the use can pass either a numeric or a function that would calculate that numeric.simFile to generate file names for use with e.g., saveSimListzipSimList is now exportedspades will now attempt to load reqdPkgs, which is already done in simInit. In cases where simInit was not run, e.g., Cache(simInitAndSpades, ..., events = "init"), then modules will not have access to packages. For cases where simInit was called, then this should add very little overhead.saveSimList will now convert file-backed Raster* class objects to memory if fileBackend = 0. Previously, it left them as is (on disk if on disk, in memory if in memory).time(sim) in the case where it is equal to or after end(sim). Previously, this would not run any events if time(sim) >= end(sim) && events(sim)[[1]] < time(sim)..seed parameter for modules (#163)defineParameter was throwing is.na(default) warning when a parameter was not an atomic.spades.useRequire = FALSE (#141)sim; user informed with a warningtry() with communities() to skip tests on systems without igraph GLPK support.RandomFields being unavailable.spades.futureEvents option. If set to TRUE, spades will run module events in a “future” (see future package), if they do not produce outputs for other modules.use_gha() and corresponding vignette; #74)newProject creates Rstudio .Rproj file if invoked in RstudiopaddedFloatToChar to reproducible; but re-exported here, so still usable..seed which is a named list where names are the events and the elements are the seed with which to run the event. During doEvent, SpaDES.core will now set.seed(P(sim)$.seed[[currentEvent]]) and reset to random number stream afterwards.dplyr, lubridate, R.utils, tools, backports and rlang from dependenciestcltk to Suggestsdevtools, microbenchmark from Suggestsall.equal(..., check.environment = FALSE) for internal testingRCurl dependency (#120)sp because it’s linked in documentation (#120)pkgDeps example for new version of Requiredesc argument in defineParameter, expectsInput, and createsOutput can now have extraneous spaces and End-of-Line characters. This means that they can now be written more easily with a single set of quotes, without needing paste. The accessor functions, moduleParams, moduleInputs, and moduleOutputs all will strip extraneous spaces and End-of-Line characters.writeEventInfo() and writeRNGInfo() to write info to file.stringi.. for moduleParams etc. This is more accurate.Par is now an activeBinding (similar to mod) pointing to P(sim); this allows for tab autocomplete to function correctly.moduleParams(), moduleInputs(), moduleOutputs(). These are now used in default .Rmd template.memoryUse functionalitysim is now created at .pkgEnv$.sim at the start of spades call, rather than on.exit; failures due to “out of memory” were not completing the on.exitnewModule() sets useGitHub = TRUE by default).usethis to Suggests for use with GitHub ActionsFilenames function coming from reproducible packageoptions('spades.recoverMode') was creating temp folders every event and not removing them; now it does.options('spades.recoveryMode' = 0) may further helpreproduciblesimInit, e.g., simInit(times = list(start = "test")) now fails because times must be a list of 2 numeric objectsmessage instead of a mixture of message, cat and print. This allows for easier suppressing of messaging, e.g., via suppressMessages. This was requested in a downstream package, SpaDES.experiment that was submitted to CRAN but rejected due to the now former inability to suppress messages.restartR saves simulation objects using qs::qsave() which is faster and creates smaller file sizes.codetools, future, httr, logging, and tcltkarchivistqs now used for improved object serialization to disk.objSizeInclEnviros and removed
objectSynonyms caused a breakage under some conditions related to recovering a module from Cache.print and cat statements to message to allow use of suppressMessages, as recommended by CRANlogging package, invoked by setting debug argument in spades function call to a list(...). ?spades describes detailsrestartR minor bug fixesDEoptim, future.apply, Matrix, parallel, pryr, purrr, and rgenoud, which are no longer required. See “deprecated” info below.whisker to Imports to facilitate module file templating (#100)future is installed. See new vignette iv-advanced and ?memoryUse.restartR. Restarts R mid-stream to deal with apparent memory leaks in R. In our experience with large projects that have long time horizons, there appears to be a memory leak at a low level in R (identified here: https://github.com/r-lib/fastmap). This has prevented projects from running to completion. Without diagnosing the root cause of the memory inflation, we have noticed that interrupting a simulation, saving the simList, restarting R, resets the memory consumption back to levels near the start of a simulation. The new functionality allows a user who is hitting this memory leak issue to restart R as a work around. See ?restartR for instructions.newProject to initialize a SpaDES project with subdirectories cache/, inputs/, modules/, and outputs/, and setPaths() accordingly.newModule() now uses open = interactive() as default to prevent files being left open during tests.experiment(), experiment2(), and POM() have been moved to the SpaDES.experiment packageSpaDES.core. Too many dependency packages are not maintaining their backwards compatibility.backports to Imports for R-oldrel supportgoogledrive dependency (this functionality moved to reproducible)P, params, and parameters, thanks to Louis-Etienne Robert.objSize.simList method with 2 new arguments from reproducible package.robustDigest method for simList class objects now does only includes parameters that are listed within the module metadata, if Cache or .robustDigest is called within a module. This means that changes to parameter values in “other” modules will not affect the Caching of “the current” module.outputObjectNames will extract just the object names of all outputObjects across modulesrestartSpades and its associated options(spades.recoveryMode = 1), the new default, which is still experimental. Its purpose is to be able to restart a simulation in the case of an error or interruption.mod is now an active binding to sim[[currentModule(sim)]]$.objects (move from sim[[currentModule(sim)]]) and its parent environment is emptyenv(). This should cause no changes to users who use mod$..., but it will cause a change if user was calling objects directly via sim[[currentModule(sim)]]$.... This change is to separate the function enclosing environments and object enclosing environments, which should be different.sim@completed is now an environment instead of a list. Of the three event queues, this one can become the largest. The list would get increasingly slow as the number of completed events increased. There should be no user visible changes when using completed(sim)spades.debug is now set to 1 spades.recoveryMode is new and set to 1 (i.e., the current event will be kept at its initial state)
simInit especially in some weird cases of childModules.reqdPkgs not being loaded when only listed in child modules. Fixed in 5cd79ac95bc8d190e954313f125928458b0108d2.igraphRandomFields >= 3.3.4archivist and devtools added to Suggests because they are used in vignettesreproducible version 0.2.6new vignette on caching SpaDES simulations moved from SpaDES package.
simList environment now has emptyenv() as its parent.env. The biggest user-facing changes are:
envir(sim) (unusual, but may occur) won’t find objects in the .GlobalEnv;parent.env in which they are defined (little know fact identified here: http://adv-r.had.co.nz/memory.html#gc identified as a possible source of memory leaks).module’s function environment in the simList now has its parent asNamespace("SpaDES.core") instead of the envir(sim) (as mentioned above), i.e,. parent.env(sim[[currentModule(sim)]]) is asNamespace("SpaDES.core"). The main user-noticeable changes of this are that module functions will not accidentally find objects in the simList unless they are actually passed in explicitly as arguments.
New active binding, mod that works as a module-specific variable, similar to a private object, i.e., mod$a is a local object inside the module that persists across events. It is a pointer to sim[[currentModule(sim)]]$a
New function scheduleConditionalEvent, which allows an event to be scheduled based on a condition. Still experimental.
An experimental new function and feature, objectSynonyms, which will create active bindings of two names to a single object
User can now specify modulePath as a character vector, e.g., simInit(..., paths = list(modulePath = c(".", "test"))). This means that a user can organize the modules in different locations.
modulePath now has a new argument, module, where user can specify (a) specific module(s)’s path. Modifications were implemented to dataPath to utilize this new feature
simInit and spades now call setPaths(paths) or setPaths(sim$paths), unsetting them on.exit internally to make the paths used for functions e.g., reproducible::Cache to use the correct path
under-the-hood speed improvements for the DES (about 20% faster) – 38 microseconds per event under ideal conditions
improved default path settings in .inputObjects (#83)
following reproducible package updates, now uses data.table::setattr internally to avoid copying of objects (this may have very little/no effect on simList objects)
suppliedElsewhere has a new argument, returnWhere, a logical which will cause a logical of length 3 to be returned, indicating in which of the 3 other places the object may have been supplied, instead of length 1, still the default.
data.table v1.12.0 (#85, @mattdowle)Copy (error existed because function inheritance persisted even though the location of the function was moved)RandomFields to Suggests, as it is in the Suggests of SpaDES.tools and used in examples/tests.options("spades.saveSimOnExit" = TRUE). This will save the state of the simList to an object as SpaDES.core:::.pkgEnv$.sim, with a message, if there is a hard exist. There is virtually no computational cost to this, as the object is already in RAM.simList internals changed. It now inherits from environment. Amongst other things, this means that tab autocomplete in RStudio now works for objects in the simList. Also, we removed several associated methods, $, [[, ls, ls.str, objects, as the defaults for environments work correctly with the simList nowdebug arg in spades call can now take numeric, currently 1 or 2, giving a few pre-packaged informative messaging each eventelapsedTime which gives a summary of the clock time used by each module or eventcitation replaces utils::citation with an S4 generic. If package arg is a character, it dispatches utils::citation; if a simList, it gives the citation for the module(s)downloadModule() now prints the module version downloaded (#77).inputObjects() name conflict (internal .inputObjects renamed to ._inputObjectsDF; .outputObjects renamed to ._outputObjectsDF).inputObjects evaluated based on module load order (#72).robustDigest fix for simLists – needed to omit ._startClockTime and .timestampsp from importsreproducible (>=0.2.2)spades.useRequire: a logical which causes simInit to load packages with Require or require. Lower case is generally faster, but will not handle the case of uninstalled packages, so should only be used once all packages are installed.spades.keepCompleted: a logical which causes spades() to keep (TRUE) or not keep a record of completed events. Keeping track of completed events when they are many (>1e5) gets slow enough that it may be worth turning it off.testsall.equal.simList now removes all time dependent attributes, e.g., ._startClockTime and .timestampoptions("spades.keepCompleted" = FALSE)Improvements to caching of functions with simList objects:
simList in the arguments would erroneously return cached copies of functions. These now are copied through from argument simList, rather than cached simList. This means that changes to the function definitions in a module will persist (e.g., debugging via browser() will work correctly)simList in arguments that return a simList will now do a post digest of the output. This will be compared with the predigest, and only those object which changed in the simList will be modified..inputObjects function was incorrect. Fixed.module metadata now in named lists inside depends(sim)
new debugging – if debug is not FALSE, then any error will trigger a browser() call inside the event function. User can continue (c) or quit (Q) as per normal. c will trigger a reparse and events will continue as scheduled.
introduction of code checking for modules, currently turned on or off by an option spades.moduleCodeChecks, which is TRUE by default. Code checking includes various types:
codetools to check for various code problemsraster::level, raster::scale, quickPlot::Plot)checkCodeEnv on every function inside a modulesim$xxx occurrences in modules, comparing to outputObjects in metadata if used in assignment (i.e., left hand side of assign operator), or comparing to inputObjects if used on the right hand sideinputObjects have default values assigned in the .inputObjects functionoption spades.debug set to TRUE by default, instead of FALSE. This is better for new users.
moduleMetadata argument order changed, so sim is first, more consistent with all other simList accessors.
downloadData has changed dramatically, now it is a wrapper around reproducible::prepInputs which does more checking.
extractURL will extract the sourceURL from metadata, given an object name.
makeMemoiseable and unmakeMemoisable, new methods, each the inverse of the other, to deal with imperfect memoised returns under some cases of simList.
new option, spades.keepCompleted, TRUE by default, which can be useful for dramatically speeding up the DES when there are many (>10,000) events.
fileExt – use tools::file_ext insteaddata.table changes (@mattdowle, #64).start and end.newModule template modified slightly based on workshop feedback.setPaths now only sets the directories that are passed into it.all.equal.simList method strips a small number of attributes that are used internally that create false failures.tools, pryr.rgeos, RCurl and googledrive.uses reproducible::Require instead of SpaDES.core::loadPackages to load required packages. Currently, does not use version control for packages, but does use installing (from CRAN or GitHub), and loading (via require). This means a module can indicate a github package, e.g,. achubaty/amc@development
environments in modules are now as follows:
sim@.envir$<moduleName>, and it is a is a child of sim@.envir. Functions can be found in this environment, but prefixing functions is not necessary, because modules functions are within this environment already.sim@.envir is a child of SpaDES.corescoping from within a function that is defined in a module is thus:
sim@.envir$<moduleName> –> sim@.envir –> SpaDES.core –> all imported packages including base –> .GlobalEnv –> search()speed improvements:
data.table objects. For small objects (e.g., the eventQueue) that have fewer than 200 objects, lists are faster. Accessors (e.g., events(sim), completed(sim)) of the event queues still show data.table objects, but these are made on the fly..parseModule and .parseModuePartial now put their parsed content into a temporary environment (sim@.envir$.parsedFiles$<Full Filename>) during the simInit, which gets re-used. Previously, files were parsed multiple times in a given simInit call. Several functions now have envir argument to pass this through (including moduleVersion, packages, checkParams)parsing of modules is now more intelligent, allowing for modules to contain functions (the current norm) and but they can also create objects at the module level. These can use the sim object in their definition. These objects can, for example, be used to help define parameters, for example, e.g., startSimPlus1 <- start(sim) + 1 can be defined in the module and used in defineModule
remove grDevices from Imports as it was not used (#1)
remove chron and CircStats dependencies
remove functions dwrpnorm2 and move to package SpaDES.tools
remove unused function F() due to conflicts with F/FALSE.
improved download of module data: added new quickCheck argument
improved download of modules: use fuzzy matching
new option: spades.switchPkgNamespaces which allows the user to turn off the SpaDES feature that loads and unloads libraries specific to each module. While useful, it slows down computations, in some cases, by a lot.
bug fixes:
zipModule that omitted the checksum file from being included when data = FALSE (#3).inputObjects functions was evaluating outputObjects instead of inputObjects. Now corrected.If .inputObjects contains arguments other than just sim, these will be evaluated as function inputs by the Cache mechanism (via .useCache), therefore correctly assessing when those inputs changed, e.g., if they are files and the arg is wrapped in asPath, then any change to the underlying file will cause a re-cache. e.g., .inputObjects <- function(sim, importantFile = asPath(file.path(inputPath(sim), "theFile.rdata"))) { ... }
default debug option in spades() now uses the package option spades.debug and default is set to FALSE (#5)
various other speed improvements and bug fixes
convert P to a function, rather than S4 generic and method, for speed.
importFrom only used functions from utils due to name conflicts with raster::stack and utils::stack
new function remoteFileSize to check the size of remote files
new namespaced function dataPath will return file.path(modulePath(sim), currentModule(sim), "data"), which will return a different path, depending on which module it is placed inside.
add crayon to imports – now messages are more colour coded
bug fix in ‘inputs’ for the case of loading objects from the global environment, either from the same object to the same object, or from different global objects overwriting on the same simList object
A new package, which takes all core DES functionality out of the SpaDES package:
?SpaDES.core for an overviewvarious speed improvements and bug fixes