Simulation data storage
A key problem in simulation-based inference is handling output from the simulator. In the most trivial case of forward map simulations, the simulator is treated as a black box function $f: \Phi \mapsto \mathbf{Y}$ where the burden of selecting appropriate outputs is typically left to the implementation of $f$. For more complex simulators, however, the size and complexity of the outputs may require a more complex observation operator that maps from the space of simulator states/outputs to observables. SimulationBasedInference provides a generic API for handling simulation data that is consumed by Observables.
The simulation data storage API consists of three layers:
| Type | Role |
|---|---|
StorageBackend | Represents the backing store for all simulation data; e.g. in memory vs on disk. |
SimulationData | Lightweight view of the data for a single simulation stored in the backend. |
SimulationDataSet | Lightweight view of the data for multiple simulations stored in the backend. |
In addition, DataBuffers, provide array-like access to a single named output or scratch series inside a SimulationData.
Storage backends
SimulationBasedInference.StorageBackend — Type
StorageBackendAbstract type for the storage backend that owns all persistent simulation data. There is exactly one backend per SimulationDataSet; the higher-level types (SimulationData, DataBuffer) are lightweight views that route all reads and writes through it. The backend decides where the bytes actually live — in memory (InMemoryStorage) or out-of-core (e.g. the JLD2 backend provided by the SimulationBasedInferenceJLD2Ext extension).
Each simulation has an input, a metadata dictionary, and a set of named output series. Scratch (transient working) series are not part of the backend; they live on a StorageHandle and are discarded when the handle is closed.
Operations come in two forms:
- a fast path taking a
StorageHandleas the first argument (file kept open), and - a slow path taking the backend directly, which auto-opens/closes a handle per call via
open(backend) do h ... end(backward compatible).
The default InMemoryStorage backend keeps everything in RAM:
SimulationBasedInference.InMemoryStorage — Type
InMemoryStorage{inputType,outputType,metadataType} <: StorageBackendIn-memory storage backend. Simulation inputs, output series elements, and metadata values are strongly typed as inputType, outputType, and metadataType respectively. The default constructor InMemoryStorage() uses Any for all three.
The in-memory backend is suitable for simulators with relatively small output sizes or those where simulation data storage and postprocessing is already handled internally. However, in some cases, we may want to define observables that process the simulation data efficiently "online" (i.e. during the simulation) rather than accumulate everything into memory. For these cases, a DiskStorageBackend can be used instead:
SimulationBasedInference.DiskStorageBackend — Function
DiskStorageBackend(::Type{DataFormat{format}}, path::AbstractString, args...; kwargs...) where {format}Construct a disk-backed StorageBackend whose backend persists each simulation to disk at the given path. Requires a supported file I/O backend to be loaded, e.g. JLD2.
Currently, SimulationBasedInference provides a single implementation of DiskStorageBackend using distributed (per simulation) JLD2 files. This is implemented by the SimulationBasedInferenceJLD2Ext extension module which is auto-loaded with JLD2:
using JLD2 # loads SimulationBasedInferenceJLD2Ext
storage = SimulationDataSet(
backend = DiskStorageBackend(format"JLD2", "simulations/")
)StorageHandle
StorageBackends provide an I/O handle API that keeps a connection open for the duration of a batch of operations on a single simulation. Handles are obtained with open(backend, index) where index is the index of the simulation.
SimulationBasedInference.StorageHandle — Type
StorageHandleAbstract base type for the object returned by open(backend::StorageBackend, index::Integer). A handle holds the active connection to a backend (e.g. an open file or in-memory view) for the duration of a batch of operations on a specific simulation and owns the scratch namespace for that simulation. Concrete handle types must provide a scratch::Dict{Symbol,Any} field; scratch is cleared when the handle is closed.
Use with the do-block form for guaranteed cleanup:
open(backend, index) do h
# use handle h for simulation index
end # handle closed (and scratch cleared) automaticallyThe open implementation for backends supports a standard do-block syntax for automatically closing the StorageHandle after use:
open(backend, 3) do h
store_output!(h, :y, value)
getinputs(h)
end # handle closed automaticallyScratch storage, i.e. transient working buffers needed during a single simulation and accessed via get_scratch_buffer, are tied to the handle and are discarded once it is closed.
SimulationData
SimulationData represents the main interface for reading and writing data for a single simulation. It is passed to observe!, initialize!, and getvalue when computing or reading observables during a forward solve.
SimulationBasedInference.SimulationData — Type
SimulationData{B}A view of the data for a single simulation stored in the given StorageBackend. The optional handle field maintains a persistent storage handle when needed (e.g., for observables that require scratch storage across multiple observe! calls).
Reading and writing
SimulationBasedInference.store! — Method
store!(data::SimulationData, name::Symbol, value)Append value to the persistent output series for observable name.
SimulationBasedInference.getoutput — Method
getoutput(data::SimulationData, name::Symbol)Return the collected output sequence (a Vector) for observable name.
SimulationBasedInference.getoutputs — Method
getoutputs(data::SimulationData)Return a NamedTuple mapping each output name to its corresponding data series.
getinputs(data) and getmetadata(data) return the input parameters and metadata dictionary stored for the simulation, respectively.
Buffer access
For observable implementations that need to accumulate values over several simulator steps (e.g. TimeSampled), SimulationData provides lazily-opened buffer views:
SimulationBasedInference.get_output_buffer — Function
get_output_buffer(data::SimulationData, name::Symbol)Return a DataBuffer view of the output data for variable name, creating it if necessary. The handle is opened lazily and reused across calls until explicitly closed.
SimulationBasedInference.get_scratch_buffer — Function
get_scratch_buffer(data::SimulationData, name::Symbol=:scratch)Return a DataBuffer view of the transient scratch series name. The handle is opened lazily and reused across calls until explicitly closed.
The with_output_buffer and with_scratch_buffer helpers open a buffer, run a function, and close the handle in a single call:
SimulationBasedInference.with_output_buffer — Function
with_output_buffer(func!, data::SimulationData, name::Symbol)Open an output DataBuffer for name, call func!(buffer), then close the handle. Equivalent to calling get_output_buffer, using the result, and then calling close(data).
SimulationBasedInference.with_scratch_buffer — Function
with_scratch_buffer(func!, data::SimulationData, name::Symbol=:scratch)Open a scratch DataBuffer for name, call func!(buffer), then close the handle. Equivalent to calling get_scratch_buffer, using the result, and then calling close(data).
Lifecycle
Base.close — Method
close(data::SimulationData)Close the persistent handle for this simulation if it is open. Should be called when the simulation is complete to release file descriptors and flush any pending writes.
Base.empty!(data::SimulationData) clears all output series stored for the simulation.
SimulationDataSet
SimulationDataSet is the top-level container used throughout an inference run. It is passed to init and accumulates one SimulationData entry per forward solve.
SimulationBasedInference.SimulationDataSet — Type
SimulationDataSet{B<:StorageBackend}A view of the whole collection of simulations held in a single StorageBackend — the simulations run during an inference procedure. SimulationDataSet() uses an in-memory backend; OnDiskSimulationDataSet(path) (provided by the SimulationBasedInferenceJLD2Ext extension) uses a disk-backed one.
Indexing returns a SimulationData view of the i-th simulation; iterating yields (input, outputs, metadata) triples.
Allocating and storing simulations
SimulationBasedInference.allocate! — Function
allocate!(storage::SimulationDataSet, inputs=nothing; metadata...)Allocate simulation data storage in the underlying backend (with the given input) and return a SimulationData view of it.
SimulationBasedInference.store! — Method
store!(storage::SimulationDataSet, data::SimulationData; metadata...)Append an existing SimulationData (possibly held in a different backend, e.g. a per-member forward solve) to storage, copying its input and output series into the backing store and merging any extra metadata.
Indexing and iteration
SimulationDataSet supports standard Julia indexing and iteration:
# index — returns a SimulationData view
sim = storage[3]
# iteration — yields (inputs, outputs, metadata) triples
for (inputs, outputs, metadata) in storage
@show inputs, metadata[:iter]
end
# number of completed simulations
length(storage)Querying across iterations
SimulationBasedInference.iterations — Method
iterations(storage::SimulationDataSet)Return the number of distinct iter values recorded in the simulations' metadata, i.e. the number of inference iterations. Returns 0 for an empty storage and 1 when no iter metadata is present.
DataBuffer
DataBuffer is a low-level array-like view of a single named series (output or scratch) inside a SimulationData. It is primarily used internally by observable implementations.
SimulationBasedInference.DataBuffer — Type
DataBuffer{kind, H} where {H <:StorageHandle}A lightweight view onto a variable with name with data accessed via the given StorageHandle. The handle owns the simulation index (handle.index). kind is either :output (persistent output) or :scratch (a transient working buffer). A DataBuffer owns no data of its own; store!, getindex, length, and empty! all forward to the I/O handle.
DataBuffer supports the standard Julia push!, getindex, length, empty!, collect, first, last, and iterate methods, all forwarding through the underlying StorageHandle.
Custom backend implementations
New StorageBackend subtypes must implement the following methods (dispatching on both the backend directly for the slow path, and on a concrete StorageHandle subtype for the fast path):
| Method | Description |
|---|---|
open(backend, index) -> StorageHandle | Open handle for simulation index |
allocate!(handle, input; metadata...) | Allocate a new simulation slot |
num_simulations(backend) | Total number of stored simulations |
getinputs(handle) / setinputs!(handle, x) | Read/write inputs |
getmetadata(handle) / setmetadata!(handle; kwargs...) | Read/write metadata |
ensure_output!(handle, name) | Create an output series if absent |
store_output!(handle, name, x) | Append to an output series |
get_output(handle, name, j) | Retrieve the j-th element of an output series |
get_outputs(handle, name) | Retrieve the full output series as a Vector |
output_length(handle, name) | Length of an output series |
output_names(handle) | Names of all output series |
has_output(handle, name) | Check if an output series exists |
empty_output!(handle, name) | Clear an output series |
Base.close(handle) | Close the handle and discard scratch |
Base.isopen(handle) | Whether the handle is still open |
Scratch storage is handled by default through the h.scratch::Dict{Symbol,Any} field on the handle (see StorageHandle); backends can override it if needed.