solprop.output.metadata#

The run record: what produced an output file, and the resume key it is checked by.

An output file has to answer one question years later: what produced this? RunMetadata is that answer: the grid, the modes, the serialised physics configuration, the package versions, the git provenance of the running checkout, the noise seed and realisation, and the stepper settings.

The resume key. RunMetadata.hashed() is the part of the record that says what was computed: the schema version, the grid, the modes, the configuration, the noise, the stepper settings with the stepping method when it is not the tableau (SEP 13), and the identities and policies of the run’s material manifest, and nothing else: not the wall-clock time the run started, not the package versions, and not the git provenance. RunMetadata.configuration_hash() is a SHA-256 of its canonical JSON rather than hash(), so it is stable across processes; it is stamped in every file and is the exact identity a resume, a suspended writer, a snapshot and a scan row are checked by (ADR-066): a resume with different physics or numerics of the same shape is refused, naming the fields that changed (RunMetadata.summarise_differences()). A record read from a file is compared as it was written.

The manual page is docs/guide/results/provenance.md.

Module Attributes

`SCHEMA_VERSION`	Version of the saved-output layout; incremented when a stored field changes meaning.
`PEAK_FIELDS_ANNOTATION`	Name of the post-run annotation holding the material manifest's peak-field report.
`INTERPOLATED_RESUMES_ANNOTATION`	Name of the post-run annotation listing where a dense run was resumed, m.
`RUN_SUMMARY_ANNOTATION`	Name of the annotation holding the start-of-run summaries of a plane store.

Functions

resolve_metadata_write(existing, new, *[, force])

Decide whether a handler may write metadata, under one rule for both handlers.

Classes

RunMetadata(*, grid, modes, simulation_kind, ...)

Everything needed to say what produced a file, and to refuse a wrong resume.

solprop.output.metadata.INTERPOLATED_RESUMES_ANNOTATION = 'interpolated_resumes'#

Name of the post-run annotation listing where a dense run was resumed, m.

A list of the positions of the interpolated save planes a run observed densely was continued from with resume="interpolated": each continuation started afresh from a field interpolated inside a step, not a state the march passed through (SEP 14). Written by solprop.driver.simulation.Simulation.run(), outside the configuration hash.

solprop.output.metadata.PEAK_FIELDS_ANNOTATION = 'material_peak_fields'#

Name of the post-run annotation holding the material manifest’s peak-field report.

A list with one plain mapping per population with an optmat ionisation rate: the largest field it reached against its model’s maximum field, and the energy per event there against the constant the run paid (solprop.nonlinear.optmat_rates.PeakFieldReport). Written by solprop.driver.simulation.Simulation.run() when a run closes, outside the configuration hash (SEP 3, Q13).

solprop.output.metadata.RUN_SUMMARY_ANNOTATION = 'run_summary'#

Name of the annotation holding the start-of-run summaries of a plane store.

A list with one mapping per start, {"title": str, "rows": [[key, value], ...]}: a run that is resumed adds the summary of its continuation to the first one’s. Written when a run starts, not when it closes, and outside the configuration hash.

class solprop.output.metadata.RunMetadata(*, grid, modes, simulation_kind, configuration, stepper, noise=None, backend=None, versions=<factory>, git=<factory>, created='', schema_version=2, spectral_convention='optics', field_layout='(n_z, n_modes, n_pol, n_omega)', spatial=None, modal=None, materials=<factory>, initial_z=0.0, tdse=None, quadrature=None, reference=None, transverse=None, noise_resolution=None, defaults=None)[source]#

Bases: Displayed

Everything needed to say what produced a file, and to refuse a wrong resume.

Equality is by identity (eq=False); two runs are the same run when their configuration_hash() agree, which is what a resume actually asks.

Attributes:

gridGridRecord: The grid the run was computed on.
modestuple: One ModeRecord per propagated mode, in state order.
simulation_kind{‘mode_averaged’, ‘multimode’, ‘radial’, ‘free’, ‘slab’}: Which transform the run used.
configurationdict: The physics objects that built the run – responses, profiles, pulses, boundaries – as serialise() records them. Nothing is silently dropped: a callable is present as its repr with reproducible: False.
stepperStepperRecord: The solver settings and the schedule taken.
noiseNoiseRecord or None: The shot-noise realisation, or None for a noiseless run.
noise_resolutionNoiseResolution or None: How the noise model was chosen (SEP 37): the requested and the resolved model and any fallback’s reason. Outside the hash; None when the record’s builder did not say, as for a record built by hand.
defaultsDefaultsResolution or None: Which of the integrator, the controller and the collar scheme took the route’s default. Outside the hash; None when the record’s builder did not say.
backendBackendRecord or None: What executed the run, written by the driver from the backend’s own metadata(); None for a record built before a backend existed. Outside the hash, like the versions and the provenance beside it: a run continued on the other backend is the same run.
versionsdict: Package versions, from package_versions(). Outside the hash.
gitGitProvenance or None: Provenance of the running checkout, or None. Outside the hash.
createdstr: When the run started, ISO 8601 in UTC. Outside the hash.
schema_versionint: Version of this record layout, SCHEMA_VERSION; any other is refused.
spectral_conventionstr: Sign convention of the stored field, always 'optics'.
field_layoutstr: Axis order of the stored field.
spatialcollections.abc.Mapping or None: Spatial geometry, axis meanings, integration measures, field units and the plane measure (flux_model and flux_plan, SEP 34). Required for the radial, free and slab routes and None for modal runs. Included in the resume key.
modalcollections.abc.Mapping or None: The identity of a general modal guide: the ordered channel labels, the table, metric, response and provider identities, the sampling the reference is rebuilt from, the gauge, the frame, the active band, the noise reference basis and the accessors this basis admits, as solprop.translate.general.modal_identity() writes it. None for every other route. Included in the resume key, less the accessor list, so that a run continued against a different guide, a different table or a reordered mode axis is refused by the field that changed (solprop.output.capability).
materialssolprop.physdata.selection.MaterialManifest: The run’s material manifest: the optmat version, the identity of every optmat model the run resolved, its species, compositions and solids, the policies in force and what was measured about them. Empty for a run of solprop’s own materials. Its identities and policies are in the resume key (hashed()), so a run continued with another model, policy or optmat version is refused by the entry that changed.
initial_zfloat: Absolute launch position in metres; zero for a run launched at the entrance. Included in the resume key.
tdsecollections.abc.Mapping or None: Resolved scientific atomic descriptor, included in the resume key; None for a run without a TDSE response. Provider execution and device facts belong to the backend record.
quadraturecollections.abc.Mapping or None: The transverse rule of a resolved modal run, as solprop.modes.quadrature.TransverseQuadrature.record() writes it: the family, the counts, the nodes and the physical weights, in the resume key (SEP 19). None for the mode-averaged and spatial routes.
referencecollections.abc.Mapping or None: The plane-reference record (SEP 31): carrier, the grid’s reference frequency in rad/s; origin, the reference of the run’s path origin (phase, delay, calibrated), inherited from a parent plane for a run launched from one; parent, that parent’s provenance or None; frame, the frame integral of solprop.translate.coupling.frame_record(), or None for an operator supplied without frame tables; and basis, the route and the basis identity (solprop.coupling.plane.basis_identity()) at each planned plane z, the launch plane first. solprop.coupling.plane.reference_at() rebuilds any saved plane’s reference from it. Derived numbers, so outside the hash; the inherited origin enters it through configuration["coupling"]. Every run records one; None on a record built without it, as by hand, whose planes cannot be exchanged (solprop.processing.planes.field_plane()).
transversecollections.abc.Mapping or None: A free-space run’s resolved transverse boundary (SEP 40), as solprop.spatial_absorber.SpatialAbsorberDesign.record() writes it: the safety ramp’s start and strength, and a collar’s strength, band, threshold and step limit. Derived numbers, so outside the hash; the specification is in configuration["spatial_absorber"] where one was given. None without a transverse boundary.

Examples

>>> from solprop.grid import EnvelopeGrid
>>> from solprop.output.metadata import RunMetadata
>>> from solprop.output.records import GridRecord, StepperRecord
>>> from solprop.stepper import DOPRI5, PIController
>>> grid = EnvelopeGrid.from_limits(1.0, 835e-9, (400e-9, 2e-6), 2e-12)
>>> metadata = RunMetadata(
...     grid=GridRecord.from_grid(grid),
...     modes=(),
...     simulation_kind="mode_averaged",
...     configuration={"gamma": 0.11},
...     stepper=StepperRecord.from_settings(
...         DOPRI5, PIController(rtol=1e-8, atol=0.0)
...     ),
... )
>>> len(metadata.configuration_hash())
64
>>> metadata.spectral_convention
'optics'

backend: BackendRecord | None = None#

configuration: Mapping[str, object]#

configuration_hash()[source]#

Return the resume key: a digest of everything that was computed.

A SHA-256 of the canonical JSON of hashed(), so it is stable across processes – unlike hash(), which is salted per interpreter. It covers the physics and the numerics, and not only the statistic names and the state shape, so a resume continues exactly when the two hashes are equal (ADR-066).

Returns:

str: Sixty-four lowercase hexadecimal characters.

Examples

>>> from solprop.output.metadata import _example_metadata
>>> metadata = _example_metadata()
>>> metadata.configuration_hash() == metadata.configuration_hash()
True

created: str = ''#

defaults: DefaultsResolution | None = None#

differences(other)[source]#

Return the dotted paths at which two records describe different runs.

This is what a refused resume names, so that the reason is the field that changed rather than “the hash does not agree”.

Parameters:

otherRunMetadata: The record to compare against.

Returns:

tuple: Dotted paths into hashed(), in depth-first order; empty when the two describe the same run.

Examples

>>> import dataclasses
>>> from solprop.output.metadata import _example_metadata
>>> one = _example_metadata()
>>> two = dataclasses.replace(one, configuration={"gamma": 0.22})
>>> one.differences(two)
('configuration.gamma',)

field_layout: str = '(n_z, n_modes, n_pol, n_omega)'#

classmethod from_json(text, arrays=None)[source]#

Return the record a JSON document describes.

The configuration comes back as the plain structures serialise() made of it, not as the physics objects: a callable was recorded by repr and cannot be rebuilt. That is enough for the hash, which is what a resume compares.

Parameters:

textstr: A document produced by to_json().
arrayscollections.abc.Mapping or None, optional: What record_arrays() gave for it, by digest, or an HDF5 group holding them. Each comes back in place of its reference, its digest checked.

Returns:

RunMetadata: The record, hashing identically to the one that produced text.

Raises:

OutputError: If the document is not valid JSON, declares an output schema other than SCHEMA_VERSION, a required field is missing, or an array it names is absent from arrays or does not have its digest.

Examples

>>> from solprop.output.metadata import _example_metadata
>>> metadata = _example_metadata()
>>> back = RunMetadata.from_json(metadata.to_json())
>>> back.configuration_hash() == metadata.configuration_hash()
True

git: GitProvenance | None#

grid: GridRecord#

hashed()[source]#

Return the part of the record the resume key is taken over.

Everything that describes what was computed, and nothing that describes when or with which build: the wall-clock time, the package versions and the git provenance are excluded, so that a resume survives an unrelated commit or a dirty working tree, as long as they move no number the record holds; a new library or platform may move one, and the resume is then refused. modal.accessors is excluded for the same reason – it is what this build can compute, not what this run did (_hashed_modal).

Returns:

dict: The serialised schema version, grid, modes, simulation kind, configuration, noise and stepper settings, the spatial record, the power convention, the general modal identity, the material manifest’s identities and policies, the launch position, the TDSE descriptor and the transverse quadrature.

Examples

>>> from solprop.output.metadata import _example_metadata
>>> sorted(_example_metadata().hashed())[:4]
['configuration', 'grid', 'initial_z', 'materials']

initial_z: float = 0.0#

materials: MaterialManifest#

modal: Mapping[str, object] | None = None#

modes: tuple[ModeRecord, ...]#

noise: NoiseRecord | None = None#

noise_resolution: NoiseResolution | None = None#

property power_convention: str[source]#

Amplitude label of the stored field.

Always "poynting": the interface modal amplitude, whose squared modulus is the true power in watts (docs/dev/conventions.md §4).

quadrature: Mapping[str, object] | None = None#

record_arrays()[source]#

Return the arrays stored beside the record, by digest.

What to_json()’s references name: each numeric array of more than RECORD_ARRAY_LIMIT elements. A store keeps them with the document, and from_json() takes them back.

Returns:

dict: The arrays, not copied, each by the digest of its bytes with its dtype and shape.

Examples

>>> import dataclasses
>>> import numpy as np
>>> from solprop.output.metadata import _example_metadata
>>> metadata = dataclasses.replace(
...     _example_metadata(), configuration={"rate": np.zeros(5000)}
... )
>>> [array.size for array in metadata.record_arrays().values()]
[5000]

reference: Mapping[str, object] | None = None#

schema_version: int = 2#

simulation_kind: Literal['mode_averaged', 'multimode', 'radial', 'free', 'slab']#

spatial: Mapping[str, object] | None = None#

spectral_convention: str = 'optics'#

stepper: StepperRecord#

summarise_differences(other, *, shown=4)[source]#

Say how two records differ, briefly enough to read in a refusal.

The paths of differences(), with every array’s element paths grouped under the array’s name and counted. Every part of the record that changed (configuration, materials, noise, schema_version, …) is named, with its first shown entries and a count of the rest, so that no cause is hidden behind another part’s many entries. Where the noise differs and this record’s realisation was drawn from a known seed, the summary says which seed draws it again.

Parameters:

otherRunMetadata: The record to compare against; this one is the stored record.
shownint, optional: How many entries to name in each part of the record.

Returns:

str: "these fields changed: ...", empty of entries when the two describe the same run.

Examples

>>> import dataclasses
>>> from solprop.output.metadata import _example_metadata
>>> one = _example_metadata()
>>> two = dataclasses.replace(
...     one, configuration={"gamma": 0.22, "betas": [1.0, 2.0, 3.0]}
... )
>>> one.summarise_differences(two)
'these fields changed: configuration.betas, configuration.gamma'
>>> three = dataclasses.replace(one, configuration={"gamma": [0.1, 0.2, 0.3]})
>>> three.summarise_differences(
...     dataclasses.replace(one, configuration={"gamma": [0.4, 0.5, 0.6]})
... )
'these fields changed: configuration.gamma (3 elements)'

tdse: Mapping[str, object] | None = None#

to_json()[source]#

Return the whole record as one JSON document.

An array of more than RECORD_ARRAY_LIMIT elements is a reference to what record_arrays() holds.

Returns:

str: Canonical JSON carrying the hashed part plus the whole general modal identity, whose published accessor list the hashed part leaves out, the whole material manifest, the plane-reference record, the noise model’s resolution, the conventions, the backend record, the versions, the git provenance and the creation time.

Examples

>>> import json
>>> from solprop.output.metadata import _example_metadata
>>> sorted(json.loads(_example_metadata().to_json()))[:3]
['backend', 'configuration', 'created']

transverse: Mapping[str, object] | None = None#

property transverse_measure: Literal['area', 'per_unit_width'][source]#

What the run’s transverse integrals measure.

A run on modes uniform in y (solprop.modes.slab.SlabMode) records configuration["transverse_measure"] = "per_unit_width": its powers are W/m, its energies J/m and its intensities still W/m². Every other run measures an area. The key sits in the free-form configuration and is written only where it is not the default.

"area" or "per_unit_width".

Raises:

OutputError: If the stored value is neither.

Examples

>>> from solprop.output.metadata import _example_metadata
>>> _example_metadata().transverse_measure
'area'

versions: Mapping[str, str]#

with_backend(backend)[source]#

Return this record with a run’s execution facts, its resume key kept.

The backend record is outside the hash, so the hashed part and the resume key already computed stand: a run opening its output does not serialise and digest its configuration a second time, which cost a capillary run 30 ms and a large launch seconds (solprop issue #347).

Parameters:

backendBackendRecord: The execution facts of the prepared backend.

Returns:

RunMetadata: A copy of this record with backend in place of its own.

Examples

>>> from solprop.output.metadata import _example_metadata
>>> from solprop.output.provenance import BackendRecord
>>> metadata = _example_metadata()
>>> backend = BackendRecord(name="numpy_ref", propagator_mode="tabulate")
>>> executed = metadata.with_backend(backend)
>>> executed.backend.name
'numpy_ref'
>>> executed.configuration_hash() == metadata.configuration_hash()
True

solprop.output.metadata.SCHEMA_VERSION = 2#

Version of the saved-output layout; incremented when a stored field changes meaning.

Schema 1 is the first released layout, 0.1.0’s: the record types of this module, written by solprop.output.memory.MemoryOutput and solprop.output.hdf5.HDF5Output as docs/guide/results/output.md describes them. Schema 2 records DataPulse.spectral_phase. A record that declares any other version is refused by name (RunMetadata): files written by an earlier release or by development versions of solprop, which declared output schemas up to 28, are not read, and nor is a file from a later release. Read such a file with the solprop version that wrote it.

solprop.output.metadata.resolve_metadata_write(existing, new, *, force=False)[source]#

Decide whether a handler may write metadata, under one rule for both handlers.

Metadata is written once. A second write describing the same run is a no-op; one describing a different run is a mistake and raises, naming the fields that changed, unless the caller insists with force, in which case the record is replaced and the replacement is logged.

“The same run” means the same RunMetadata.configuration_hash(), so a rerun of identical physics under a newer NumPy, or from a checkout that has since gained a commit, is still the same run.

Parameters:

existingRunMetadata or None: What the handler already holds, or None when it holds nothing.
newRunMetadata: What the caller is offering.
forcebool, optional: Whether to overwrite a different record instead of refusing.

Returns:

bool: Whether the caller should write new.

Raises:

OutputError: If a different record is already held and force is false.

Examples

>>> from solprop.output.metadata import _example_metadata, resolve_metadata_write
>>> metadata = _example_metadata()
>>> resolve_metadata_write(None, metadata)
True
>>> resolve_metadata_write(metadata, metadata)
False