specsr.data.ingest

Discovery and reading of raw JADES NIRSpec x1d products.

The raw tree is laid out by field, disperser and tier:

DR4/<field>/spectra/<disperser>/<tier>/hlsp_jades_jwst_nirspec_<tier>-<id>_<disperser>_v1.0_x1d.fits

Two facts about this layout drive the design.

A target can appear under more than one tier. In DR4, 33 of 5,157 targets are observed in two tiers (for example goods-n-mediumhst and goods-n-mediumjwst). Keying a galaxy on (field, tier, target_id) would turn each of those into two independent “galaxies”, which then land on opposite sides of a train/test split — reintroducing exactly the duplicate-object leak that the DR3 sample suffered from. The identity of a galaxy here is therefore (field, target_id), and multiple tiers are alternative observations of the same object.

Not every target has every disperser. DR4 has 5,106 prism, 4,728 G140M, 4,539 G235M and 4,728 G395M exposures across GOODS-N and GOODS-S. Pairing must be explicit about what it requires rather than assuming a complete grid.

Module Attributes

PRISM

Directory names of the dispersers we use.

Functions

discover_spectra(release_dir[, fields, ...])

Enumerate x1d products under a release directory.

group_by_target(files[, require_prism, ...])

Group files into candidate pairs, keyed by (field, target_id).

read_spectrum(path[, extension])

Read one x1d product.

Classes

SpectrumFile(path, field, tier, target_id, ...)

One x1d product on disk, identified without opening it.
class specsr.data.ingest.SpectrumFile(path, field, tier, target_id, disperser)[source]

Bases: object

One x1d product on disk, identified without opening it.

Parameters:
path: Path
field: str
tier: str
target_id: str
disperser: str
property key: tuple[str, str]

(field, target_id).

Deliberately excludes the tier — see the module docstring.

Type:

Galaxy identity

property is_prism: bool
specsr.data.ingest.PRISM = 'clear-prism'

Directory names of the dispersers we use.

specsr.data.ingest.discover_spectra(release_dir, fields=('goods-n', 'goods-s'), dispersers=('clear-prism', 'f070lp-g140m', 'f170lp-g235m', 'f290lp-g395m'))[source]

Enumerate x1d products under a release directory.

Filenames are parsed rather than headers opened, so this is fast enough to run over the whole tree (~20k files) before deciding what to read.

Parameters:
Return type:

list[SpectrumFile]

specsr.data.ingest.read_spectrum(path, extension='EXTRACT3PIX1D')[source]

Read one x1d product.

Parameters:

  • path (Path | str) – FITS file to read.

  • extension (str) – Which extraction to use. EXTRACT3PIX1D is the 3-pixel extraction; EXTRACT5PIX1D is the wider aperture. The choice must be the same for the low- and high-resolution members of a pair, or their flux scales differ systematically and the model learns the aperture difference as if it were a resolution effect.

Returns:

  • dict with wavelength (µm), flux, flux_err (erg s⁻¹ cm⁻² Å⁻¹),

  • plus ra, dec, target, grating, filter, tier.

  • Non-finite samples are preserved rather than filled (downstream resampling)

  • ignores them, and silently interpolating over a detector gap would invent

  • flux that was never observed.

Return type:

dict

specsr.data.ingest.group_by_target(files, require_prism=True, require_gratings=1)[source]

Group files into candidate pairs, keyed by (field, target_id).

Parameters:

  • require_prism (bool) – Drop targets with no prism exposure; the prism is the model’s input.

  • require_gratings (int) – Minimum number of distinct medium gratings required. The reference is stitched from up to three, and a target contributing only one covers a small fraction of the 1–5 µm range.

  • files (list[SpectrumFile])

Returns:

  • {(field, target_id): {disperser: [SpectrumFile, ...]}}. A disperser maps

  • to a *list because the same target may have been observed in more than one*

  • tier; keeping all of them lets the pairing step choose (or combine) rather

  • than having the choice made silently here.

Return type:

dict[tuple[str, str], dict[str, list[SpectrumFile]]]