Helpers for Loading Datasets

Provides functions to easily load lyDATA CSV tables as pandas.DataFrame.

The loading itself is implemented in the LyDatasetConfig class, which is a pydantic.BaseModel subclass. It validates the unique specification that identifies a dataset and then allows loading it from the disk (if present) or from GitHub.

The available_datasets() function can be used to create a generator of such LyDatasetConfig instances, corresponding to all available datasets that are either found on disk or on GitHub.

Consequently, the load_datasets() function can be used to load all datasets matching the given specs/pattern. It takes the same arguments as the function available_datasets() but returns a generator of pandas.DataFrame instead of LyDatasetConfig.

Lastly, with the join_datasets() function, one can load and concatenate all datasets matching the given specs/pattern into a single pandas.DataFrame.

The docstring of all functions contains some basic doctest examples.

exception lydata.loader.SkipDiskError

Raised when the user wants to skip loading from disk.

class lydata.loader.LyDatasetConfig(*, year: Annotated[int, Gt(gt=0), Le(le=2024)], institution: Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)], subsite: Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)], repo: Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)] = 'rmnldwg/lydata', ref: Annotated[str, StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)] = 'main')

Specification of a dataset.

property name: str

Get the name of the dataset.

>>> conf = LyDatasetConfig(year=2023, institution="clb", subsite="multisite")
>>> conf.name
'2023-clb-multisite'

property path: Path

Get the path to the dataset.

>>> conf = LyDatasetConfig(year="2021", institution="usz", subsite="oropharynx")
>>> conf.path.exists()
True

get_url(file: str) → str

Get the URL to the dataset’s directory, CSV file, or README file.

>>> LyDatasetConfig(
...     year=2021,
...     institution="clb",
...     subsite="oropharynx",
...     ref="6ac98d",
... ).get_url("data.csv")
'https://raw.githubusercontent.com/rmnldwg/lydata/6ac98d/2021-clb-oropharynx/data.csv'

get_description() → str

Get the description of the dataset.

First, try to load it from the README.md file that should sit right next to the data.csv file. If that fails, try to look for the README.md file in the GitHub repository.

>>> conf = LyDatasetConfig(year=2021, institution="clb", subsite="oropharynx")
>>> print(conf.get_description())   
# 2021 CLB Oropharynx
...

load(skip_disk: bool = False, **load_kwargs) → DataFrame

Load the data.csv file from disk or from GitHub.

One can also choose to skip_disk. Any keyword arguments are passed to pandas.read_csv().

The method will store the output of model_dump() in the attrs attribute of the returned DataFrame.

>>> conf = LyDatasetConfig(year=2021, institution="clb", subsite="oropharynx")
>>> df_from_disk = conf.load()
>>> df_from_disk.shape
(263, 82)
>>> df_from_github = conf.load(skip_disk=True)
>>> np.all(df_from_disk.fillna(0) == df_from_github.fillna(0))
np.True_

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'institution': FieldInfo(annotation=str, required=True, description="Institution's short code. E.g., University Hospital Zurich: `usz`.", metadata=[StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)]), 'ref': FieldInfo(annotation=str, required=False, default='main', description='Branch/tag/commit of the repo.', metadata=[StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)]), 'repo': FieldInfo(annotation=str, required=False, default='rmnldwg/lydata', description='GitHub `repository/owner`.', metadata=[StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)]), 'subsite': FieldInfo(annotation=str, required=True, description='Subsite(s) this dataset covers.', metadata=[StringConstraints(strip_whitespace=None, to_upper=None, to_lower=True, strict=None, min_length=1, max_length=None, pattern=None)]), 'year': FieldInfo(annotation=int, required=True, description='Release year of dataset.', metadata=[Gt(gt=0), Le(le=2024)])}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

lydata.loader.remove_subheadings(tokens: Iterable[Token], min_level: int = 1) → list[Token]

Remove anything under min_level headings.

With this, one can truncate markdown content to e.g. to the top-level heading and the text that follows immediately after. Any subheadings after that will be removed.

lydata.loader.format_description(readme: TextIOWrapper | str, short: bool = False, max_line_length: int = 60) → str

Get a markdown description from a file.

Truncate the description before the first second-level heading if short is set to True.

lydata.loader.available_datasets(year: int | str = '*', institution: str = '*', subsite: str = '*', search_paths: list[Path] | None = None, skip_disk: bool = False, repo: str = 'rmnldwg/lydata', ref: str = 'main') → Generator[LyDatasetConfig, None, None]

Generate LyDatasetConfig instances of available datasets.

The arguments year, institution, and subsite represent glob patterns and all datasets matching these patterns can be iterated over using the returned generator.

By default, the functions will look for datasets on the disk at paths specified in the search_paths argument. If no paths are provided, it will look in the the parent directory of the directory containing this file. If the library is installed, this will be the site-packages directory.

With skip_disk set to True, the function will not look for datasets on disk, but will instead look for them on GitHub. The repo and ref arguments can be used to specify the repository and the branch/tag/commit to look in.

>>> avail_gen = available_datasets()
>>> sorted([ds.name for ds in avail_gen])   
['2021-clb-oropharynx',
 '2021-usz-oropharynx',
 '2023-clb-multisite',
 '2023-isb-multisite']
>>> avail_gen = available_datasets(skip_disk=True)
>>> sorted([ds.name for ds in avail_gen])   
['2021-clb-oropharynx',
 '2021-usz-oropharynx',
 '2023-clb-multisite',
 '2023-isb-multisite']
>>> avail_gen = available_datasets(
...     institution="hvh",
...     ref="6ac98d",
...     skip_disk=True,
... )
>>> sorted([ds.get_url("") for ds in avail_gen])   
['https://raw.githubusercontent.com/rmnldwg/lydata/6ac98d/2024-hvh-oropharynx/']

lydata.loader.load_datasets(year: int | str = '*', institution: str = '*', subsite: str = '*', search_paths: list[Path] | None = None, skip_disk: bool = False, repo: str = 'rmnldwg/lydata', ref: str = 'main', **kwargs) → Generator[DataFrame, None, None]

Load matching datasets from the disk.

It loads every dataset from the LyDatasetConfig instances generated by the available_datasets() function, which also receives all arguments of this function.

lydata.loader.join_datasets(year: int | str = '*', institution: str = '*', subsite: str = '*', search_paths: list[Path] | None = None, skip_disk: bool = False, repo: str = 'rmnldwg/lydata', ref: str = 'main', **kwargs) → DataFrame

Join matching datasets from the disk.

This uses the load_datasets() function to load the datasets and then concatenates them along the index axis. All arguments are also directly passed to the load_datasets() function.

>>> join_datasets(year="2023").shape
(705, 219)
>>> join_datasets(year="2023", skip_disk=True).shape
(705, 219)