tokio.tools.common module

Common routines used to apply site-specific info to connectors

tokio.tools.common._expand_check_paths(template, lookup_key)[source]

Generate paths to examine from a variable-type template.

template may be one of three data structures:

  • str: search for files matching this exact template
  • list of str: search for files matching each template listed.
  • dict: use lookup_key to determine the element in the dictionary to use as the template. That value is treated as a new template object and can be of any of these three types.
Parameters:
  • template (str, list, or dict) – Template string(s) which should be passed to datetime.datetime.strftime to be converted into specific time-delimited files.
  • lookup_key (str or None) – When type(template) is dict, use this key to identify the key-value to use as template. If None and template is a dict, iterate through all values of template.
Returns:

List of strings, each describing a path to an existing HDF5 file that should contain data relevant to the requested start and end dates.
Return type:

list
tokio.tools.common._match_files(check_paths, use_time, match_first, use_glob)[source]

Locate file(s) that match a templated file path for a given time

Parameters:
  • check_paths (list of str) – List of templates to pass to strftime
  • use_time (datetime.datetime) – Time to pass through strftime to generate an actual file path to check for existence.
  • match_first (bool) – If True, only return the first matching file for each time increment checked. Otherwise, return _all_ matching files.
  • use_glob (bool) – Expand file globs in template
Returns:

List of strings, each describing a path to an existing HDF5 file that should contain data relevant to the requested start and end dates.
Return type:

list
tokio.tools.common.enumerate_dated_files(start, end, template, lookup_key=None, match_first=True, timedelta=datetime.timedelta(1), use_glob=False)[source]

Locate existing time-indexed files between a start and end time.

Given a start time, end time, and template data structure that describes a pattern by which the files of interest are indexed, locate all existing files that fall between the start and end time.

The template argument (template) are paths that are passed through datetime.strftime and then checked for existence for every timedelta increment between start and end, inclusive. template may be one of three data structures:

  • str: search for files matching this template
  • list of str: search for files matching each template. If match_first is True, only the first hit per list item per time interval is returned; otherwise, every file matching every template in the entire list is returned.
  • dict: use lookup_key to determine the element in the dictionary to use as the template. That value is treated as a new template object and can be of any of these three types.
Parameters:
  • start (datetime.datetime) – Begin including files corresponding to this start date, inclusive.
  • end (datetime.datetime) – Stop including files with timestamps that follow this end date. Resulting files _will_ include this date.
  • template (str, list, or dict) – Template string(s) which should be passed to datetime.strftime to be converted into specific time-delimited files.
  • lookup_key (str or None) – When type(template) is dict, use this key to identify the key-value to use as template. If None and template is a dict, iterate through all values of template.
  • match_first (bool) – If True, only return the first matching file for each time increment checked. Otherwise, return _all_ matching files.
  • timedelta (datetime.timedelta) – Increment to use when iterating between start and end while looking for matching files.
  • use_glob (bool) – Expand file globs in template
Returns:

List of strings, each describing a path to an existing HDF5 file that should contain data relevant to the requested start and end dates.
Return type:

list