tokio.connectors.slurm module

Connect to Slurm via Slurm CLI outputs.

This connector provides Python bindings to retrieve information made available through the standard Slurm saccount and scontrol CLI commands. It is currently very limited in functionality.

class tokio.connectors.slurm.Slurm(jobid=None, *args, **kwargs)

Bases: tokio.connectors.common.SubprocessOutputDict

Dictionary subclass that self-populates with Slurm output data

Presents a schema that is keyed as:

{
    taskid: {
        slurmfield1: value1
        slurmfield2: value2
        ...
    }
}

where taskid can be any of

• jobid
• jobid.<step>
• jobid.batch

__init__(jobid=None, *args, **kwargs)

Load basic information from Slurm

Parameters:jobid (str) – Slurm Job ID associated with data this object contains Variables:jobid (str) – Slurm Job ID associated with data contained in this object

__repr__()

Serialize object in the same format as sacct.

Returns:Serialized version of self in a similar format as the sacct output so that this object can be circularly serialized and deserialized. Return type:str

_recast_keys(*target_keys)

Convert own keys into native Python objects.

Scan self and convert special keys into native Python objects where appropriate. If no keys are given, scan everything. Do NOT attempt to recast anything that is not a string–this is to avoid relying on expand_nodelist if a key is already recast since expand_nodelist does not function outside of an environment containing Slurm.

Parameters:*target_keys (list, optional) – Only convert these keys into native Python object types. If omitted, convert all keys.

from_json(json_string)

Initialize self from a JSON-encoded string.

Parameters:json_string (str) – JSON representation of self

get_job_ids()

Return the top-level jobid(s) contained in object.

Retrieve the jobid(s) contained in self without any accompanying taskid information.

Returns:list of jobid(s) contained in self. Return type:list of str

get_job_nodes()

Return a list of all job nodes used.

Creates a list of all nodes used across all tasks for the self.jobid. Useful if the object contains only a subset of tasks executed by the Slurm job.

Returns:Set of node names used by the job described by this object Return type:set

get_job_startend()

Find earliest start and latest end time for a job.

For an entire job and all its tasks, find the absolute earliest start time and absolute latest end time.

Returns:Two-item tuple of (earliest start time, latest end time) in whatever type self['start'] and self['end'] are stored Return type:tuple

load()

Initialize values either from cache or sacct

load_keys(*keys)

Retrieve a list of keys from sacct and insert them into self.

This always invokes sacct and can be used to overwrite the contents of a cache file.

Parameters:*keys (list) – Slurm attributes to include; names should be valid input to sacct –format CLI utility.

load_str(input_str)

Load from either a json cache or the output of sacct

to_dataframe()

Convert self into a Pandas DataFrame.

Returns a Pandas DataFrame representation of this object.

Returns:DataFrame representation of the same schema as the Slurm sacct command. Return type:pandas.DataFrame

to_json(**kwargs)

Return a json-encoded string representation of self.

Serializes self to json using _RECAST_KEY_MAP to convert Python types back into JSON-compatible types.

Returns:JSON representation of self Return type:str

class tokio.connectors.slurm.SlurmEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)

Bases: json.encoder.JSONEncoder

Encode sets as lists and datetimes as ISO 8601.

default(o)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

tokio.connectors.slurm._RECAST_KEY_MAP = {'end': (<function <lambda>>, <function <lambda>>), 'nodelist': (<function expand_nodelist>, <function compact_nodelist>), 'start': (<function <lambda>>, <function <lambda>>)}

Methods to convert Slurm string outputs into Python objects

This table provides the methods to apply to various Slurm output keys to convert them from strings (the default Slurm output type) into more useful Python objects such as datetimes or lists.

• value[0] is the function to cast to Python
• value[1] is the function to cast back to a string

Type:dict

tokio.connectors.slurm.compact_nodelist(node_string)

Convert a string of nodes into compact representation.

Wraps scontrol show hostlist nid05032,nid05033,... to compress a list of nodes to a Slurm nodelist string. This is effectively the reverse of expand_nodelist()

Parameters:node_string (str) – Comma-separated list of node names (e.g., nid05032,nid05033,...) Returns:The compact representation of node_string (e.g., nid0[5032-5159]) Return type:str

tokio.connectors.slurm.expand_nodelist(node_string)

Expand Slurm compact nodelist into a set of nodes.

Wraps scontrol show hostname nid0[5032-5159] to expand a Slurm nodelist string into a list of nodes.

Parameters:node_string (str) – Node list in Slurm’s compact notation (e.g., nid0[5032-5159]) Returns:Set of strings which encode the fully expanded node names contained in node_string. Return type:set

tokio.connectors.slurm.jobs_running_between(start, end, keys=None)

Generate a list of Slurm jobs that ran between a time range

Parameters:

• start (datetime.datetime) – Find jobs that ended at or after this time
• end (datetime.datetime) – Find jobs that started at or before this time
• state (str) – Any valid sacct state
• keys (list) – List of Slurm fields to return for each running job

Returns:

Slurm object containing jobs whose runtime overlapped with the start and end times

Return type:

tokio.connectors.slurm.Slurm

tokio.connectors.slurm.parse_sacct(sacct_str)

Convert output of sacct -p into a dictionary.

Parses the output of sacct -p and return a dictionary with the full (raw) contents.

Parameters:sacct_str (str) – stdout of an invocation of sacct -p Returns:Keyed by Slurm Job ID and whose values are dicts containing key-value pairs corresponding to the Slurm quantities returned by sacct -p. Return type:dict