qbitbridge package¶
Submodules¶
qbitbridge.clusters module¶
qbitbridge.options module¶
options class for handling arguments of workflow. Not yet implemented
qbitbridge.utils module¶
Collection of functions and tooling intended for general usage. The key functionality to explore here is the EventFile class.
- class qbitbridge.utils.EventFile(name: str, loc: str, sampling: float = 0.01, id: str | None = None, etime: str | None = None, eset: int | None = None)[source]¶
Bases:
objectSimple class to create a file for a given event.
- event_loc: str¶
directory where to store file event locks
- event_name: str¶
The name of the event
- event_set: int¶
Counter for number of times set
- event_time: str¶
Time of event creation
- fname: str¶
File name where event will be saved
- identifer: str¶
unique identifer
- sampling: float¶
how often to check for event file
- class qbitbridge.utils.PostgresConfiguration(scratch: str, user: str = 'postgres', db: str = 'orion', password: str = 'qbitbridge_test', port: int = 5432, max_connections: int = 1000, shared_buffers: str = '1024MB', container: str = 'postgres_latest.sif', container_engine: str = 'singularity', container_engine_args: str | None = None, dry_run: bool = False, delay_time: int = 20)[source]¶
Bases:
NamedTupleSimple class to store postgres launcher information
- container: str¶
The container image used to run postgres
- container_engine: str¶
The container engine used to run the postgres container
- container_engine_args: str | None¶
The container engine arguments used to run the postgres container
- db: str¶
The database name for the postgres database
- delay_time: int¶
The delay in seconds to wait before starting the prefect server after starting postgres
- dry_run: bool¶
If True, do not actually launch the postgres container, just print the command
- max_connections: int¶
The maximum number of connections for the postgres database
- password: str¶
The password for the postgres database
- port: int¶
The port for postgres database
- scratch: str¶
The scratch directory for the postgres database
The shared buffers for the postgres database.
- user: str¶
The user for the postgres database
- class qbitbridge.utils.PrefectConfiguration(home: str, web_concurrency: int = 16, sqlalchemy_pool_size: int = 5, sqlalchemy_max_overflow: int = 10, port: int = 4200, timeout_keep_alive: int = 10, limit_max_requests: int = 4096, timeout_graceful_shutdown: int = 7200, dry_run: bool = False, delay_time: int = 20)[source]¶
Bases:
NamedTupleSimple class to store prefect launcher information
- delay_time: int¶
The delay in seconds to wait before starting the prefect server after starting postgres
- dry_run: bool¶
If True, do not actually launch the prefect server, just print the command
- home: str¶
Path to the prefect home directory
- limit_max_requests: int¶
The limit for the prefect server
- port: int¶
The port for the prefect server
- sqlalchemy_max_overflow: int¶
The max overflow for the sqlalchemy connection
- sqlalchemy_pool_size: int¶
The pool size for the sqlalchemy connection
- timeout_graceful_shutdown: int¶
The timeout for the prefect server
- timeout_keep_alive: int¶
The timeout for the prefect server
- web_concurrency: int¶
Number of workers for prefect webserver (uvicorn under the hood)
- class qbitbridge.utils.QBitBridgeLauncher(config: Dict)[source]¶
Bases:
object- logger = <Logger qbitbridge.utils (WARNING)>¶
Simple class to store qbitbridge launcher information and start postgres and prefect services
- class qbitbridge.utils.SlurmInfo(hostname: str, resource: str | None = None, job_id: str | None = None, task_id: str | None = None, time: str | None = None)[source]¶
Bases:
NamedTupleSimple class to store slurm information
- hostname: str¶
The hostname of the slurm job
- job_id: str | None¶
The job ID of the slurm job
- resource: str¶
The slurm resource request
- task_id: str | None¶
The task ID of the slurm job
- time: str | None¶
The time time the job information was gathered
- async qbitbridge.utils.async_create_markdown_artifcat(key, markdown, description) None[source]¶
create a markdown artifact in a asynchronous fashion. Wrapper allows more complexity to be added.
- qbitbridge.utils.check_file_can_be_created(filename: str) bool[source]¶
check if file can be created
- Parameters:
filename (str) – filename to check
- Returns:
bool if creatable
- qbitbridge.utils.check_python_installation(library: str)[source]¶
Check if library present and otherwise catch ImporError and report missing library
- Parameters:
library (str) – name of library to check
- Returns:
bool of whether library can be imported
- qbitbridge.utils.get_argparse_args(arguments: str, parser: ArgumentParser) Namespace[source]¶
Parse a string based on an argparser and also strip out _ from an argument
- Parameters:
arguments (str) – string of arguments
parser (argparse.ArgumentParser) – parser that processes list of strings
- Returns:
The argparser namespace
- qbitbridge.utils.get_environment_variable(variable: str | None, default: str | None = None) str | None[source]¶
Get the value of an environment variable if it exists. If it does not a None is returned.
- Parameters:
variable (Union[str,None]) – The variable to lookup. If it starts with $ it is removed. If None is provided None is returned.
default (Optional[str], optional) – If the variable lookup is not resolved this is returned. Defaults to None.
- Returns:
Value of environment variable if it exists. None if it does not.
- Return type:
Union[str,None]
- async qbitbridge.utils.get_flow_runs(flow_run_filter: prefect.client.schemas.filters.FlowRunFilter, sort: str = '-start_time', limit: int = 100) List[prefect.client.schemas.objects.FlowRun][source]¶
Get list of flow runs that satisfy some filter
- qbitbridge.utils.get_job_info(mode: str = 'slurm') SlurmInfo[source]¶
Get the job information for the supplied mode
- Parameters:
mode (str, optional) – Which mode to poll information for. Defaults to “slurm”.
- Raises:
ValueError – Raised if the mode is not supported
- Returns:
The specified mode
- Return type:
Union[SlurmInfo]
- qbitbridge.utils.get_num_gpus() Tuple[int, str][source]¶
Poll node for number of gpus on host
- Returns:
int number of gpus on a node and the type
- qbitbridge.utils.get_slurm_info() SlurmInfo[source]¶
Collect key slurm attributes of a job
- Returns:
Collection of slurm items from the job environment
- Return type:
- qbitbridge.utils.get_task_run_id() str[source]¶
Get the Task ID of the task calling this function. If there is no context, then the task_run_id is set to a descriptive, non-unique value
- qbitbridge.utils.load_config(config_path: str, log_level: str = 'INFO') QBitBridgeLauncher[source]¶
Load a YAML configuration file for launching all relevant services and return it as a QBitBridgeLauncher object. :param config_path: Path to the YAML configuration file. :type config_path: str :param log_level: Logging level. :type log_level: str
- Returns:
An instance of QBitBridgeLauncher initialized with the loaded configuration.
- Return type:
- qbitbridge.utils.log_slurm_job_environment(logger) SlurmInfo[source]¶
Log components of the slurm environment.
- Returns:
Collection of slurm items from the job environment
- Return type:
- qbitbridge.utils.run_a_process(shell_cmd: list, add_output_to_log: bool = False, logger=None)[source]¶
Runs a process given by the shell command. If given a logger and asked to append, adds to the logger.
- Returns:
new proccess spawned by the shell_cmd
- Return type:
subprocess
- qbitbridge.utils.run_a_process_bg(shell_cmd: list, add_output_to_log: bool = False, sleeplength: float = 5, logger=None) None[source]¶
Runs a process given by the shell command. If given a logger and asked to append, adds to the logger.
- Returns:
new proccess spawned by the shell_cmd
- Return type:
subprocess
- qbitbridge.utils.run_a_srun_process(shell_cmd: list, srunargs: list = [], add_output_to_log: bool = False, logger=None) Popen[source]¶
runs a srun process given by the shell command. If given a logger and asked to append, adds to the logger.
- Returns:
new proccess spawned by the shell_cmd
- Return type:
subprocess.Popen
- async qbitbridge.utils.save_artifact(data: Any, key: str = 'key', description: str = 'Data to be shared between subflows')[source]¶
Use this to save data between workflows and tasks. Best used for small artifacts
- Parameters:
() (data) – data to be saved
key (str) – key for accessing the data
description (str) – description of the data
- Returns :
a markdown artifact to transmit data between workflows
- async qbitbridge.utils.upload_image_as_artifact(image_path: Path, key: str = '', description: str | None = None) None[source]¶
Create and submit a markdown artifact tracked by prefect for an input image. Currently supporting png formatted images.
The input image is converted to a base64 encoding, and embedded directly within the markdown string. Therefore, be mindful of the image size as this is tracked in the postgres database.
- Parameters:
image_path (Path) – Path to the image to upload
key (str) – A key. Defaults to filename with lower_case.
description (Optional[str], optional) – A description passed to the markdown artifact. Defaults to None.