gc3libs.cmdline

Base classes for GC3Libs-based scripts.

Classes implemented in this file provide common and recurring functionality for GC3Libs command-line utilities and scripts. User applications should implement their specific behavior by subclassing and overriding a few customization methods.

The following public classes are exported from this module:

SessionBasedScript

Base class for the grosetta/ggamess/gcodeml scripts. Implements a long-running script to submit and manage a large number of tasks grouped into a “session”.

SessionBasedDaemon

Base class for GC3Pie servers. Implements a long-running daemon with XML-RPC interface and support for “inboxes” (which can add or remove tasks based on external events).

DaemonClient

Command-line client for interacting with instances of a SessionBasedDaemon via XML-RPC.

class gc3libs.cmdline.DaemonClient(**extra_args)

Send XML-RPC requests to a running SessionBasedDaemon.

The generic command line looks like the following:

PROG client SERVER CMD [ARG [ARG …]]

The SERVER string is the URL where the XML-RPC server can be contacted. A pair hostname:port is accepted as abbreviation for http://hostname:port/ and a simple :port string is a valid alias for http://localhost:port/. Alternatively, the SERVER argument can be the path to the daemon.url path where a running server writes its contact information.

COMMAND is an XML-RPC command name; valid commands depend on the server and can be listed by using help as the COMMAND string (with no further arguments). The remaining ARGs (if any) depend on COMMAND.

pre_run()

Perform parsing of standard command-line options and call into parse_args() to do non-optional argument processing.

Also sets up the gc3.gc3utils logger; it is controlled by the -v/--verbose command-line option. Up to self.verbose_logging_threshold occurrences of -v are ignored, after which they start to lower the level of messages sent to standard error output.

setup_args()

Override this method to replace standard command-line arguments.

setup_options()

Override this method to add command-line options.

class gc3libs.cmdline.SessionBasedDaemon(**extra_args)

Base class for GC3Pie daemons. Implements a long-running script that can daemonize, provides an XML-RPC interface to interact with the current workflow and implement the concept of “inbox” to trigger the creation of new jobs as soon as a new file is created on a folder, or is available on an HTTP(S) or SWIFT endpoint.

The generic script implements a command line like the following:

PROG [server options] INBOX [INBOX ...]

class Commands(parent)

User-visible XML-RPC methods.

Subclass this to override default methods or add new ones.

Note

Every public attribute of this class is exposed by the server; make sure that anithing which is not a public method is prefixed with _.

kill(jobid=None)

Usage: kill JOBID

Abort execution of a task and set it to TERMINATED state.

list(*opts)

Usage: list [daemon|session] [json|text|yaml]

List IDs of tasks managed by this daemon. If the word session is present on the command-line, then tasks stored in the session are printed instead (which may be a superset of the tasks managed by the engine).

One of the words json, yaml, or text (simple list of IDs, one per line) can be used to choose the output format, with text being the default.

list_details(*opts)

Usage: list_details [daemon|session] [json|text|yaml]

Give information about tasks managed by this daemon; for each task, the following information are printed:

• task name
• execution state (e.g., NEW, RUNNING, etc.)
• process exit code (only meaningful if state is TERMINATED)
• last line in the execution log

If the word session is present on the command-line, then tasks stored in the session are printed instead (which may be a superset of the tasks managed by the engine).

One of the words json, yaml, or text (human-readable plain text table) can be used to choose the output format, with text being the default.

manage(jobid=None)

Usage: manage JOBID

Tell daemon to start actively managing a task.

redo(jobid=None, from_stage=None)

Usage: redo JOBID [STAGE]

Resubmit the task identified by JOBID. If task is a SequentialTaskCollection, then resubmit it from the given stage (identified by its integer index in the collection; by default, sequential task collections resume from the very first task).

Only tasks in TERMINATED state can be resubmitted; if necessary kill the task first.

remove(jobid=None)

Usage: remove JOBID

Unmanage a task and remove it from the session.

WARNING: All traces of the task are removed and it will not be possible to load or manage it again.

show(jobid=None, *attrs)

Usage: show JOBID [attributes]

Same output as ginfo -v JOBID [-p attributes]

stats(*opts)

Usage: stats [json|text|yaml]

Print how many jobs are in any given state.

One of the words json, yaml, or text (human-readable plain text table) can be used to choose the output format, with text being the default.

unmanage(jobid=None)

Usage: unmanage JOBID

Tell daemon to stop actively managing a task.

The task will keep its state until the daemon is told to manage it again. In particular, tasks that are in RUNNING state keep running and may complete even while unmanaged.

class Server(parent, commands=None, addr='localhost', port=0, portfile=None)

hello()

Print server URL.

Probably only useful for checking if the server is up and responsive.

start()

Start serving requests.

Calls into this method never return, so it should be run in a separate thread.

stop()

Shut down the XML-RPC server and remove the URL file.

created(inbox, subject)

React to creation of subject in inbox.

A typical scenario is this: a new file is created in a watched directory; this method could then react by creating a new task to process that file.

This method should be overridden in derived classes, as the default implementation does nothing.

deleted(inbox, subject)

React to removal of subject from inbox.

This method should be overridden in derived classes, as the default implementation does nothing.

help(cmd=None)

Show available commands, or get information about a specific command.

modified(inbox, subject)

React to modification of subject in inbox.

Note

Not all Pollers are capable of generating modified events reliably. This method is provided for completeness, but likely only useful for filesystem-watching inboxes.

This method should be overridden in derived classes, as the default implementation does nothing.

parse_args()

Do any parsing of the command-line arguments before the main loop starts. This is the place to check validity of the parameters passed as command-line arguments, and to perform setup of shared data structures and default values.

The default implementation does nothing; you are free to override this method in derived classes.

setup()

Setup standard command-line parsing.

GC3Libs scripts should probably override setup_args() to modify command-line parsing.

setup_args()

Set up command-line argument parsing.

The default command line parsing considers every argument as an (input) path name; processing of the given path names is done in parse_args()

setup_options()

Override this method to add command-line options.

shutdown()

Terminate daemon.

terminate(exc_type=None, exc_value=None, tb=None)

Called to stop the script from running.

By default this does nothing; override in derived classes.

class gc3libs.cmdline.SessionBasedScript(**extra_args)

Base class for grosetta/ggamess/gcodeml and like scripts. Implements a long-running script to submit and manage a large number of jobs grouped into a “session”.

The generic scripts implements a command-line like the following:

PROG [options] INPUT [INPUT ...]

First, the script builds a list of input files by recursively scanning each of the given INPUT arguments for files matching the self.input_file_pattern glob string (you can set it via a keyword argument to the ctor). To perform a different treatment of the command-line arguments, override the process_args() method.

Then, new jobs are added to the session, based on the results of the process_args() method above. For each tuple of items returned by process_args(), an instance of class self.application (which you can set by a keyword argument to the ctor) is created, passing it the tuple as init args, and added to the session.

The script finally proceeds to updating the status of all jobs in the session, submitting new ones and retrieving output as needed. When all jobs are done, the method done() is called, and its return value is used as the script’s exit code.

The script’s exitcode tracks job status, in the following way. The exitcode is a bitfield; only the 4 least-significant bits are used, with the following meaning:

Bit	Meaning
0	Set if a fatal error occurred: the script could not complete
1	Set if there are jobs in FAILED state
2	Set if there are jobs in RUNNING or SUBMITTED state
3	Set if there are jobs in NEW state

This boils down to the following rules:

• exitcode == 0: all jobs terminated successfully, no further action
• exitcode == 1: an error interrupted script execution
• exitcode == 2: all jobs terminated, not all of them successfully
• exitcode > 3: run the script again to progress jobs

new_tasks(extra)

Iterate over jobs that should be added to the current session. Each item yielded must be a valid Task instance.

This method is called by the default process_args(), passing self.extra as the extra parameter.

The default implementation of this method scans the arguments on the command-line for files matching the glob pattern self.input_filename_pattern, and for each matching file returns a job name formed by the base name of the file (sans extension), the class given by self.application, and the full path to the input file as sole argument.

If self.instances_per_file and self.instances_per_job are set to a value other than 1, for each matching file N jobs are generated, where N is the quotient of self.instances_per_file by self.instances_per_job.

See also: process_args()

pre_run()

Perform parsing of standard command-line options and call into parse_args() to do non-optional argument processing.

print_summary_table(output, stats)

Print a text summary of the session status to output. This is used to provide the “normal” output of the script; when the -l option is given, the output of the print_tasks_table function is appended.

Override this in subclasses to customize the report that you provide to users. By default, this prints a table with the count of tasks for each possible state.

The output argument is a file-like object, only the write method of which is used. The stats argument is a dictionary, mapping each possible Run.State to the count of tasks in that state; see Engine.counts for a detailed description.

print_tasks_table(output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>, states=Enum({'NEW', 'UNKNOWN', 'STOPPED', 'TERMINATING', 'TERMINATED', 'RUNNING', 'SUBMITTED'}), only=<class 'object'>)

Output a text table to stream output, giving details about tasks in the given states.

Optional second argument states restricts the listing to tasks that are in one of the specified states. By default, all task states are allowed. The states argument should be a list or a set of Run.State values.

Optional third argument only further restricts the listing to tasks that are instances of a subclass of only. By default, there is no restriction and all tasks are listed. The only argument can be a Python class or a tuple – anything infact, that you can pass as second argument to the isinstance operator.

Parameters:

• output – An output stream (file-like object)
• states – List of states (Run.State items) to consider.
• only – Root class (or tuple of root classes) of tasks to consider.

setup()

Setup standard command-line parsing.

GC3Libs scripts should probably override setup_args() to modify command-line parsing.

setup_args()

Set up command-line argument parsing.

The default command line parsing considers every argument as an (input) path name; processing of the given path names is done in parse_args()

gc3libs.cmdline.nonnegative_int(num)

Raise ArgumentTypeError if num is a negative integer (<0), and return int(num) otherwise. num can be any object which can be converted to an int.

>>> nonnegative_int('1')
1
>>> nonnegative_int(1)
1
>>> try:
...   nonnegative_int('-1')
... except argparse.ArgumentTypeError as err:
...   print(err)
'-1' is not a non-negative integer number.
>>> try:
...   nonnegative_int(-1)
... except argparse.ArgumentTypeError as err:
...   print(err)
'-1' is not a non-negative integer number.

Please note that 0 and ‘-0’ are ok:

>>> nonnegative_int(0)
0
>>> nonnegative_int(-0)
0
>>> nonnegative_int('0')
0
>>> nonnegative_int('-0')
0

Floats are ok too:

>>> nonnegative_int(3.14)
3
>>> nonnegative_int(0.1)
0

>>> try:
...   nonnegative_int('ThisWillRaiseAnException')
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'ThisWillRaiseAnException' is not a non-negative ...

gc3libs.cmdline.positive_int(num)

Raise ArgumentTypeError if num is not a strictly positive integer (>0) and return int(num) otherwise. num can be any object which can be converted to an int.

>>> positive_int('1')
1
>>> positive_int(1)
1
>>> try:
...   positive_int('-1')
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'-1' is not a positive integer number.
>>> try:
...   positive_int(-1)
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'-1' is not a positive integer number.
>>> try:
...   positive_int(0)
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'0' is not a positive integer number.

Floats are ok too:

>>> positive_int(3.14)
3

but please take care that float greater than 0 but still less than 1 will fail:

>>> try:
...    positive_int(0.1)
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'0.1' is not a positive integer number.

Also note that 0 is not OK:

>>> try:
...   positive_int(-0)
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'0' is not a positive integer number.
>>> try:
...   positive_int('0')
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'0' is not a positive integer number.
>>> try:
...   positive_int('-0')
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'-0' is not a positive integer number.

Any string which does cannot be converted to an integer will fail:

>>> try:
...   positive_int('ThisWillRaiseAnException')
... except argparse.ArgumentTypeError as err:
...   print(err) # doctest:+ELLIPSIS
'ThisWillRaiseAnException' is not a positive integer ...