gc3libs.backends.shellcmd

Run applications as local processes.

class gc3libs.backends.shellcmd.ShellcmdLrms(name, architecture, max_cores, max_cores_per_job, max_memory_per_core, max_walltime, auth=None, frontend='localhost', transport='local', time_cmd=None, override='False', spooldir=None, resourcedir=None, ssh_config=None, keyfile=None, ignore_ssh_host_keys=False, ssh_timeout=None, **extra_args)

Execute an Application instance as a local process.

Construction of an instance of ShellcmdLrms takes the following optional parameters (in addition to any parameters taken by the base class LRMS):

Parameters:
  • time_cmd (str) –

    Path to the GNU time command. Default is /usr/bin/time which is correct on all known Linux distributions.

    This backend uses many of the extended features of GNU time, so the shell-builtins or the BSD time will not work.

  • spooldir (str) – Path to a filesystem location where to create temporary working directories for processes executed through this backend. The default value None means to use $TMPDIR or /var/tmp (see tempfile.mkftemp for details).
  • resourcedir (str) – Path to a filesystem location where to create a temporary directory that will contain information on the jobs running on the machine. The default value None means to use $HOME/.gc3/shellcmd.d.
  • transport (str) – Transport to use to connecet to the resource. Valid values are ssh or local.
  • frontend (str) – If transport is ssh, then frontend is the hostname of the remote machine where the jobs will be executed.
  • ignore_ssh_host_key (bool) – When connecting to a remote resource using ssh the server ssh public key is usually checked against a database of known hosts, and if the key is found but it does not match with the one saved in the database the connection will fail. Setting ignore_ssh_host_key to True will disable this check, thus introducing a potential security issue, but allowing connection even though the database contain old/invalid keys (the use case is when connecting to VM on a cloud, since the IP is usually reused and therefore the ssh key is recreated).
  • override (bool) – ShellcmdLrms by default will try to gather information on the machine the resource is running on, including the number of cores and the available memory. These values may be different from the values stored in the configuration file. If override is True, then the values automatically discovered will be used instead of the ones in the configuration file. If override is False, instead, the values in the configuration file will be used.
  • ssh_timeout (int) – If transport is ssh, this value will be used as timeout (in seconds) for the TCP connect.
cancel_job(app)

Cancel a running job. If app is associated to a queued or running remote job, tell the execution middleware to cancel it.

close()

Implement gracefully close on LRMS dependent resources e.g. transport

free(app)

Delete the temporary directory where a child process has run. The temporary directory is removed with all its content, recursively.

If the deletion is successful, the lrms_execdir attribute in app.execution is reset to None; subsequent invocations of this method on the same applications do nothing.

free_slots

Returns the number of cores free

get_resource_status()

Update the status of the resource associated with this LRMS instance in-place. Return updated Resource object.

get_results(app, download_dir, overwrite=False, changed_only=True)

Retrieve job output files into local directory download_dir.

Directory download_dir must already exists.

If optional 3rd argument overwrite is False (default), then existing files within download_dir (or subdirectories thereof) will not be altered in any way.

If overwrite is instead True, then the (optional) 4th argument changed_only determines what files are overwritten:

  • if changed_only is True (default), then only files for which the source has a different size or has been modified more recently than the destination are copied;
  • if changed_only is False, then all files in source will be copied into destination, unconditionally.

Output files that do not exist in download_dir will be copied, independently of the overwrite and changed_only settings.

Parameters:
  • job (Task) – the Task instance whose output should be retrieved
  • download_dir (str) – path to download files into
  • overwrite (bool) – if False, do not download files that already exist
  • changed_only (bool) – if both this and overwrite are True, only overwrite those files such that the source is newer or different in size than the destination.
peek(app, remote_filename, local_file, offset=0, size=None)

Download size bytes (at offset offset from the start) from remote file remote_filename and write them into local_file. If size is None (default), then snarf contents of remote file from offset unto the end.

First argument remote_filename is the path to a file relative to the remote job “sandbox”.

Argument local_file is either a local path name (string), or a file-like object supporting a .write() method. If local_file is a path name, it is created if not existent, otherwise overwritten. In any case, upon exit from this procedure, the stream will be positioned just after the written bytes.

Fourth optional argument offset is the offset from the start of the file. If offset is negative, it is interpreted as an offset from the end of the remote file.

Any exception raised by operations will be re-raised to the caller.

submit_job(app)

Run an Application instance as a local process.

See:LRMS.submit_job
update_job_state(app)

Query the running status of the local process whose PID is stored into app.execution.lrms_jobid, and map the POSIX process status to GC3Libs Run.State.

validate_data(data_file_list=[])

Return False if any of the URLs in data_file_list cannot be handled by this backend.

The shellcmd backend can only handle file URLs.