Welcome to GC3Pie documentation!¶

Computational job lifecycle¶

A computational job (for short: job) is a single run of a non-interactive application. The prototypical example is a run of GAMESS on a single input file.

The GC3Utils commands support the following workflow:

1. Submit a GAMESS job (with a single input file): ggamess
2. Monitor the status of the submitted job: gstat
3. Retrieve the output of a job once it’s finished: gget

Usage and some examples on how to use the mentioned commands are provided in the next sections

Managing jobs with GC3Libs¶

GC3Libs takes an application-oriented approach to asynchronous computing. A generic Application class provides the basic operations for controlling remote computations and fetching a result; client code should derive specialized sub-classes to deal with a particular application, and to perform any application-specific pre- and post-processing.

The generic procedure for performing computations with GC3Libs is the following:

1. Client code creates an instance of an Application sub-class.
2. Asynchronous computation is started by submitting the application object; this associates the application with an actual (possibly remote) computational job.
3. Client code can monitor the state of the computational job; state handlers are called on the application object as the state changes.
4. When the job is done, the final output is retrieved and a post-processing method is invoked on the application object.

At this point, results of the computation are available and can be used by the calling program.

The Application class (and its sub-classes) alow client code to control the above process by:

1. Specifying the characteristics (computer program to run, input/output files, memory/CPU/duration requirements, etc.) of the corresponding computational job. This is done by passing suitable values to the Application constructor. See the Application constructor documentation for a detailed description of the parameters.

2. Providing methods to control the “life-cycle” of the associated computational job: start, check execution state, stop, retrieve a snapshot of the output files. There are actually two different interfaces for this, detailed below:

   1. A passive interface: a Core or a Engine object is used to start/stop/monitor jobs associated with the given application. For instance:

      a = GamessApplication(...)
      
      # create a `Core` object; only one instance is needed
      g = Core(...)
      
      # start the remote computation
      g.submit(a)
      
      # periodically monitor job execution
      g.update_job_state(a)
      
      # retrieve output when the job is done
      g.fetch_output(a)
      

      The passive interface gives client code full control over the lifecycle of the job, but cannot support some use cases (e.g., automatic application re-start).

      As you can see from the above example, the passive interface is implemented by methods in the Core and Engine classes (they implement the same interface). See those classes documentation for more details.

   2. An active interface: this requires that the Application object be attached to a Core or Engine instance:

      a = GamessApplication(...)
      
      # create a `Core` object; only one instance is needed
      g = Core(...)
      
      # tell application to use the active interface
      a.attach(g)
      
      # start the remote computation
      a.submit()
      
      # periodically monitor job execution
      a.update_job_state()
      
      # retrieve output when the job is done
      a.fetch_output()
      

      With the active interface, application objects can support automated restart and similar use-cases.

      When an Engine object is used instead of a Core one, the job life-cycle is automatically managed, providing a fully asynchronous way of executing computations.

      The active interface is implemented by the Task class and all its descendants (including Application).

3. Providing “state transition methods” that are called when a change in the job execution state is detected; those methods can implement application specific behavior, like restarting the computational job with changed input if the alloted duration has expired but the computation has not finished. In particular, a postprocess method is called when the final output of an application is available locally for processing.

   The set of “state transition methods” currently implemented by the Application class are: new(), submitted(), running(), stopped(), terminated() and postprocess(). Each method is called when the execution state of an application object changes to the corresponding state; see each method’s documentation for exact information.

In addition, GC3Libs provides collection classes, that expose interfaces 2. and 3. above, allowing one to control a set of applications as a single whole. Collections can be nested (i.e., a collection can hold a mix of Application and TaskCollection objects), so that workflows can be implemented by composing collection objects.

Note that the term computational job (or just job, for short) is used here in a quite general sense, to mean any kind of computation that can happen independently of the main thread of the calling program. GC3Libs currently provide means to execute a job as a separate process on the same computer, or as a batch job on a remote computational cluster.

Execution model of GC3Libs applications¶

An Application can be regarded as an abstraction of an independent asynchronous computation, i.e., a GC3Libs’ Application behaves much like an independent UNIX process (but it can actually run on a separate remote computer). Indeed, GC3Libs’ Application objects mimic the POSIX process model: Application are started by a parent process, run independently of it, and need to have their final exit code and output reaped by the calling process.

The following table makes the correspondence between POSIX processes and GC3Libs’ Application objects explicit.

Slides on specific topics¶

This is a list of tutorials made for the GC3Pie 2012 Training event that has been held in Zurich on 1st and 2nd of October 2012. The slides and tutorials are an introduction to the GC3Pie python package.

Introduction ot GC3Pie

Introduction to the software.

Basic GC3Pie programming

The Application class, the smallest building block of GC3Pie. Introduction to the concept of Job, states of an application and to the Core class.

Application requirements

How to define extra requirements for an application, such as the minimum amount of memory it will use, the number of cores needed or the architecture of the CPUs.

Managint applications: the SessionBasedScript class

Introduction to the higest level interface to build applications with GC3Pie, the SessionBasedScript. Info on how to create simple scripts which will take care of the execution of your applications, from submission to getting back the results.

GC3Pie utils

Low level tools to debug your scripts.

Introduction to Workflows with GC3Pie

Using a practical example (the Warholize Tutorial) we show how workflows are implemented with GC3Pie. The following slides will cover in more details the single steps needed to produce a complex workflow.

ParallelTaskCollection

Description of the ParallelTaskCollection class, used to run tasks in parallel.

StagedTaskCollection

Description of the StagedTaskCollection class, used to run a sequence of a fixed number of jobs.

SequentialTaskCollection

Description of the SequentialTaskCollection class, used to run a sequence of jobs that can be altered during runtime.

Example scripts¶

gdemo_simple.py

Simple script that only uses Application and Core classes to create an application, submit it, check its status and retrieve its output.

gdemo_simple_engine.py

Similar to the previous script, this one uses, instead of Core, a more high-level class to manage jobs workflow: the Engine class.

warholize.py

an enhanched version of the warholize script proposed in the Warholize Tutorial

Warholize Tutorial¶

In this tutorial we will show you how to use GC3Pie libraries in order to build a command line script which will run a complex workflow with both parallel and sequential tasks.

The tutorial itself contains the complete source code of the application (cfr. Literate Programming on Wikipedia), so that you will be able to test/modify it and produce a working warholize.py script by downloading pylit from the PyLit Homepage and running the following command on the gc3pie/docs/gc3libs/tutorial/warholize.txt file, from whitin the svn tree of GC3Pie:

$    ./pylit warholize.txt warholize.py

Introduction¶

Warholize is a GC3Pie demo application to produce, from a generic image picture, a new picture like the famous Warhol’s work: Marylin. The script uses the powerful ImageMagick set of tools (at least version 6.3.5-7). This tutorial will assume that both ImageMagick and GC3Pie are already installed and configured.

In order to produce a similar image we have to do a series of transformations on the picture:

1. convert the original image to grayscale.

2. colorize the grayscale image using three different colors each time, based on the gray levels. We may, for instance, make all pixels with luminosity between 0-33% in red, pixels between 34-66% in yellow and pixels between 67% and 100% in green.

   To do that, we first have to:

   1. create a Color Lookup Table (LUT) using a combination of three randomly chosen colors
   2. apply the LUT to the grayscale image

3. Finally, we can merge together all the colorized images and produce our warholized image.

Clearly, step 2) depends on the step 1), and 3) depends on 2), so we basically have a sequence of tasks, but since step 2) need to create N different independent images, we can parallelize this step.

Workflow of the warholize script

From top to bottom¶

We will write our script starting from the top and will descend to the bottom, from the command line script, to the workflow and finally to the single execution units which compose the application.

The script¶

The SessionBasedScript class in the gc3libs.cmdline module is used to create a generic script. It already have all what is needed to read gc3pie configuration files, manage resources, schedule jobs etc. The only missing thing is, well, your application!

Let’s start by creating a new empty file and importing some basic modules:

import os
import gc3libs
from gc3libs.cmdline import SessionBasedScript

we then create a class which inherits from SessionBasedScript (in GC3Pie, most of the customizations are done by inheriting from a more generic class and overriding the __init__ method and possibly others):

class WarholizeScript(SessionBasedScript):
    """
    Demo script to create a `Warholized` version of an image.
    """
    version='1.0'

Please note that you must either write a small docstring, or add a description attribute. These values are used when the script is called with options --help or --version, which are automatically added by GC3Pie.

The way we want to use our script is straightforward:

$  warholize.py inputfile [inputfiles ...]

and this will create a directory Warholized.<inputfile> in which there will be a file called warhol_<inputfile> containing the desired warholized image (and a lot of temporary files, at least for now).

But we may want to add some additional options to the script, in order to decide how many colorized pictures the warholized image will be made of, or if we want to resize the image. SessionBasedScript uses the PyCLI module which is, in turn, a wrapper around standard argparse (or optparse for older pythons) module. To customize the script you may define a setup_options method and put in there some calls to SessionBasedScript.add_param(), which is inherited from cli.app.CommandLineApp:

def setup_options(self):
    self.add_param('--copies', default=4, type=int,
                   help="Number of copyes (Default:4). It has to be a perfect square!")

In this example we will accept a --copies option to define how many colorized copies the final picture will be made of. Please refer to the documentation of the PyCLI module for details on the syntax of the add_param method.

The heart of the script is, however, the new_tasks method, which will be called to create the initial tasks of the scripts. In our case it will be something like:

def new_tasks(self, extra):
    gc3libs.log.info("Creating main sequential task")
    for (i, input_file) in enumerate(self.params.args):
        extra_args = extra.copy()
        extra_args['output_dir'] = 'Warholized.%s' % os.path.basename(input_file)
        yield WarholizeWorkflow(input_file,
                                self.params.copies,
                                **extra_args)

new_tasks is used as a generator (but it could return a list as well). Each yielded object is a task. In GC3Pie, a task is either a single application or a complex workflow, and rapresents an execution unit. In our case we create a WarholizeWorkflow task which is the workflow described before.

In our case we yield a different WarholizeWorkflow task for each input file. These tasks will run in parallel.

Please note that we are using the gc3libs.log module to log informations about the execution. This module works like the logging module and has methods like error, warning, info or debug, but its logging level is automatically configured by SessionBasedScript‘s constructor. This way you can increase the verbosity of your script by simply adding -v options from the command line.

The workflows¶

Main sequential workflow¶

The module gc3libs.workflow contains two main objects: SequentialTaskCollection and ParallelTaskCollection. They execute tasks in serial and in parallel, respectively. We will use both of them to create our workflow; the first one, WarholizeWorkflow, is a sequential task, therefore we have to inherit from SequentialTaskCollection and customize its __init__ method:

from gc3libs.workflow import SequentialTaskCollection, ParallelTaskCollection
import math
from gc3libs import Run

class WarholizeWorkflow(SequentialTaskCollection):
    """
    Main workflow.
    """

    def __init__(self, input_image,  copies, **extra_args):
        self.input_image = input_image
        self.output_image = "warhol_%s" % os.path.basename(input_image)

        gc3libs.log.info(
            "Producing a warholized version of input file %s "
            "and store it in %s" % (input_image, self.output_image))

        self.output_dir = os.path.relpath(extra_args.get('output_dir'))

        self.copies = copies

        # Check that copies is a perfect square
        if math.sqrt(self.copies) != int(math.sqrt(self.copies)):
            raise gc3libs.exceptions.InvalidArgument(
                "`copies` argument must be a perfect square.")

        self.jobname = extra_args.get('jobname', 'WarholizedWorkflow')
        self.grayscaled_image = "grayscaled_%s" % os.path.basename(self.input_image)

Up to now we just parsed the arguments. The following lines, instead, create the first task that we want to execute. By now, we can create only the first one, GrayScaleConvertApplication, which will produce a grayscale image from the input file:

self.tasks = [
    GrayScaleConvertApplication(
        self.input_image, self.grayscaled_image, self.output_dir,
        self.output_dir),
    ]

Finally, we call the parent’s constructor.:

SequentialTaskCollection.__init__(
    self, self.tasks)

This will create the initial task list, but we have to run also step 2 and 3, and this is done by creating a next method. This method will be called after all the tasks in self.tasks are finished. We cannot create all the jobs at once because we don’t have all the needed input files yet. Please note that by creating the tasks in the next method you could decide at runtime which tasks to run next and what arguments we may want to give to them.

In our case, however, the next method is quite simple:

def next(self, iteration):
    last = self.tasks[-1]

    if iteration == 0:
        # first time we got called.  We have the grayscaled image,
        # we have to run the Tricolorize task.
        self.add(TricolorizeMultipleImages(
            os.path.join(self.output_dir, self.grayscaled_image),
            self.copies, self.output_dir))
        return Run.State.RUNNING
    elif iteration == 1:
        # second time, we already have the colorized images, we
        # have to merge them together.
        self.add(MergeImagesApplication(
            os.path.join(self.output_dir, self.grayscaled_image),
            last.warhol_dir,
            self.output_image))
        return Run.State.RUNNING
    else:
        self.execution.returncode = last.execution.returncode
        return Run.State.TERMINATED

At each iteration, we call self.add() to add an instance of a task-like class (gc3libs.Application, gc3libs.workflow.ParallelTaskCollection or gc3libs.workflow.SequentialTaskCollection, in our case) to complete the next step, and we return the current state, which will be gc3libs.Run.State.RUNNING unless we have finished the computation.

Step one: convert to grayscale¶

GrayScaleConvertApplication is the application responsible to convert to grayscale the input image. The command we want to execute is:

$    convert -colorspace gray <input_image> grayscaled_<input_image>

To create a generic application we create a class which inherit from gc3libs.Application and we usually only need to customize the __init__ method:

# An useful function to copy files
from gc3libs.utils import copyfile

class GrayScaleConvertApplication(gc3libs.Application):
    def __init__(self, input_image, grayscaled_image, output_dir, warhol_dir):
        self.warhol_dir = warhol_dir
        self.grayscaled_image = grayscaled_image

        arguments = [
            'convert',
            os.path.basename(input_image),
            '-colorspace',
            'gray',
            ]

        gc3libs.log.info(
            "Craeting  GrayScale convert application from file %s"
            "to file %s" % (input_image, grayscaled_image))

        gc3libs.Application.__init__(
            self,
            arguments = arguments + [grayscaled_image],
            inputs = [input_image],
            outputs = [grayscaled_image, 'stderr.txt', 'stdout.txt'],
            output_dir = output_dir,
            stdout = 'stdout.txt',
            stderr = 'stderr.txt',
            )

Creating a gc3libs.Application is straigthforward: you just call the constructor with the executable, the arguments, and the input/output files you will need.

If you don’t specify the output_dir directory, gc3pie libraries will create one starting from the class name. If the output directory exists already, the old one will be renamed.

To do any kind of post processing you can define a terminate method for your application. It will be called after your application will terminate. In our case we want to copy the gray scale version of the image to the warhol_dir, so that it will be easily reachable by all other applications:

def terminated(self):
    """Move grayscale image to the main output dir"""
    copyfile(
        os.path.join(self.output_dir, self.grayscaled_image),
        self.warhol_dir)

Step two: parallel workflow to create colorized images¶

The TricolorizeMultipleImages is responsible to create multiple versions of the grayscale image with different coloration chosen randomly from a list of available colors. It does it by running multiple instance of TricolorizeImage with different arguments. Since we want to run the various colorization in parallel, it inherits from gc3libs.workflow.ParallelTaskCollection class. Like we did for GrayScaleConvertApplication, we only need to customize the constructor __init__, creating the various subtasks we want to run:

import itertools
import random

class TricolorizeMultipleImages(ParallelTaskCollection):
    colors = ['yellow', 'blue', 'red', 'pink', 'orchid',
              'indigo', 'navy', 'turquoise1', 'SeaGreen', 'gold',
              'orange', 'magenta']

    def __init__(self, grayscaled_image, copies, output_dir):
        gc3libs.log.info(
            "TricolorizeMultipleImages for %d copies run" % copies)
        self.jobname = "Warholizer_Parallel"
        ncolors = 3
        ### XXX Why I have to use basename???
        self.output_dir = os.path.join(
            os.path.basename(output_dir), 'tricolorize')
        self.warhol_dir = output_dir

        # Compute a unique sequence of random combination of
        # colors. Please note that we can have a maximum of N!/3! if N
        # is len(colors)
        assert copies <= math.factorial(len(self.colors)) / math.factorial(ncolors)

        combinations = [i for i in itertools.combinations(self.colors, ncolors)]
        combinations = random.sample(combinations, copies)

        # Create all the single tasks
        self.tasks = []
        for i, colors in enumerate(combinations):
            self.tasks.append(TricolorizeImage(
                os.path.relpath(grayscaled_image),
                "%s.%d" % (self.output_dir, i),
                "%s.%d" % (grayscaled_image, i),
                colors,
                self.warhol_dir))

        ParallelTaskCollection.__init__(self, self.tasks)

The main loop will fill the self.tasks list with various TricolorizedImage tasks, each one with an unique combination of three colors to use to generate the colorized image. The GC3Pie framework will then run these tasks in parallel, on any available resource.

The TricolorizedImage class is indeed a SequentialTaskCollection, since it has to generate the LUT first, and then apply it to the grayscale image. We already saw how to create a SequentialTaskCollection: we modify the constructor in order to add the first job (CreateLutApplication), and the next method will take care of running the ApplyLutApplication application on the output of the first job:

class TricolorizeImage(SequentialTaskCollection):
    """
    Sequential workflow to produce a `tricolorized` version of a
    grayscale image
    """
    def __init__(self, grayscaled_image, output_dir, output_file,
                 colors, warhol_dir):
        self.grayscaled_image = grayscaled_image
        self.output_dir = output_dir
        self.warhol_dir = warhol_dir
        self.jobname = 'TricolorizeImage'
        self.output_file = output_file

        if not os.path.isdir(output_dir):
            os.mkdir(output_dir)

        gc3libs.log.info(
            "Tricolorize image %s to %s" % (
                self.grayscaled_image, self.output_file))

        self.tasks = [
            CreateLutApplication(
                self.grayscaled_image,
                "%s.miff" % self.grayscaled_image,
                self.output_dir,
                colors, self.warhol_dir),
            ]

        SequentialTaskCollection.__init__(self, self.tasks)

    def next(self, iteration):
        last = self.tasks[-1]
        if iteration == 0:
            # First time we got called. The LUT has been created, we
            # have to apply it to the grayscale image
            self.add(ApplyLutApplication(
                self.grayscaled_image,
                os.path.join(last.output_dir, last.lutfile),
                os.path.basename(self.output_file),
                self.output_dir, self.warhol_dir))
            return Run.State.RUNNING
        else:
            self.execution.returncode = last.execution.returncode
            return Run.State.TERMINATED

The CreateLutApplication is again an application which inherits from gc3libs.Application. The command we want to execute is something like:

$    convert -size 1x1 xc:<color1> xc:<color2> xc:<color3> +append -resize 256x1! <output_file.miff>

This will basically create an image 256x1 pixels big, made of a gradient using all the listed colors. The code will look like:

class CreateLutApplication(gc3libs.Application):
    """Create the LUT for the image using 3 colors picked randomly
    from CreateLutApplication.colors"""

    def __init__(self, input_image, output_file, output_dir, colors, working_dir):
        self.lutfile = os.path.basename(output_file)
        self.working_dir = working_dir
        gc3libs.log.info("Creating lut file %s from %s using "
                         "colors: %s" % (
            self.lutfile, input_image, str.join(", ", colors)))
        gc3libs.Application.__init__(
            self,
            arguments = [
                'convert',
                '-size',
                '1x1'] + [
                "xc:%s" % color for color in colors] + [
                '+append',
                '-resize',
                '256x1!',
                self.lutfile,
                ],
            inputs = [input_image],
            outputs = [self.lutfile, 'stdout.txt', 'stderr.txt'],
            output_dir = output_dir + '.createlut',
            stdout = 'stdout.txt',
            stderr = 'stderr.txt',
            )

Similarly, the ApplyLutApplication application will run the following command:

$    convert grayscaled_<input_image> <lutfile.N.miff> -clut grayscaled_<input_image>.<N>

This command will apply the LUT to the grayscaled image: it will modify the grayscaled image by coloring a generic pixel with a luminosity value of n (which will be an integer value from 0 to 255, of course) with the color at position n in the LUT image (actually, n+1). Each ApplyLutApplication will save the resulting image to a file named as grayscaled_<input_image>.<N>.

The class will look like:

class ApplyLutApplication(gc3libs.Application):
    """Apply the LUT computed by `CreateLutApplication` to
    `image_file`"""

    def __init__(self, input_image, lutfile, output_file, output_dir, working_dir):

        gc3libs.log.info("Applying lut file %s to %s" % (lutfile, input_image))
        self.working_dir = working_dir
        self.output_file = output_file

        gc3libs.Application.__init__(
            self,
            arguments = [
                'convert',
                os.path.basename(input_image),
                os.path.basename(lutfile),
                '-clut',
                output_file,
                ],
            inputs = [input_image, lutfile],
            outputs = [output_file, 'stdout.txt', 'stderr.txt'],
            output_dir = output_dir + '.applylut',
            stdout = 'stdout.txt',
            stderr = 'stderr.txt',
            )

The terminated method:

def terminated(self):
    """Copy colorized image to the output dir"""
    copyfile(
        os.path.join(self.output_dir, self.output_file),
        self.working_dir)

will copy the colorized image file in the top level directory, so that it will be easier for the last application to find all the needed files.

Step three: merge all them together¶

At this point we will have in the main output directory a bunch of files named after grayscaled_<input_image>.N with N a sequential integer and <input_image> the name of the original image. The last application, MergeImagesApplication, will produce a warhol_<input_image> image by merging all of them using the command:

$    montage grayscaled_<input_image>.* -tile 3x3 -geometry +5+5 -background white warholized_<input_image>

Now it should be easy to write such application:

import re

class MergeImagesApplication(gc3libs.Application):
    def __init__(self, grayscaled_image, input_dir, output_file):
        ifile_regexp = re.compile(
            "%s.[0-9]+" % os.path.basename(grayscaled_image))
        input_files = [
            os.path.join(input_dir, fname) for fname in os.listdir(input_dir)
            if ifile_regexp.match(fname)]
        input_filenames = [os.path.basename(i) for i in input_files]
        gc3libs.log.info("MergeImages initialized")
        self.input_dir = input_dir
        self.output_file = output_file

        tile = math.sqrt(len(input_files))
        if tile != int(tile):
            gc3libs.log.error(
                "We would expect to have a perfect square"
                "of images to merge, but we have %d instead" % len(input_files))
            raise gc3libs.exceptions.InvalidArgument(
                "We would expect to have a perfect square of images to merge, but we have %d instead" % len(input_files))

        gc3libs.Application.__init__(
            self,
            arguments = ['montage'] + input_filenames + [
                '-tile',
                '%dx%d' % (tile, tile),
                '-geometry',
                '+5+5',
                '-background',
                'white',
                output_file,
                ],
            inputs = input_files,
            outputs = [output_file, 'stderr.txt', 'stdout.txt'],
            output_dir = os.path.join(input_dir, 'output'),
            stdout = 'stdout.txt',
            stderr = 'stderr.txt',
            )

Making the script executable¶

Finally, in order to make the script executable, we add the following lines to the end of the file. The WarholizeScritp().run() call will be executed only when the file is run as a script, and will do all the magic related to argument parsing, creating the session etc...:

if __name__ == '__main__':
    import warholize
    warholize.WarholizeScript().run()

Please note that the import warholize statement is important to address issue 95 and make the gc3pie scripts work with your current session (gstat, ginfo...)

Testing¶

To test this script I would suggest to use the famous Lena picture, which can be found in the miscelaneous section of the Signal and Image Processing Institute page. Download the image, rename it as lena.tiff and run the following command:

$    ./warholize.py -C 1 lena.tiff --copies 9

(add -r localhost if your gc3pie.conf script support it and you want to test it locally).

After completion a file Warholized.lena.tiff/output/warhol_lena.tiff will be created.

Warholized version of Lena

gc3libs¶

GC3Libs is a python package for controlling the life-cycle of a Grid or batch computational job.

GC3Libs provides services for submitting computational jobs to Grids and batch systems, controlling their execution, persisting job information, and retrieving the final output.

GC3Libs takes an application-oriented approach to batch computing. A generic Application class provides the basic operations for controlling remote computations, but different Application subclasses can expose adapted interfaces, focusing on the most relevant aspects of the application being represented.

class gc3libs.Application(arguments, inputs, outputs, output_dir, **extra_args)¶

Support for running a generic application with the GC3Libs. The following parameters are required to create an Application instance:

arguments

List or sequence of program arguments. The program to execute is the first one.; any object in the list will be converted to string via Python’s str().

inputs

Files that will be copied to the remote execution node before execution starts.

There are two possible ways of specifying the inputs parameter:

• It can be a Python dictionary: keys are local file paths or URLs, values are remote file names.

• It can be a Python list: each item in the list should be a pair (source, remote_file_name): the source can be a local file or a URL; remote_file_name is the path (relative to the execution directory) where source will be downloaded. If remote_file_name is an absolute path, an InvalidArgument error is raised.

  A single string file_name is allowed instead of the pair and results in the local file file_name being copied to file_name on the remote host.

outputs

Files and directories that will be copied from the remote execution node back to the local computer (or a network-accessible server) after execution has completed. Directories are copied recursively.

There are three possible ways of specifying the outputs parameter:

• It can be a Python dictionary: keys are remote file or directory paths (relative to the execution directory), values are corresponding local names.

• It can be a Python list: each item in the list should be a pair (remote_file_name, destination): the destination can be a local file or a URL; remote_file_name is the path (relative to the execution directory) that will be uploaded to destination. If remote_file_name is an absolute path, an InvalidArgument error is raised.

  A single string file_name is allowed instead of the pair and results in the remote file file_name being copied to file_name on the local host.

• The constant gc3libs.ANY_OUTPUT which instructs GC3Libs to copy every file in the remote execution directory back to the local output path (as specified by the output_dir attribute).

Note that no errors will be raised if an output file is not present. Override the terminated() method to raise errors for reacting on this kind of failures.

output_dir

Path to the base directory where output files will be downloaded. Output file names are interpreted relative to this base directory.

requested_cores,`requested_memory`,`requested_walltime`

specify resource requirements for the application: * the number of independent execution units (CPU cores), * amount of memory (as a gc3libs.quantity.Memory object), * amount of wall-clock time to allocate for the computational job (as a gc3libs.quantity.Duration object).

The following optional parameters may be additionally specified as keyword arguments and will be given special treatment by the Application class logic:

requested_architecture

specify that this application can only be executed on a certain processor architecture; see Run.Arch for a list of possible values. The default value None means that any architecture is valid, i.e., there are no requirements on the processor architecture.

environment

a list of pairs (name, value): the environment variable whose name is given by the contents of the string name will be defined as the content of string value (i.e., as if “export name=value” was executed prior to starting the application). Alternately, one can pass in a list of strings of the form “name=value”.

output_base_url

if not None, this is prefixed to all output files (except stdout and stderr, which are always retrieved), so, for instance, having output_base_url=”gsiftp://example.org/data” will upload output files into that remote directory.

stdin

file name of a file whose contents will be fed as standard input stream to the remote-executing process.

stdout

name of a file where the standard output stream of the remote executing process will be redirected to; will be automatically added to outputs.

stderr

name of a file where the standard error stream of the remote executing process will be redirected to; will be automatically added to outputs.

join

if this evaluates to True, then standard error is redirected to the file specified by stdout and stderr is ignored. (join has no effect if stdout is not given.)

tags

list of tag names (string) that must be present on a resource in order to be eligible for submission; in the ARC backend, tags are interpreted as run-time environments (RTE) to request.

Any other keyword arguments will be set as instance attributes, but otherwise ignored by the Application constructor.

After successful construction, an Application object is guaranteed to have the following instance attributes:

arguments

list of strings specifying command-line arguments for executable invocation. The first element must be the executable.

inputs

dictionary mapping source URL (a gc3libs.url.Url object) to a remote file name (a string); remote file names are relative paths (root directory is the remote job folder)

outputs

dictionary mapping remote file name (a string) to a destination (a gc3libs.url.Url); remote file names are relative paths (root directory is the remote job folder)

output_dir

Path to the base directory where output files will be downloaded. Output file names (those which are not URLs) are interpreted relative to this base directory.

execution

a Run instance; its state attribute is initially set to NEW (Actually inherited from the Task)

environment

dictionary mapping environment variable names to the requested value (string); possibly empty

stdin

None or a string specifying a (local) file name. If stdin is not None, then it matches a key name in inputs

stdout

None or a string specifying a (remote) file name. If stdout is not None, then it matches a key name in outputs

stderr

None or a string specifying a (remote) file name. If stdout is not None, then it matches a key name in outputs

join

boolean value, indicating whether stdout and stderr are collected into the same file

tags

list of strings specifying the tags to request in each resource for submission; possibly empty.

application_name = 'generic'¶

A name for applications of this class.

This string is used as a prefix for configuration items related to this application in configured resources. For example, if the application_name is foo, then the application interface code in GC3Pie might search for foo_cmd, foo_extra_args, etc. See qsub_sge() for an actual example.

bsub(resource, _suppress_warning=False, **extra_args)¶

Get an LSF qsub command-line invocation for submitting an instance of this application. Return a pair (cmd_argv, app_argv), where cmd_argv is a list containing the argv-vector of the command to run to submit an instance of this application to the LSF batch system, and app_argv is the argv-vector to use when invoking the application.

In the construction of the command-line invocation, one should assume that all the input files (as named in Application.inputs) have been copied to the current working directory, and that output files should be created in this same directory.

The default implementation just prefixes any output from the cmdline method with an LSF bsub invocation of the form bsub -cwd . -L /bin/sh + resource limits.

Override this method in application-specific classes to provide appropriate invocation templates and/or add resource-specific submission options.

cmdline(resource)¶

Return list of command-line arguments for invoking the application.

This is exactly the argv-vector of the application process: the application command name is included as first item (index 0) of the list, further items are command-line arguments.

Hence, to get a UNIX shell command-line, just concatenate the elements of the list, separating them with spaces.

compatible_resources(resources)¶

Return a list of compatible resources.

fetch_output(download_dir, overwrite, **extra_args)¶

Calls the corresponding method of the controller.

fetch_output_error(ex)¶

Invocation of Core.fetch_output() on this object failed; ex is the Exception that describes the error.

If this method returns an exception object, that is raised as a result of the Core.fetch_output(), otherwise the return value is ignored and Core.fetch_output returns None.

Default is to return ex unchanged; override in derived classes to change this behavior.

qsub_pbs(resource, _suppress_warning=False, **extra_args)¶

Similar to qsub_sge(), but for the PBS/TORQUE resource manager.

qsub_sge(resource, **extra_args)¶

Get an SGE qsub command-line invocation for submitting an instance of this application.

Return a pair (cmd_argv, app_argv). Both cmd_argv and app_argv are argv-lists: the command name is included as first item (index 0) of the list, further items are command-line arguments; cmd_argv is the argv-list for the submission command (excluding the actual application command part); app_argv is the argv-list for invoking the application. By overriding this method, one can add futher resource-specific options at the end of the cmd_argv argv-list.

In the construction of the command-line invocation, one should assume that all the input files (as named in Application.inputs) have been copied to the current working directory, and that output files should be created in this same directory.

The default implementation just prefixes any output from the cmdline method with an SGE qsub invocation of the form qsub -cwd -S /bin/sh + resource limits. Note that there is no generic way of requesting a certain number of cores in SGE: it all depends on the installed parallel environment, and these are totally under control of the local sysadmin; therefore, any request for cores is ignored and a warning is logged.

Override this method in application-specific classes to provide appropriate invocation templates and/or add different submission options.

rank_resources(resources)¶

Sort the given resources in order of preference.

By default, less-loaded resources come first; see _cmp_resources.

sbatch(resource, **extra_args)¶

Get a SLURM sbatch command-line invocation for submitting an instance of this application.

Return a pair (cmd_argv, app_argv). Both cmd_argv and app_argv are argv-lists: the command name is included as first item (index 0) of the list, further items are command-line arguments; cmd_argv is the argv-list for the submission command (excluding the actual application command part); app_argv is the argv-list for invoking the application. By overriding this method, one can add futher resource-specific options at the end of the cmd_argv argv-list.

In the construction of the command-line invocation, one should assume that all the input files (as named in Application.inputs) have been copied to the current working directory, and that output files should be created in this same directory.

Override this method in application-specific classes to provide appropriate invocation templates and/or add different submission options.

submit_error(exs)¶

Invocation of Core.submit() on this object failed; exs is a list of Exception objects, one for each attempted submission.

If this method returns an exception object, that is raised as a result of the Core.submit(), otherwise the return value is ignored and Core.submit returns None.

Default is to always return the first exception in the list (on the assumption that it is the root of all exceptions or that at least it refers to the preferred resource). Override in derived classes to change this behavior.

update_job_state_error(ex)¶

Handle exceptions that occurred during a Core.update_job_state call.

If this method returns an exception object, that exception is processed in Core.update_job_state() instead of the original one. Any other return value is ignored and Core.update_job_state proceeds as if no exception had happened.

Argument ex is the exception that was raised by the backend during job state update.

Default is to return ex unchanged; override in derived classes to change this behavior.

xrsl(resource)¶

Return a string containing an xRSL sequence, suitable for submitting an instance of this application through ARC’s ngsub command.

The default implementation produces XRSL content based on the construction parameters; you should override this method to produce XRSL tailored to your application.

Warning

WARNING: ARClib SWIG bindings cannot resolve the overloaded constructor if the xRSL string argument is a Python ‘unicode’ object; if you overload this method, force the result to be a ‘str’!

class gc3libs.Default¶

A namespace for all constants and default values used in the GC3Libs package.

ARC_CACHE_TIME = 30¶

only update ARC resources status every this seconds

ARC_LOST_JOB_TIMEOUT = 120¶

consider a submitted job lost if it does not show up in the information system after this duration

PROXY_VALIDITY_THRESHOLD = 600¶

Proxy validity threshold in seconds. If proxy is expiring before the thresold, it will be marked as to be renewed.

class gc3libs.Run(initializer=None, attach=None, **keywd)¶

A specialized dict-like object that keeps information about the execution state of an Application instance.

A Run object is guaranteed to have the following attributes:

log

A gc3libs.utils.History instance, recording human-readable text messages on events in this job’s history.

info

A simplified interface for reading/writing messages to Run.log. Reading from the info attribute returns the last message appended to log. Writing into info appends a message to log.

timestamp

Dictionary, recording the most recent timestamp when a certain state was reached. Timestamps are given as UNIX epochs.

For properties state, signal and returncode, see the respective documentation.

Run objects support attribute lookup by both the [...] and the . syntax; see gc3libs.utils.Struct for examples.

class Arch¶

Processor architectures, for use as values in the requested_architecture field of the Application class constructor.

The following values are currently defined:

X86_64

64-bit Intel/AMD/VIA x86 processors in 64-bit mode.

X86_32

32-bit Intel/AMD/VIA x86 processors in 32-bit mode.

Run.exitcode¶

The “exit code” part of a Run.returncode, see os.WEXITSTATUS. This is an 8-bit integer, whose meaning is entirely application-specific. (However, the value 255 is often used to mean that an error has occurred and the application could not end its execution normally.)

Run.in_state(*names)¶

Return True if the Run state matches any of the given names.

In addition to the states from Run.State, the two additional names ok and failed are also accepted, with the following meaning:

• ok: state is TERMINATED and returncode is 0.
• failed: state is TERMINATED and returncode is non-zero.

Run.info¶

A simplified interface for reading/writing entries into history.

Setting the info attribute appends a message to the log:

>>> j1 = Run()
>>> j1.info = 'a message'
>>> j1.info = 'a second message'

Getting the value of the info attribute returns the last message entered in the log:

>>> j1.info 
u'a second message ...'

Run.returncode¶

The returncode attribute of this job object encodes the Run termination status in a manner compatible with the POSIX termination status as implemented by os.WIFSIGNALED and os.WIFEXITED.

However, in contrast with POSIX usage, the exitcode and the signal part can both be significant: in case a Grid middleware error happened after the application has successfully completed its execution. In other words, os.WEXITSTATUS(returncode) is meaningful iff os.WTERMSIG(returncode) is 0 or one of the pseudo-signals listed in Run.Signals.

Run.exitcode and Run.signal are combined to form the return code 16-bit integer as follows (the convention appears to be obeyed on every known system):

Bit	Encodes...
0..7	signal number
8	1 if program dumped core.
9..16	exit code

Note: the “core dump bit” is always 0 here.

Setting the returncode property sets exitcode and signal; you can either assign a (signal, exitcode) pair to returncode, or set returncode to an integer from which the correct exitcode and signal attribute values are extracted:

>>> j = Run()
>>> j.returncode = (42, 56)
>>> j.signal
42
>>> j.exitcode
56

>>> j.returncode = 137
>>> j.signal
9
>>> j.exitcode
0

See also Run.exitcode and Run.signal.

static Run.shellexit_to_returncode(rc)¶

Convert a shell exit code to a POSIX process return code.

A POSIX shell represents the return code of the last-run program within its exit code as follows:

• If the program was terminated by signal N, the shell exits with code 128+N,
• otherwise, if the program terminated with exit code C, the shell exits with code C.

Run.signal¶

The “signal number” part of a Run.returncode, see os.WTERMSIG for details.

The “signal number” is a 7-bit integer value in the range 0..127; value 0 is used to mean that no signal has been received during the application runtime (i.e., the application terminated by calling exit()).

The value represents either a real UNIX system signal, or a “fake” one that GC3Libs uses to represent Grid middleware errors (see Run.Signals).

Run.state¶

The state a Run is in.

The value of Run.state must always be a value from the Run.State enumeration, i.e., one of the following values.

Run.State value	purpose	can change to
NEW	Job has not yet been submitted/started (i.e., gsub not called)	SUBMITTED (by gsub)
SUBMITTED	Job has been sent to execution resource	RUNNING, STOPPED
STOPPED	Trap state: job needs manual intervention (either user- or sysadmin-level) to resume normal execution	TERMINATING(by gkill), SUBMITTED (by miracle)
RUNNING	Job is executing on remote resource	TERMINATING
TERMINATING	Job has finished execution on remote resource; output not yet retrieved	TERMINATED
TERMINATED	Job execution is finished (correctly or not) and will not be resumed; output has been retrieved	None: final state

When a Run object is first created, it is assigned the state NEW. After a successful invocation of Core.submit(), it is transitioned to state SUBMITTED. Further transitions to RUNNING or STOPPED or TERMINATED state, happen completely independently of the creator progra; the Core.update_job_state() call provides updates on the status of a job.

The STOPPED state is a kind of generic “run time error” state: a job can get into the STOPPED state if its execution is stopped (e.g., a SIGSTOP is sent to the remote process) or delayed indefinitely (e.g., the remote batch system puts the job “on hold”). There is no way a job can get out of the STOPPED state automatically: all transitions from the STOPPED state require manual intervention, either by the submitting user (e.g., cancel the job), or by the remote systems administrator (e.g., by releasing the hold).

The TERMINATED state is the final state of a job: once a job reaches it, it cannot get back to any other state. Jobs reach TERMINATED state regardless of their exit code, or even if a system failure occurred during remote execution; actually, jobs can reach the TERMINATED status even if they didn’t run at all, for example, in case of a fatal failure during the submission step.

class gc3libs.Task(**extra_args)¶

Mix-in class implementing a facade for job control.

A Task can be described as an “active” job, in the sense that all job control is done through methods on the Task instance itself; contrast this with operating on Application objects through a Core or Engine instance.

The following pseudo-code is an example of the usage of the Task interface for controlling a job. Assume that GamessApplication is inheriting from Task (it actually is):

t = GamessApplication(input_file)
t.submit()
# ... do other stuff
t.update_state()
# ... take decisions based on t.execution.state
t.wait() # blocks until task is terminated

Each Task object has an execution attribute: it is an instance of class Run, initialized with a new instance of Run, and at any given time it reflects the current status of the associated remote job. In particular, execution.state can be checked for the current task status.

After successful initialization, a Task instance will have the following attributes:

changed

evaluates to True if the Task has been changed since last time it has been saved to persistent storage (see gclibs.persistence)

execution

a Run instance; its state attribute is initially set to NEW.

attach(controller)¶

Use the given Grid interface for operations on the job associated with this task.

detach()¶

Remove any reference to the current grid interface. After this, calling any method other than attach() results in an exception TaskDetachedFromGridError being thrown.

fetch_output(output_dir=None, overwrite=False, **extra_args)¶

Retrieve the outputs of the computational job associated with this task into directory output_dir, or, if that is None, into the directory whose path is stored in instance attribute .output_dir.

If the execution state is TERMINATING, transition the state to TERMINATED (which runs the appropriate hook).

See gc3libs.Core.fetch_output() for a full explanation.

Returns:	Path to the directory where the job output has been collected.

free(**extra_args)¶

Release any remote resources associated with this task.

See gc3libs.Core.free() for a full explanation.

kill(**extra_args)¶

Terminate the computational job associated with this task.

See gc3libs.Core.kill() for a full explanation.

new()¶

Called when the job state is (re)set to NEW.

Note this will not be called when the application object is created, rather if the state is reset to NEW after it has already been submitted.

The default implementation does nothing, override in derived classes to implement additional behavior.

peek(what='stdout', offset=0, size=None, **extra_args)¶

Download size bytes (at offset offset from the start) from the associated job standard output or error stream, and write them into a local file. Return a file-like object from which the downloaded contents can be read.

See gc3libs.Core.peek() for a full explanation.

progress()¶

Advance the associated job through all states of a regular lifecycle. In detail:

1. If execution.state is NEW, the associated job is started.
2. The state is updated until it reaches TERMINATED
3. Output is collected and the final returncode is returned.

An exception TaskError is raised if the job hits state STOPPED or UNKNOWN during an update in phase 2.

When the job reaches TERMINATING state, the output is retrieved; if this operation is successfull, state is advanced to TERMINATED.

oNCE the job reaches TERMINATED state, the return code (stored also in .returncode) is returned; if the job is not yet in TERMINATED state, calling progress returns None.

Raises:	exception UnexpectedStateError if the associated job goes into state STOPPED or UNKNOWN
Returns:	final returncode, or None if the execution state is not TERMINATED.

running()¶

Called when the job state transitions to RUNNING, i.e., the job has been successfully started on a (possibly) remote resource.

The default implementation does nothing, override in derived classes to implement additional behavior.

stopped()¶

Called when the job state transitions to STOPPED, i.e., the job has been remotely suspended for an unknown reason and cannot automatically resume execution.

The default implementation does nothing, override in derived classes to implement additional behavior.

submit(resubmit=False, **extra_args)¶

Start the computational job associated with this Task instance.

submitted()¶

Called when the job state transitions to SUBMITTED, i.e., the job has been successfully sent to a (possibly) remote execution resource and is now waiting to be scheduled.

The default implementation does nothing, override in derived classes to implement additional behavior.

terminated()¶

Called when the job state transitions to TERMINATED, i.e., the job has finished execution (with whatever exit status, see returncode) and the final output has been retrieved.

The location where the final output has been stored is available in attribute self.output_dir.

The default implementation does nothing, override in derived classes to implement additional behavior.

terminating()¶

Called when the job state transitions to TERMINATING, i.e., the remote job has finished execution (with whatever exit status, see returncode) but output has not yet been retrieved.

The default implementation does nothing, override in derived classes to implement additional behavior.

unknown()¶

Called when the job state transitions to UNKNOWN, i.e., the job has not been updated for a certain period of time thus it is placed in UNKNOWN state.

Two possible ways of changing from this state: 1) next update cycle, job status is updated from the remote server 2) derive this method for Application specific logic to deal with this case

The default implementation does nothing, override in derived classes to implement additional behavior.

update_state(**extra_args)¶

In-place update of the execution state of the computational job associated with this Task. After successful completion, .execution.state will contain the new state.

After the job has reached the TERMINATING state, the following attributes are also set:

execution.duration

Time lapse from start to end of the job at the remote execution site, as a gc3libs.quantity.Duration value. (This is also often referred to as the ‘wall-clock time’ or walltime of the job.)

execution.max_used_memory

Maximum amount of RAM used during job execution, represented as a gc3libs.quantity.Memory value.

execution.used_cpu_time

Total time (as a gc3libs.quantity.Duration value) that the processors has been actively executing the job’s code.

The execution backend may set additional attributes; the exact name and format of these additional attributes is backend-specific. However, you can easily identify the backend-specific attributes because their name is prefixed with the (lowercased) backend name; for instance, the PbsLrms backend sets attributes pbs_queue, pbs_end_time, etc.

wait(interval=60)¶

Block until the associated job has reached TERMINATED state, then return the job’s return code. Note that this does not automatically fetch the output.

Parameters:	interval (integer) – Poll job state every this number of seconds

gc3libs.configure_logger(level=40, name=None, format='sphinx-build: [%(asctime)s] %(levelname)-8s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S')¶

Configure the gc3.gc3libs logger.

Arguments level, format and datefmt set the corresponding arguments in the logging.basicConfig() call.

If a user configuration file exists in file NAME.log.conf in the Default.RCDIR directory (usually ~/.gc3), it is read and used for more advanced configuration; if it does not exist, then a sample one is created.

gc3libs.cmdline¶

gc3libs.config¶

Deal with GC3Pie configuration files.

class gc3libs.config.Configuration(*locations, **extra_args)¶

In-memory representation of the GC3Pie configuration.

This class provides facilities for:

• parsing configuration files (methods load() and merge_file());
• validating the loaded values;
• instanciating the internal GC3Pie objects resulting from the configuration (methods make_auth() and make_resource()).

The constructor takes a list of files to load (locations) and a list of key=value pairs to provide defaults for the configuration. Both lists are optional and can be omitted, resulting in a configuration containing only GC3Pie default values.

Example 1: initialization from config file:

>>> import os
>>> example_cfgfile = os.path.join(os.path.dirname(__file__), 'etc/gc3pie.conf.example')
>>> cfg = Configuration(example_cfgfile)
>>> cfg.debug
'0'

Example 2: initialization from key=value list:

>>> cfg = Configuration(auto_enable_auth=False, foo=1, bar='baz')
>>> cfg.auto_enable_auth
False
>>> cfg.foo
1
>>> cfg.bar
'baz'

When both a configuration file and a key=value list is present, values in the configuration files override those in the key=value list:

>>> cfg = Configuration(example_cfgfile, debug=1)
>>> cfg.debug
'0'

Example 3: default initialization:

>>> cfg = Configuration()
>>> cfg.auto_enable_auth
True

auth_factory¶

The instance of gc3libs.authentication.Auth used to manage auth access for the resources.

This is a read-only attribute, created upon first access with the values set in self.auths and self.auto_enabled.

load(*locations)¶

Merge settings from configuration files into this Configuration instance.

Environment variables and ~ references are expanded in the location file names.

If any of the specified files does not exist or cannot be read (for whatever reason), a message is logged but the error is ignored. However, a NoConfigurationFile exception is raised if none of the specified locations could be read.

Raises gc3libs.exceptions.NoConfigurationFile:
	if none of the specified files could be read.

make_auth(name)¶

Return factory for auth credentials configured in section [auth/name].

make_resources(ignore_errors=True)¶

Make backend objects corresponding to the configured resources.

Return a dictionary, mapping the resource name (string) into the corresponding backend object.

By default, errors in constructing backends (e.g., due to a bad configuration) are silently ignored: the offending configuration is just dropped. This can be changed by setting the optional argument ignore_errors to False: in this case, an exception is raised whenever we fail to construct a backend.

merge_file(filename)¶

Read configuration files and merge the settings into this Configuration object.

Contrary to load() (which see), the file name is taken literally and an error is raised if the file cannot be read for whatever reason.

Any parameter which is set in the configuration files [DEFAULT] section, and whose name does not start with underscore (_) defines an attribute in the current Configuration.

Warning

No type conversion is performed on values set this way - so they all end up being strings!

Raises gc3libs.exceptions.ConfigurationError:
	if the configuration file does not exist, cannot be read, is corrupt or has wrong format.

gc3libs.core¶

Top-level interface to Grid functionality.

class gc3libs.core.Core(cfg)¶

Core operations: submit, update state, retrieve (a snapshot of) output, cancel job.

Core operations are blocking, i.e., they return only after the operation has successfully completed, or an error has been detected.

Operations are always performed by a Core object. Core implements an overlay Grid on the resources specified in the configuration file.

add(task)¶

This method is here just to allow Core and Engine objects to be used interchangeably. It’s effectively a no-op, as it makes no sense in the synchronous/blocking semantics implemented by Core.

close()¶

Used to invoke explicitly the destructor on objects e.g. LRMS

fetch_output(app, download_dir=None, overwrite=False, **extra_args)¶

Retrieve output into local directory app.output_dir; optional argument download_dir overrides this.

The download directory is created if it does not exist. If it already exists, and the optional argument overwrite is False (default), it is renamed with a .NUMBER suffix and a new empty one is created in its place. Otherwise, if ‘overwrite` is True, files are downloaded over the ones already present.

If the task is in TERMINATING state, the state is changed to TERMINATED, attribute output_dir is set to the absolute path to the directory where files were downloaded, and the terminated transition method is called on the app object.

Task output cannot be retrieved when app.execution is in one of the states NEW or SUBMITTED; an OutputNotAvailableError exception is thrown in these cases.

Raise:	gc3libs.exceptions.OutputNotAvailableError if no output can be fetched from the remote job (e.g., the Application/Task object is in NEW or SUBMITTED state, indicating the remote job has not started running).

free(app, **extra_args)¶

Free up any remote resources used for the execution of app. In particular, this should delete any remote directories and files.

It is an error to call this method if app.execution.state is anything other than TERMINATED: an InvalidOperation exception will be raised in this case.

Raise:	gc3libs.exceptions.InvalidOperation if app.execution.state differs from Run.State.TERMINATED.

get_resources(**extra_args)¶

Return list of resources configured into this Core instance.

kill(app, **extra_args)¶

Terminate a job.

Terminating a job in RUNNING, SUBMITTED, or STOPPED state entails canceling the job with the remote execution system; terminating a job in the NEW or TERMINATED state is a no-op.

peek(app, what='stdout', offset=0, size=None, **extra_args)¶

Download size bytes (at offset bytes from the start) from the remote job standard output or error stream, and write them into a local file. Return file-like object from which the downloaded contents can be read.

If size is None (default), then snarf all available contents of the remote stream from offset unto the end.

The only allowed values for the what arguments are the strings ‘stdout’ and ‘stderr’, indicating that the relevant section of the job’s standard output resp. standard error should be downloaded.

remove(task)¶

This method is here just to allow Core and Engine objects to be used interchangeably. It’s effectively a no-op, as it makes no sense in the synchronous/blocking semantics implemented by Core.

select_resource(match)¶

Alter the configured list of resources, and retain only those that satisfy predicate match.

Argument match can be:

• either a function (or a generic callable) that is passed each Resource object in turn, and should return a boolean indicating whether the resources should be kept (True) or not (False);
• or it can be a string: only resources whose name matches (wildcards * and ? are allowed) are retained.

submit(app, resubmit=False, **extra_args)¶

Submit a job running an instance of the given app. Upon successful submission, call the submitted method on the app object.

At the beginning of the submission process, the app.execution state is reset to NEW; if submission is successfull, the task will be in SUBMITTED or RUNNING state when this call returns.

Raise:	gc3libs.exceptions.InputFileError if an input file does not exist or cannot otherwise be read.

update_job_state(*apps, **extra_args)¶

Update state of all applications passed in as arguments.

If keyword argument update_on_error is False (default), then application execution state is not changed in case a backend error happens; it is changed to UNKNOWN otherwise.

Note that if state of a job changes, the Run.state calls the appropriate handler method on the application/task object.

Raise:	gc3libs.exceptions.InvalidArgument in case one of the passed Application or Task objects is invalid. This can stop updating the state of other objects in the argument list.
Raise:	gc3libs.exceptions.ConfigurationError if the configuration of this Core object is invalid or otherwise inconsistent (e.g., a resource references a non-existing auth section).

update_resources(**extra_args)¶

Update the state of resources configured into this Core instance.

Each resource object in the returned list will have its updated attribute set to True if the update operation succeeded, or False if it failed.

class gc3libs.core.Engine(controller, tasks=[], store=None, can_submit=True, can_retrieve=True, max_in_flight=0, max_submitted=0, output_dir=None, fetch_output_overwrites=False)¶

Submit tasks in a collection, and update their state until a terminal state is reached. Specifically:

• tasks in NEW state are submitted;
• the state of tasks in SUBMITTED, RUNNING or STOPPED state is updated;
• when a task reaches TERMINATED state, its output is downloaded.

The behavior of Engine instances can be further customized by setting the following instance attributes:

can_submit

Boolean value: if False, no task will be submitted.

can_retrieve

Boolean value: if False, no output will ever be retrieved.

max_in_flight

If >0, limit the number of tasks in SUBMITTED or RUNNING state: if the number of tasks in SUBMITTED, RUNNING or STOPPED state is greater than max_in_flight, then no new submissions will be attempted.

max_submitted

If >0, limit the number of tasks in SUBMITTED state: if the number of tasks in SUBMITTED, RUNNING or STOPPED state is greater than max_submitted, then no new submissions will be attempted.

output_dir

Base directory for job output; if not None, each task’s results will be downloaded in a subdirectory named after the task’s permanent_id.

fetch_output_overwrites

Default value to pass as the overwrite argument to Core.fetch_output() when retrieving results of a terminated task.

Any of the above can also be set by passing a keyword argument to the constructor (assume g is a Core instance):

| >>> e = Engine(g, can_submit=False)
| >>> e.can_submit
| False

add(task)¶

Add task to the list of tasks managed by this Engine. Adding a task that has already been added to this Engine instance results in a no-op.

close()¶

Call explicilty finalize methods on relevant objects e.g. LRMS

fetch_output(task, output_dir=None, overwrite=False, **extra_args)¶

Proxy for Core.fetch_output (which see).

free(task, **extra_args)¶

Proxy for Core.free, which see.

kill(task, **extra_args)¶

Schedule a task for killing on the next progress run.

peek(task, what='stdout', offset=0, size=None, **extra_args)¶

Proxy for Core.peek (which see).

progress()¶

Update state of all registered tasks and take appropriate action. Specifically:

• tasks in NEW state are submitted;
• the state of tasks in SUBMITTED, RUNNING, STOPPED or UNKNOWN state is updated;
• when a task reaches TERMINATING state, its output is downloaded.
• tasks in TERMINATED status are simply ignored.

The max_in_flight and max_submitted limits (if >0) are taken into account when attempting submission of tasks.

remove(task)¶

Remove a task from the list of tasks managed by this Engine.

stats(only=None)¶

Return a dictionary mapping each state name into the count of tasks in that state. In addition, the following keys are defined:

• ok: count of TERMINATED tasks with return code 0
• failed: count of TERMINATED tasks with nonzero return code
• total: total count of managed tasks, whatever their state

If the optional argument only is not None, tasks whose whose class is not contained in only are ignored. : param tuple only: Restrict counting to tasks of these classes.

submit(task, resubmit=False, **extra_args)¶

Submit task at the next invocation of perform.

The task state is reset to NEW and then added to the collection of managed tasks.

update_job_state(*tasks, **extra_args)¶

Return list of current states of the given tasks. States will only be updated at the next invocation of progress; in particular, no state-change handlers are called as a result of calling this method.

gc3libs.exceptions¶

Exceptions specific to the gc3libs package.

In addition to the exceptions listed here, gc3libs functions try to use Python builtin exceptions with the same meaning they have in core Python, namely:

• TypeError is raised when an argument to a function or method has an incompatible type or does not implement the required protocol (e.g., a number is given where a sequence is expected).
• ValueError`is raised when an argument to a function or method has the correct type, but fails to satisfy other constraints in the function contract (e.g., a positive number is required, and `-1 is passed instead).
• AssertionError is raised when some internal assumption regarding state or function/method calling contract is violated. Informally, this indicates a bug in the software.

exception gc3libs.exceptions.ApplicationDescriptionError(msg, do_log=True)¶

Raised when the dumped description on a given Application produces something that the LRMS backend cannot process. As an example, for arc backends, this error is raised when the parsing of the Application’s XRSL fails

exception gc3libs.exceptions.AuthError(msg, do_log=False)¶

Base class for Auth-related errors.

Should never be instanciated: create a specific error class describing the actual error condition.

exception gc3libs.exceptions.ConfigurationError(msg, do_log=True)¶

Raised when the configuration file (or parts of it) could not be read/parsed. Also used to signal that a required parameter is missing or has an unknown/invalid value.

exception gc3libs.exceptions.DataStagingError(msg, do_log=False)¶

Base class for data staging and movement errors.

Should never be instanciated: create a specific error class describing the actual error condition.

exception gc3libs.exceptions.DetachedFromGridError(msg, do_log=False)¶

Raised when a method (other than attach()) is called on a detached Task instance.

exception gc3libs.exceptions.DuplicateEntryError(msg, do_log=False)¶

Raised by Application.__init__ if not all (local or remote) entries in the input or output files are distinct.

exception gc3libs.exceptions.Error(msg, do_log=False)¶

Base class for all error-level exceptions in GC3Pie.

Generally, this indicates a non-fatal error: depending on the nature of the task, steps could be taken to continue, but users must be aware that an error condition occurred, so the message is sent to the logs at the ERROR level.

Exceptions indicating an error condition after which the program cannot continue and should immediately stop, should use the FatalError base class.

exception gc3libs.exceptions.FatalError(msg, do_log=True)¶

A fatal error: execution cannot continue and program should report to user and then stop.

The message is sent to the logs at CRITICAL level when the exception is first constructed.

This is the base class for all fatal exceptions.

exception gc3libs.exceptions.InputFileError(msg, do_log=True)¶

Raised when an input file is specified, which does not exist or cannot be read.

exception gc3libs.exceptions.InternalError(msg, do_log=False)¶

Raised when some function cannot fulfill its duties, for reasons that do not depend on the library client code. For instance, when a response string gotten from an external command cannot be parsed as expected.

exception gc3libs.exceptions.InvalidArgument(msg, do_log=False)¶

Raised when the arguments passed to a function do not honor some required contract. For instance, either one of two optional arguments must be provided, but none of them was.

exception gc3libs.exceptions.InvalidOperation(msg, do_log=False)¶

Raised when an operation is attempted, that is not considered valid according to the system state. For instance, trying to retrieve the output of a job that has not yet been submitted.

exception gc3libs.exceptions.InvalidResourceName(msg, do_log=False)¶

Raised to signal that no computational resource with the given name is defined in the configuration file.

exception gc3libs.exceptions.InvalidType(msg, do_log=False)¶

A specialization of`InvalidArgument` for cases when the type of the passed argument does not match expectations.

exception gc3libs.exceptions.InvalidUsage(msg, do_log=True)¶

Raised when a command is not provided all required arguments on the command line, or the arguments do not match the expected syntax.

Since the exception message is the last thing a user will see, try to be specific about what is wrong on the command line.

exception gc3libs.exceptions.InvalidValue(msg, do_log=False)¶

A specialization of`InvalidArgument` for cases when the type of the passed argument does not match expectations.

exception gc3libs.exceptions.LoadError(msg, do_log=False)¶

Raised upon errors loading a job from the persistent storage.

exception gc3libs.exceptions.NoConfigurationFile(msg, do_log=True)¶

Raised when the configuration file cannot be read (e.g., does not exist or has wrong permissions), or cannot be parsed (e.g., is malformed).

exception gc3libs.exceptions.NoResources(msg, do_log=False)¶

Raised to signal that no resources are defined, or that none are compatible with the request.

exception gc3libs.exceptions.OutputNotAvailableError(msg, do_log=False)¶

Raised upon attempts to retrieve the output for jobs that are still in NEW or SUBMITTED state.

exception gc3libs.exceptions.RecoverableDataStagingError(msg, do_log=False)¶

Raised when transient problems with copying data to or from the remote execution site occurred.

This error is considered to be transient (e.g., network connectivity interruption), so trying again at a later time could solve the problem.

exception gc3libs.exceptions.RecoverableError(msg, do_log=False)¶

Used to mark transient errors: retrying the same action at a later time could succeed.

This exception should never be instanciated: it is only to be used in except clauses to catch “try again” situations.

exception gc3libs.exceptions.TaskError(msg, do_log=False)¶

Generic error condition in a Task object.

exception gc3libs.exceptions.UnexpectedStateError(msg, do_log=False)¶

Raised by Task.progress() when a job lands in STOPPED or TERMINATED state.

exception gc3libs.exceptions.UnknownJob(msg, do_log=False)¶

Raised when an operation is attempted on a task, which is unknown to the remote server or backend.

exception gc3libs.exceptions.UnknownJobState(msg, do_log=False)¶

Raised when a job state is gotten from the Grid middleware, that is not handled by the GC3Libs code. Might actually mean that there is a version mismatch between GC3Libs and the Grid middleware used.

exception gc3libs.exceptions.UnrecoverableDataStagingError(msg, do_log=False)¶

Raised when problems with copying data to or from the remote execution site occurred.

exception gc3libs.exceptions.UnrecoverableError(msg, do_log=False)¶

Used to mark permanent errors: there’s no point in retrying the same action at a later time, because it will yield the same error again.

This exception should never be instanciated: it is only to be used in except clauses to exclude “try again” situations.