This is the course material prepared for the “GC3Pie for Programmers” training, held at the University of Zurich for the first time on July 11-14, 2016. (The slides presented here are revised at each course re-run.)
The course aims at showing how to implement patterns commonly seen in scientific computational workflows using Python and GC3Pie, and provide users with enough knowledge of the tools available in GC3Pie to extend and adapt the examples provided.
A presentation of the training material and outline of the course. Probably not much useful unless you’re actually sitting in class.
A quick overview of the kind of computational use cases that GC3Pie can easily solve.
The basics needed to write simple GC3Pie scripts: the minimal session-based script scaffolding, and the properties and features of the
Recall a few GC3Pie utilities that are especially useful when debugging code.
How to set up command-line argument and option processing in GC3Pie’s
How to specify running requirements for
Applicationtasks, e.g., how much memory is needed to run.
How to check and react on the termination status of a GC3Pie Task/Application.
A worked-out example of a many-step workflow.
How to run tasks in sequence: basic usage of
How to run independent tasks in parallel: the
How to use the
DependentTaskCollectionfor automated arrangement of tasks given their dependencies.
How to construct
SequentialTaskCollectionclasses that change the sequence of tasks while being run.
In this tutorial we show how to use the GC3Pie libraries in order to build a command line script which runs a complex workflow with both parallelly- and sequentially-executing tasks.
The tutorial itself contains the complete source code of the
application (see Literate Programming on Wikipedia), so that you
will be able to test/modify it and produce a working
script by downloading the
pylit.py:file: script from the PyLit
Homepage and running the following command on the
from within the source tree of GC3Pie:
$ ./pylit warholize.rst warholize.py
A collection of small example scripts highlighting different features
of GC3Pie is available in the source distribution, in folder
Simplest script you can create. It only uses Application and Engine classes to create an application, submit it, check its status and retrieve its output.
a SessionBasedScript that executes its argument as command. It can also run it multiple times by wrapping it in a ParallelTaskCollection or a SequentialTaskCollection, depending on a command line option. Useful for testing a configured resource.
a simple SessionBasedScript that sums two values by customizing a SequentialTaskCollection.
an enhanced version of the warholize script proposed in the The “Warholize” Workflow Tutorial