This is a list of user-visible changes worth mentioning. In each new release, items are added to the top of the file and identify the version they pertain to.
- User-visible changes across releases
- GC3Pie 2.6
- GC3Pie 2.5
- GC3Pie 2.4
- GC3Pie 2.3
- GC3Pie 2.2
- GC3Pie 2.1
- GC3Pie 2.0
- GC3Pie 1.0
- GC3Pie 0.10
GC3Pie 2.6.0 introduces compatibility with Python 3.5+. The changes for this are rather extensive, but lucklily mostly confined to GC3Pie internals, so users of the library should not notice.
This release can introduce a few backwards-incompatible changes in the format for persisting tasks in files and databases (see Important changes below). Be sure to have all your currently-running sessions done before you upgrade!
This release depends on a few new external packages; if you’re upgrading from earlier sources, be sure to re-run pip install . in the GC3Pie source directory; no such additional step is needed if you’re installing from PyPI with pip install gc3pie or using GC3Pie’s own install.py script.
- Python 3.5+ is now fully supported and tested!
- The on-disk format for saving jobs might have changed incompatibly in some cases: a few internal classes have completely changed their inheritance hierarchy so Python’s pickle might not be able to read them back.
- GC3Pie now defaults to using “unicode” strings everywhere, but will
make a best attempt at converting parameters passed as byte strings:
- command-line arguments and paths for I/O need to be converted using the locale’s own encoding/charset and revert to mapping byte strings to Unicode code points by keeping the numeric value of bytes (instead of the textual / glyph value) if the former attempt has failed
- output from commands (e.g., when interacting with a batch-queuing system): we assume that programs are complying with the locale-defined encoding and use the locale’s own encoding to convert the output into a unicode text string.
- Minor (internal) API changes:
- class gc3libs.Default is now a separate module gc3libs.default.
- a few unused utility methods have been removed from module gc3libs.utils.
SessionBasedDaemonand accompanying “Inbox” classes to implement scripts that automatically detach themselves into background and react to events in configurable sources (filesystem, database, S3/SWIFT storage).
- Dropping cloud infrastructure support in Python 2.6; if you run GC3Pie on Python 2.6, you will only be able to run tasks on the “localhost” resource, or on any of the supported batch-queuing systems.
- Terminal log output is now colorized according to message level! (Thanks to Adrian Etter for suggesting this feature.)
- Old-style sessions are not supported any more. (This should not be a problem, as they have been automatically converted to “new-style” since years now. In the unlikely case you still have an old-style session directory on disk, just run any session command from version 2.4 and it will convert the format automatically.)
- The environment variable
GC3PIE_RESOURCE_INIT_ERRORS_ARE_FATALcan be set to
1to cause GC3Pie to abort if any errors occur while initializing the configured resources. The default behavior of GC3Pie is instead to keep running until there is at least one resource that can be used.
- A resource is now automatically disabled if an unrecoverable error occurs during its use.
The ARC backends and supporting code have been removed: it is no longer possible to use GC3Pie to submit tasks to an ARC job manager.
The environment variable
GC3PIE_NO_CATCH_ERRORSnow can specify a list of patterns to selectively unignore unexpected/generic errors in the code. As this feature should only be used in debugging code, we allow ourselves to break backwards compatibility.
The cloud and mathematics libraries are no longer installed by default with
pip install gc3pie– please use:
pip install gc3pie[openstack,ec2,optimizer]
to install support for all optional backends and libraries.
The gc3libs.utils.ifelse function was removed in favor of Python’s ternary operator.
- New task collection
DependentTaskCollectionto run a collection of tasks with given pre/post dependencies across them.
- GC3Pie will now parse and obey the
ProxyCommandoptions from the SSH config file. Location of an alternate configuration file to use with GC3Pie can be set in any
[auth/*]section of type SSH; see the Configuration File section for details. Thanks to Niko Eherenfeuchter and Karandash8 for feature requests and preliminary implementations.
- Application prologue and epilogue scripts can now be embedded in the GC3Pie configuration file, or referenced by file name.
- New selection options have been added to the gselect: select job ids from from a session command.
- gc3libs.Configuration will now raise different exceptions
depending on whether no files could be read
NoAccessibleConfigurationFile) or could not be parsed (
- Shell metacharacters are now allowed in Application arguments. Each argument string is now properly quoted before passing it to the execution layer.
- LSF backend updated to work with both bjobs and bacct for accounting, or to parse information provided in the final output file as a last resort.
- All backends should now set a Task’s returncode and exitcode values according to the documented meaning. Thanks to Y. Yakimovitch for reporting the issue.
openstackbackend for running jobs on ephemeral VMs on OpenStack-compatible IaaS cloud systems. This is preferred over the OpenStack EC2 compatibility layer.
- New configurable scheduler for GC3Pie’s
- Session-based scripts can now snapshot the output of RUNNING jobs at every cycle.
- ARC backends are now deprecated: they will be removed in the next major version of GC3Pie.
pbsbackend can now handle also Altair’s PBSPro.
gget: New option -A to download output files of all tasks in a session.
gget: New option
--changed-onlyto only download files that have apparently changed remotely.
- The GC3Apps collection has been enriched with several new applications.
- GC3Pie now requires Python 2.6 or above to run.
ec2backend for running jobs on ephemeral VMs on EC2-compatible IaaS cloud systems.
- New package gc3libs.optimizer to find local optima of functions that can be computed through a job. Currently only implements the “Differential Evolution” algorithm, but the framework is generic enough to plug any genetic algorithm.
- New configuration options
epilog_content, to allow execute oneliners before or after the command without having to create an auxiliary file.
shellcmdresources. This is used to modify the default value for the directory containing job informations.
- GC3Pie can now run on MacOSX.
- A session now has a configurable storage location, which can be a directory on the filesystem (FilesystemStore, the default so far) or can be a table in an SQL database (of any kind supported by SQLAlchemy).
- New ARC1 backend to use ARC resources through the new NorduGrid 1.x library API.
- New backend “subprocess”: execute applications as local processes.
- New backends for running on various batch-queueing systems: SLURM, LSF, PBS.
- Implement recursive upload and download of directories if they are specified in an Application’s input or output attribute.
- New execution state TERMINATING: task objects are in this state when execution is finished remotely, but the task output has not yet been retrieved.
- Reorganize documentation and move it to http://gc3pie.readthedocs.org/
- Script logging is now controlled by a single configuration file .gc3/gc3utils.log.conf
- Session-based scripts now print WARNING messages to STDERR by default (previously, only ERROR messages were logged).
- Add caching to ARC backends, to reduce the number of network queries.
- Use GNU “.~NUMBER~” format for backup directories.
- New gsession command to manage sessions.
glistcommand was renamed to
gnotifycommands were removed.
PATHtag no longer gets any special treatment in session-based scripts
ginfo: New option
--tabularto print information in table format.
gkill: New option -A/–all to remove all jobs in a session.
- Use the rungms script to execute GAMESS.
gc3libs.daghas been renamed to
- API changes in
gc3libs.cmdline.SessionBasedScriptallow new_tasks() in SessionBasedScript instances to return Task instances instead of quadruples.
- Interpret Application.requested_memory as the total memory for the job.
LRMSobjects were merged
gc3libs.schedulermodule has been removed; its functionality is now incorporated in the
- configuration-related code moved into gc3libs.config module
- removed the application registry.
- New package gc3libs.compat to provide 3rd-party functionality that is not present in all supported versions of Python.
- Implement gc3libs.ANY_OUTPUT to retrieve the full contents of the output directory, whatever it is.
- New RetryableTask class to wrap a task and re-submit it on failure until some specified condition is met.
- Renamed configuration file to
gc3pie.conf: the file
gc3utils.confwill no longer be read!
- SGE clusters must now have
type = sgein the configuration file (instead of
type = ssh-sge)
- All computational resource must have an
architecture = ...line; see the ConfigurationFile wiki page for details
- Probably more changes than it’s worth to list here: check your configuration against the Configuration File page!
- GC3Utils and GC3Apps (
ggamess/etc.) now all accept a
--sessionoption for locating the job storage directory: this allows grouping jobs into folders instead of shoveling them all into
- GC3Apps: replaced option
--states. The new option prints a table of submitted jobs in addition to the summary stats; if a comma-separated list of job states follows the option, only job in those states are printed.
gstatwill now print a summary of the job states if the list is too long to fit on screen; use the -v option to get the full job listing regardless of its length.
gstatcan now print information on jobs in a certain state only; see help text for option
-loption from ginfo; use
- GC3Utils: all commands accepting multiple job IDs on the command line, now exit with the number of errors/failures occurred. Since exit codes are practically limited to 7 bits, exit code 126 means that more than 125 failures happened.