Welcome to GC3Pie documentation!

Introduction

GC3Pie is a Python package for running large job campaigns on diverse batch-oriented execution environments (for instance: a Sun/Oracle/Open Grid Engine cluster, or the Swiss National Distributed Computing Infrastructure SMSCG). It also provides facilities for implementing command-line driver scripts, in the form of Python object classes whose behavior can be customized by overriding specified object methods.

GC3Pie documentation is divided in three sections:

• User Documentation: info on how to install, configure and run GC3Pie applications.
• Programmer Documentation: info for programmers who want to use the GC3Pie libraries to write their own scripts and applications.
• Contributors documentation: detailed information on how to contribute to GC3Pie and get your code included in the main library.

Table of Contents

• User Documentation
  • Table of Contents
    • Installation of GC3Pie
      • Quick start
      • Non-standard installation options
      • Manual installation
      • Upgrade
      • MacOSX Installation
      • HTML Documentation
    • Configuration File
      • Location
      • Configuration file format
      • The DEFAULT section
      • auth sections
        • Authentication types
          • none-type authentication
          • ssh-type authentication
          • ec2-type authentication
      • resource sections
        • Link to [auth/...] sections
        • Configuration keys common to all resource types
        • Configuration keys common to batch-queuing resource types
        • sge resources (all batch systems of the Grid Engine family)
        • pbs resources (TORQUE and PBSPro batch-queueing systems)
        • lsf resources (IBM LSF)
        • slurm resources
        • shellcmd resources
        • ec2+shellcmd resource
        • Example resource sections
      • Enabling/disabling selected resources
    • Environment Variables
    • GC3Pie usage tutorials
    • The GC3Apps software
      • Introduction to session-based scripts
        • Basic command-line usage and options
          • Continuous execution
          • Verbose listing of jobs
          • Maximum number of concurrent jobs
          • Location of output files
        • Job control options
        • Session control options
        • Exit code
      • The ggamess script
        • How to run GAMESS on the Grid
        • Tweaking execution
        • Updating the GAMESS code
      • The ggeotop script
        • Introduction
        • Command-line invocation of ggeotop
        • Output files for ggeotop
        • Example usage
          • Manage a set of jobs from start to end
        • Using GC3Pie utilities
          • gkill: cancel a running job
          • ginfo: accessing low-level details of a job
      • The grosetta and gdocking scripts
        • Introduction
        • Command-line invocation of grosetta
        • Output files for grosetta
        • Output files for gdocking
        • Example usage
          • Manage a set of jobs from start to end
          • Managing a session by repeated grosetta invocation
      • The gcrypto script
        • Introduction
        • Command-line invocation of gcrypto
        • Output files for gcrypto
        • Example usage
          • Manage a set of jobs from start to end
        • Using GC3Pie utilities
          • gkill: cancel a running job
          • ginfo: accessing low-level details of a job
    • The GC3Utils software
      • gsession: manage sessions
      • gstat: monitor the status of submitted jobs
      • gtail: peeking at the job output and error report
      • gkill: cancel a running job
      • gget: retrieve the output of finished jobs
      • gclean: remove a completed job from the status list
      • gresub: re-submit a failed job
      • gservers: list available resources
      • ginfo: accessing low-level details of a job
      • gselect: select job ids from from a session
      • gcloud: manage VMs created by the EC2 backend
    • Troubleshooting GC3Pie
      • What is a Python traceback?
      • Common errors using GC3Pie
        • AttributeError: module object has no attribute StringIO
        • DistributionNotFound
        • ImportError: No module named pstats
        • NoResources: Could not initialize any computational resource - please check log and configuration file.
        • ValueError: I/O operation on closed file
      • ValueError: Expected version spec in …
    • User-visible changes across releases
      • GC3Pie 2.6
        • Important changes
        • Bug fixes
      • GC3Pie 2.5
        • New features
        • Incompatible changes
      • GC3Pie 2.4
        • New features
      • GC3Pie 2.3
        • Incompatible changes
        • New features
        • Important bug fixes
      • GC3Pie 2.2
        • New features
        • Changes to command-line utilities
        • Important bug fixes
      • GC3Pie 2.1
        • New features and incompatible changes
        • Changes to command-line utilities
      • GC3Pie 2.0
        • New features and incompatible changes
        • Configuration file changes
        • Changes to command-line utilities
        • API changes
      • GC3Pie 1.0
        • Configuration file changes
        • Command-line utilities changes
      • GC3Pie 0.10
• Programmer Documentation
  • Programming overview
    • Computational job lifecycle
    • Managing jobs with GC3Libs
    • Execution model of GC3Libs applications
      • Application exit codes
      • Application execution states
      • Computational job specification
  • GC3Pie programming tutorials
    • Implementing scientific workflows with GC3Pie
    • A bottom-up introduction to programming with GC3Pie
    • The “Warholize” Workflow Tutorial
      • Introduction
      • From top to bottom
      • The script
      • The workflows
        • Main sequential workflow
        • Step one: convert to grayscale
      • Step two: parallel workflow to create colorized images
        • Step three: merge all them together
      • Making the script executable
      • Testing
    • Example scripts
  • GC3Libs programming API
    • gc3libs
    • gc3libs.application
    • gc3libs.application.apppot
    • gc3libs.application.codeml
    • gc3libs.application.demo
    • gc3libs.application.gamess
    • gc3libs.application.rosetta
    • gc3libs.application.turbomole
    • gc3libs.authentication
    • gc3libs.authentication.ec2
    • gc3libs.authentication.openstack
    • gc3libs.authentication.ssh
    • gc3libs.backends
    • gc3libs.backends.batch
    • gc3libs.backends.ec2
    • gc3libs.backends.lsf
    • gc3libs.backends.noop
    • gc3libs.backends.openstack
    • gc3libs.backends.pbs
    • gc3libs.backends.sge
    • gc3libs.backends.shellcmd
    • gc3libs.backends.slurm
    • gc3libs.backends.transport
    • gc3libs.backends.vmpool
    • gc3libs.cmdline
    • gc3libs.config
    • gc3libs.core
    • gc3libs.debug
    • gc3libs.defaults
    • gc3libs.events
    • gc3libs.exceptions
    • gc3libs.optimizer
    • gc3libs.optimizer.dif_evolution
    • gc3libs.optimizer.drivers
    • gc3libs.optimizer.extra
    • gc3libs.persistence
      • A usage warning
    • gc3libs.persistence.accessors
    • gc3libs.persistence.filesystem
    • gc3libs.persistence.idfactory
    • gc3libs.persistence.serialization
    • gc3libs.persistence.sql
    • gc3libs.persistence.store
    • gc3libs.poller
    • gc3libs.quantity
    • gc3libs.session
    • gc3libs.template
    • gc3libs.testing
    • gc3libs.testing.helpers
    • gc3libs.url
    • gc3libs.utils
    • gc3libs.workflow
    • gc3utils
    • gc3utils.commands
    • gc3utils.frontend
• Contributors documentation
  • Contributing to GC3Pie
    • Code repository organization
      • Repository structure
    • Package generation
    • Testing the code
      • Running the tests
        • Running tests with pytest
        • Running multiple tests
      • Writing tests
      • Organizing tests
    • Coding style
    • Documentation
    • Questions?
• Publications
  • GC3Pie overviews
  • Programming examples
  • Use of GC3Pie in scientific applications
• List of contributors to GC3Pie
• Glossary

Indices and tables

• Index
• Module Index
• Search Page