session - persistent collections of tasks

class gc3libs.session.Session(path, create=True, store_or_url=None, load=True, task_ids=None, **extra_args)

A ‘session’ is a persistent collection of tasks.

Tasks added to the session are persistently recorded using an instance of gc3libs.persistence.Store. Stores can be shared among different sessions: each session knows wich jobs it ‘owns’.

A session is associated to a directory, which holds all the data releated to that session. Specifically, two files are always created in the session directory andused internally by this class:

  • index.txt: contains a list of all job IDs associated with this session;
  • store.url: its contents are the URL of the store to create (as would be passed to the gc3libs.persistence.make_store factory).

To only argument needed to instantiate a session is the path of the directory; the directory name will be used as the identifier of the session itself. For example, the following code creates a temporary directory and the two files mentioned above inside it:

>>> import tempfile; tmpdir = tempfile.mktemp(dir='.')
>>> session = Session(tmpdir)
>>> for name in sorted(os.listdir(tmpdir)):
...   print(name)

When a Session object is created with a path argument pointing to an existing valid session, the index of jobs is automatically loaded into memory, and the store pointed to by the store.url file in the session directory will be used, disregarding the contents of the `store_url` argument.

In other words, the store_url argument is only used when creating a new session. If no store_url argument is passed (i.e., it has its default value), a Session object will instantiate and use a FileSystemStore store, keeping data in the jobs subdirectory of the session directory.

Methods add and remove are provided to manage the collection; the len() operator returns the number of tasks in the session; iteration over a session returns the tasks one by one:

>>> task1 = gc3libs.Task()
>>> id1 = session.add(task1)
>>> task2 = gc3libs.Task()
>>> id2 = session.add(task2)
>>> len(session)
>>> for t in session:
...     print(type(t))
<class 'gc3libs.Task'>
<class 'gc3libs.Task'>
>>> session.remove(id1)
>>> len(session)

When passed the flush=False optional argument, methods add and remove do not update the session metadata: i.e., the tasks are added or removed from the store and the in-memory task list, but the updated task list is not saved back to disk. This is useful when making many changes in a row; call Session.flush to persist the full set of changes.

The Store object is anyway accessible in the store attribute of each Session instance:

>>> type(session.store)
<class 'gc3libs.persistence.filesystem.FilesystemStore'>

However, Session defines methods save and load as a convenient proxy to the corresponding Store methods:

>>> obj = gc3libs.persistence.Persistable()
>>> oid = session.save(obj)
>>> obj2 = session.load(oid)
>>> obj.persistent_id == obj2.persistent_id

The whole session data can be removed by using method destroy:

>>> session.destroy()
>>> os.path.exists(session.path)
add(task, flush=True)

Add a Task to the current session, save it to the associated persistent storage, and return the assigned persistent_id:

>>> # create new, empty session
>>> import tempfile; tmpdir = tempfile.mktemp(dir='.')
>>> session = Session(tmpdir)
>>> len(session)

>>> # add a task to it
>>> task = gc3libs.Task()
>>> tid1 = session.add(task)
>>> len(session)

Duplicates are silently ignored: the same object can be added many times to the session, but gets the same ID each time:

>>> # add a different task
>>> tid2 = session.add(task)
>>> len(session)
>>> tid1 == tid2

>>> # do cleanup
>>> session.destroy()
>>> os.path.exists(session.path)

Remove the session directory and all the tasks it contains from the store which are associated to this session.


This will remove the associated task storage if and only if the storage is contained in the session directory!


Update session metadata.

Should be used after a save/remove operations, to ensure that the session state and metadata is correctly persisted.

forget(task_id, flush=True)

Remove task identified by task_id from the current session but not from the associated storage.


Iterate over all tasks in this session.

Recursively descends TaskCollection in a breadth-first manner.


Return set of all task IDs belonging to this session.


Return set of names of tasks belonging to this session.

load(obj_id, add=True, flush=True)

Load an object from persistent storage and return it.

This is just a convenience proxy for calling method load on the Store instance associated with this session.

load_many(obj_ids, add=True, flush=True)

Load objects given their IDs from persistent storage.

Return a dictionary mapping task ID to the actual retrieved Task object.

remove(task_id, flush=True)

Remove task identified by task_id from the current session and from the associated storage.


Save an object to the persistent storage and return persistent_id of the saved object.

This is just a convenience proxy for calling method save on the Store instance associated with this session.

The object is not added to the session, nor is session meta-data updated:

# create an empty session
>>> import tempfile; tmpdir = tempfile.mktemp(dir='.')
>>> session = Session(tmpdir)
>>> 0 == len(session)

# use `save` on an object
>>> obj = gc3libs.persistence.Persistable()
>>> oid = session.save(obj)

# session is still empty
>>> 0 == len(session)

# do cleanup
>>> session.destroy()
>>> os.path.exists(session.path)

Save all modified tasks to persistent storage.


Create a file named finished in the session directory. It’s creation/modification time will be used to know when the session has finished.

Please note that Session does not know when a session is finished, so this method should be called by a SessionBasedScript class.


Create a file named created in the session directory. It’s creation/modification time will be used to know when the session has sarted.

class gc3libs.session.TemporarySession(store_or_url, task_ids=None, delete=True, **extra_args)

Create a session from a store URL.

In contrast with the regular Session object, a TemporarySession does not persist any metadata about the task collection. In particular:

  • The session index (list of task IDs belonging to the session) is initialized from the entire list of jobs present in the given Store (unless a list is explicitly passed in the task_ids argument to the constructor). This means that, unlike plain Session objects, two TemporarySession objects cannot share the same store.
  • The session directory (path in the Session constructor) is created on a temporary location on the filesystem and deleted when the TemporarySession is destroyed.
  • Timestamps will be set to the time the TemporarySession Python object is created; two TemporarySession instances with the same backing store can have different creation timestamps, depending on when exactly they were instanciated.

The TemporarySession is only provided as a convenience to use code that was built on top of a Session with a “naked” Store.