Facade to store and retrieve Job information from permanent storage.

A usage warning

This module saves Python objects using the pickle framework: thus, the Application subclass corresponding to a job must be already loaded (or at least import-able) in the Python interpreter for pickle to be able to ‘undump’ the object from its on-disk representation.

In other words, if you create a custom Application subclass in some client code, GC3Utils won’t be able to read job files created by this code, because the class definition is not available in GC3Utils.

The recommended simple workaround is for a stand-alone script to ‘import self’ and then use the fully qualified name to run the script. In other words, start your script with this boilerplate code:

if __name__ == '__main__':
    import myscriptname

The rest of the script now runs as the myscript module, which does the trick!


Of course, the myscript.py file must be in the search path of the Python interpreter, or GC3Utils will still complain!

gc3libs.persistence.make_store(uri, *args, **extra_args)

Factory producing concrete Store instances.

Given a URL and (optionally) initialization arguments, return a fully-constructed Store instance.

The only required argument is uri; if any other arguments are present in the function invocation, they are passed verbatim to the constructor associated with the scheme of the given uri.


>>> fs1 = make_store('file:///tmp')
>>> fs1.__class__.__name__

Argument uri can also consist of a path name, in which case a URL scheme ‘file:///’ is assumed:

>>> fs2 = make_store('/tmp')
>>> fs2.__class__.__name__
class gc3libs.persistence.Persistable(*args, **kwargs)

A mix-in class to mark that an object should be persisted by its ID.

Any instance of this class is saved as an ‘external reference’ when a container holding a reference to it is saved.

class gc3libs.persistence.IdFactory(prefix=None, next_id_fn=None, id_class=<class 'gc3libs.persistence.idfactory.Id'>)

Automatically generate a “unique identifier” (of class Id). Object identifiers are temporally unique: no identifier will (ever) be re-used, even in different invocations of the program.


Return a new “unique identifier” instance (a string).


Pre-allocate n IDs. Successive invocations of the Id constructor will return one of the pre-allocated, with a potential speed gain if many Id objects are constructed in a loop.

class gc3libs.persistence.JobIdFactory(next_id_fn=None)

Override IdFactory behavior and generate IDs starting with a lowercase job prefix.

class gc3libs.persistence.FilesystemStore(directory='/home/docs/.gc3/jobs', idfactory=<gc3libs.persistence.idfactory.IdFactory object>, protocol=4, **extra_args)

Save and load objects in a given directory. Uses Python’s standard pickle module to serialize objects onto files.

All objects are saved as files in the given directory (default: gc3libs.defaults.JOBS_DIR). The file name is the object ID.

If an object contains references to other Persistable objects, these are saved in the file they would have been saved if the save method was called on them in the first place, and only an ‘external reference’ is saved in the pickled container. This ensures that: (1) only one copy of a shared object is ever saved, and (2) any shared reference to Persistable objects is correctly restored when restoring the container.

The default idfactory assigns object IDs by appending a sequential number to the class name; see class Id for details.

The protocol argument specifies the serialization protocol to use, if different from gc3libs.persistence.serialization.DEFAULT_PROTOCOL.

Any extra keyword arguments are ignored for compatibility with SqlStore.


Clear the loaded objects cache (if any).

Subsequent load() calls are guaranteed to re-load the data directly from the backing store.


Return list of IDs of saved Job objects.

This is an optional method; classes that do not implement it should raise a NotImplementedError exception.


Load a saved object given its ID, and return it.


Delete a given object from persistent storage, given its ID.

replace(id_, obj)

Replace the object already saved with the given ID with a copy of obj.


Save an object, and return an ID.