gc3libs.utils

Generic Python programming utility functions.

This module collects general utility functions, not specifically related to GC3Libs. A good rule of thumb for determining if a function or class belongs in here is the following: place a function or class in this module if you could copy its code into the sources of a different project and it would not stop working.

class gc3libs.utils.Enum

A generic enumeration class. Inspired by: http://goo.gl/1AL5N0 with some more syntactic sugar added.

An Enum class must be instanciated with a list of strings, that make the enumeration “label”:

>>> Animal = Enum('CAT', 'DOG')

Each label is available as an instance attribute, evaluating to itself:

>>> Animal.DOG
'DOG'

>>> Animal.CAT == 'CAT'
True

As a consequence, you can test for presence of an enumeration label by string value:

>>> 'DOG' in Animal
True

Finally, enumeration labels can also be iterated upon:

>>> for a in sorted(Animal): print(a)
CAT
DOG
class gc3libs.utils.ExponentialBackoff(slot_duration=0.05, max_retries=5)

Generate waiting times with the exponential backoff algorithm.

Returned times are in seconds (or fractions thereof); they are integral multiples of the basic time slot, which is set with the slot_duration constructor parameter.

After max_retries have been attempted, any call to this iterator will raise a StopIteration exception.

The ExponentialBackoff class implements the iterator protocol, so you can just retrieve waiting times with the .next() method, or by looping over it:

>>> lapses = list(ExponentialBackoff(max_retries=7))
>>> len(lapses)
8
wait()

Wait for another while.

class gc3libs.utils.History

A list of messages with timestamps and (optional) tags.

The append method should be used to add a message to the History:

>>> L = History()
>>> L.append('first message')
>>> L.append('second one')

The last method returns the text of the last message appended, with its timestamp:

>>> L.last().startswith('second one at')
True

Iterating over a History instance returns message texts in the temporal order they were added to the list, with their timestamp:

>>> for msg in L: print(msg) 
first message ...
append(message, *tags)

Append a message to this History.

The message is timestamped with the time at the moment of the call.

The optional tags argument is a sequence of strings. Tags are recorded together with the message and may be used to filter log messages given a set of labels. (This feature is not yet implemented.)

format_message(message)

Return a formatted message, appending to the message its timestamp in human readable format.

last()

Return text of last message appended. If log is empty, return empty string.

exception gc3libs.utils.NeverUsedException

this exception should never be raised

class gc3libs.utils.PlusInfinity

An object that is greater-than any other object.

>>> x = PlusInfinity()
>>> x > 1
True
>>> 1 < x
True
>>> 1245632479102509834570124871023487235987634518745 < x
True
>>> x > sys.maxsize
True
>>> x < sys.maxsize
False
>>> sys.maxsize < x
True

PlusInfinity objects are actually larger than any given Python object:

>>> x > 'azz'
True
>>> x > object()
True

Relational operators try to return the correct value when comparing PlusInfinity to other instances of PlusInfinity:

>>> y = PlusInfinity()
>>> x < y
False
>>> x <= y
True
>>> x == y
True
>>> x >= y
True
>>> x > y
False

Finally, addition and subtraction of a finite number from PlusInfinity always results in PlusInfinity:

>>> y = x - 1
>>> x == y
True
>>> y = x + 1
>>> x == y
True

Note that this used to be a singleton with special handling of ‘is’. This was removed because it was apparently unused, and a lot of extra code.

class gc3libs.utils.Struct(initializer=None, **extra_args)

A dict-like object, whose keys can be accessed with the usual ‘[…]’ lookup syntax, or with the ‘.’ get attribute syntax.

Examples:

>>> a = Struct()
>>> a['x'] = 1
>>> a.x
1
>>> a.y = 2
>>> a['y']
2

Values can also be initially set by specifying them as keyword arguments to the constructor:

>>> a = Struct(z=3)
>>> a['z']
3
>>> a.z
3

Like dict instances, Struct`s have a `copy method to get a shallow copy of the instance:

>>> b = a.copy()
>>> b.z
3
copy()

Return a (shallow) copy of this Struct instance.

keys() → a set-like object providing a view on D's keys
gc3libs.utils.WindowsError

alias of gc3libs.utils.NeverUsedException

class gc3libs.utils.YieldAtNext(generator)

Provide an alternate protocol for generators.

Wrap a Python generator object, and buffer the return values from send and throw calls, returning None instead. Return the yielded value –or raise the StopIteration exception– upon the subsequent call to the next method.

gc3libs.utils.backup(path)

Rename the filesystem entry at path by appending a unique numerical suffix; return new name.

For example,

  1. create a test file:
>>> import tempfile
>>> path = tempfile.mkstemp()[1]
  1. then make a backup of it; the backup will end in .~1~:
>>> path1 = backup(path)
>>> os.path.exists(path + '.~1~')
True

3. re-create the file, and make a second backup: this time the file will be renamed with a .~2~ extension:

>>> open(path, 'w').close()
>>> path2 = backup(path)
>>> os.path.exists(path + '.~2~')
True

cleaning up tests

>>> os.remove(path+'.~1~')
>>> os.remove(path+'.~2~')
gc3libs.utils.basename_sans(path)

Return base name without the extension.

This behaves exactly like os.path.basename() except that the last few characters, up to the rightmost dot, are removed as well:

>>> basename_sans('/tmp/foo.txt') == 'foo'
True

>>> basename_sans('bar.txt') == 'bar'
True

If there is no dot in the file name, no “extension” is chopped off:

>>> basename_sans('baz') == 'baz'
True

If there are several dots in the file name, only the last one and trailing characters are removed:

>>> basename_sans('foo.bar.baz') == 'foo.bar'
True

Leading directory components are chopped off in any case:

>>> basename_sans('/tmp/foo.bar.baz') == 'foo.bar'
True

>>> basename_sans('/tmp/foo') == 'foo'
True
gc3libs.utils.cache_for(lapse)

Cache the result of a (nullary) method invocation for a given amount of time. Use as a decorator on object methods whose results are to be cached.

Store the result of the first invocation of the decorated method; if another invocation happens before lapse seconds have passed, return the cached value instead of calling the real function again. If a new call happens after the grace period has expired, call the real function and store the result in the cache.

Note: Do not use with methods that take keyword arguments, as they will be discarded! In addition, arguments are compared to elements in the cache by identity, so that invoking the same method with equal but distinct object will result in two separate copies of the result being computed and stored in the cache.

Cache results and timestamps are stored into the objects’ _cache_value and _cache_last_updated attributes, so the caches are destroyed with the object when it goes out of scope.

The working of the cached method can be demonstrated by the following simple code:

>>> class X(object):
...     def __init__(self):
...         self.times = 0
...     @cache_for(2)
...     def foo(self):
...             self.times += 1
...             return self.times
>>> x = X()
>>> x.foo()
1
>>> x.foo()
1
>>> time.sleep(3)
>>> x.foo()
2
gc3libs.utils.cat(*args, **extra_args)

Concatenate the contents of all args into output. Both output and each of the args can be a file-like object or a string (indicating the path of a file to open).

If append is True, then output is opened in append-only mode; otherwise it is overwritten.

gc3libs.utils.check_file_access(path, mode, exception=<class 'RuntimeError'>, isdir=False)

Test for access to a path; if access is not granted, raise an instance of exception with an appropriate error message. This is a frontend to os.access(), which see for exact semantics and the meaning of path and mode.

Parameters:
  • path – Filesystem path to test.
  • mode – See os.access()
  • exception – Class of exception to raise if test fails.
  • isdir – If True then also test that path points to a directory.

If the test succeeds, True is returned:

>>> check_file_access('/bin/sh', os.F_OK)
True
>>> check_file_access('/bin/sh', os.R_OK)
True
>>> check_file_access('/bin/sh', os.X_OK)
True
>>> check_file_access('/tmp', os.X_OK)
True

However, if the test fails, then an exception is raised:

>>> check_file_access('/proc/version', os.W_OK)
Traceback (most recent call last):
  ...
RuntimeError: Cannot write to file '/proc/version'.

If the optional argument isdir is True, then additionally test that path points to a directory inode:

>>> check_file_access('/tmp', os.F_OK, isdir=True)
True

>>> check_file_access('/bin/sh', os.F_OK, isdir=True)
Traceback (most recent call last):
  ...
RuntimeError: Expected '/bin/sh' to be a directory, but it's not.
gc3libs.utils.deploy_configuration_file(filename, template_filename=None)

Ensure that configuration file filename exists; possibly copying it from the specified template_filename.

Return True if a file with the specified name exists in the configuration directory. If not, try to copy the template file over and then return False; in case the copy operations fails, a NoConfigurationFile exception is raised.

The template_filename is always resolved relative to GC3Libs’ ‘package resource’ directory (i.e., the etc/ directory in the sources. If template_filename is None, then it is assumed to be the base name of filename.

gc3libs.utils.dirname(pathname)

Same as os.path.dirname but return . in case of path names with no directory component.

gc3libs.utils.fgrep(literal, filename)

Iterate over all lines in a file that contain the literal string.

gc3libs.utils.first(seq)

Return the first element of sequence or iterator seq. Raise TypeError if the argument does not implement either of the two interfaces.

Examples:

>>> s = [0, 1, 2]
>>> first(s)
0

>>> s = {'a':1, 'b':2, 'c':3}
>>> first(sorted(s.keys())) == 'a'
True
gc3libs.utils.from_encoded_bytes(chars, encodings)

Convert chars to Python unicode string, trying different encodings.

Try converting byte string chars to a Python text string (type unicode on Py2, type str on Py3), trying each of the encodings specified until one of them succeeds.

If none of the encodings work, raise UnicodeDecodeError.

Parameters:
  • chars (bytes) – Byte string to convert to text
  • encodings (list) – List of encodings to try, in sequence. (e.g., ['utf-8', 'latin-1']
Raises:

UnicodeDecodeError – When none of the encodings can successfully convert the given byte string.

gc3libs.utils.from_filesystem_bytes(chars)

Convert chars to Python unicode string, trying different encodings.

This function should be used to make a Python text string (type unicode on Python 2, type str on Python 3) out of a byte string of characters that result from a filesystem lookup operation. Conversion to a text string is attempted using the following encodings, in order:

  1. the encoding determined by the current locale (as determined by Python’s locale.getlocale());
  2. the “user’s preferred encoding”, as determined by Python’s locale.getpreferredencoding();
  3. UTF-8 encoding;
  4. direct map of byte values 0x0 through 0xff to the corresponding Unicode code points.

The latter conversion will not be a valid text conversion (i.e., it will not preserve any text representation of the string), unless the ISO-8859-1 (aka “latin-1”) encoding is used.

However, the issue we need to solve here is the mismatch of Python’s use of text strings to represent path names with UNIX’ C library use of byte strings to represent the same. Since the UNIX kernel has no notion of character encoding in path names (as opposed to Windows, which uses UNICODE since Win95), there is actually no guarantee that a given file name can be decoded. This is particularly evident on multi-user Linux/UNIX systems where you may be using e.g. the en_US.UTF-8 locale but your Russian colleague may be using KOI8-R…

Parameters:chars (bytes) – Byte string to convert to text
Raises:UnicodeDecodeError – When none of the encodings can successfully convert the given byte string.
gc3libs.utils.from_template(template, **extra_args)

Return the contents of template, substituting all occurrences of Python formatting directives ‘%(key)s’ with the corresponding values taken from dictionary extra_args.

If template is an object providing a read() method, that is used to gather the template contents; else, if a file named template exists, the template contents are read from it; otherwise, template is treated like a string providing the template contents itself.

gc3libs.utils.from_terminal_bytes(chars)

Convert chars to Python unicode string, using current locale encoding.

This function should be used to make a Python text string (type unicode on Python 2, type str on Python 3) out of a byte string of characters that were inputed by users in a terminal application (e.g., returned by input() or typed as command-line arguments).

If the current locale encoding cannot be determined, then the byte string is assumed to be an ASCII-only string.

Parameters:chars (bytes) – Byte string to convert to text
Raises:UnicodeDecodeError – When none of the encodings can successfully convert the given byte string.
gc3libs.utils.get_available_physical_memory()

Return size of available memory (as a gc3libs.quantity.Memory object). The figure only refers to RAM, i.e., physical memory as opposed to virtual memory (swap).

Should work on any POSIX system that supports the _SC_AVPHYS_PAGES variable in the sysconf() C library call.

Raises:NotImplementedError – If syscalls to determine amount of available physical memory are not implemented on this system.
gc3libs.utils.get_linux_memcg_limit()

Return memory limit in this process’ Linux memory cgroup. Return value is a gc3libs.quantity.Memory object, or None if no limit can be detected.

As the Linux “memory cgroup” mechanism implements different limits, and not all of them might have been set/enforced, we we read possible limits in supposedly ascending order (“soft” limits should be lower than “hard” limits) and return first one that exists.

See also: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

gc3libs.utils.get_max_real_memory()

Return maximum size of available real memory. Return value is a gc3libs.quantity.Memory object, or None if no limit can be detected.

Various sources are polled for a limit, and the minimum is returned:

  • available physical memory, as reported by sysconf(3);
  • current resource limits (also known as ulimit in shell programming) RLIMIT_DATA and RLIMIT_AS;
  • current Linux memory cgroup limits.
gc3libs.utils.get_num_processors()

Return number of online processor cores.

gc3libs.utils.get_scheduler_and_lock_factory(lib)

Return factories for creating a period task scheduler and locks.

The scheduler will be a scheduler class from the APScheduler framework (which see for the API), and the lock factory is an appropriate locking object for synchronizing independently running tasks. Example:

sched_factory, lock_factory = _get_scheduler_and_lock_factory('threading')
sched = sched_factory()
sched.add_job(task1, 'interval', seconds=5)
sched.add_job(task2, 'interval', seconds=30)

shared_data_lock = lock_factory()

def task1():
  # ...
  with shared_data_lock:
    # modify shared data

Argument lib is one of: threading, gevent, tornado, asyncio (Python 3.5+ “async” system), twisted, qt; each of them selects a scheduler and lock objects compatible with the named framework for concurrent processing.

gc3libs.utils.getattr_nested(obj, name)

Like Python’s getattr, but perform a recursive lookup if name contains any dots.

gc3libs.utils.grep(pattern, filename)

Iterate over all lines in a file that match the pattern regular expression.

gc3libs.utils.ifelse(test, if_true, if_false)

Return if_true is argument test evaluates to True, return if_false otherwise.

This is just a workaround for Python 2.4 lack of the conditional assignment operator:

>>> a = 1
>>> b = ifelse(a, "yes", "no"); print(b)
yes
>>> b = ifelse(not a, 'yay', 'nope'); print(b)
nope
gc3libs.utils.irange(start, stop, step=1)

Iterate over all values greater or equal than start and less than stop. (Or the reverse, if step < 0.)

Example:

>>> list(irange(1, 5))
[1, 2, 3, 4]
>>> list(irange(0, 8, 3))
[0, 3, 6]
>>> list(irange(8, 0, -2))
[8, 6, 4, 2]

Unlike the built-in range function, irange also accepts floating-point values:

>>> list(irange(0.0, 1.0, 0.5))
[0.0, 0.5]

Also unlike the built-in range, both start and stop have to be specified:

>>> try:
...   irange(42)
... except TypeError:
...   print("missing required argument!")
missing required argument!

Of course, a null step is not allowed:

>>> try:
...   list(irange(1, 2, 0))
... except AssertionError as err:
...   assert 'Null step in irange.' in str(err)
gc3libs.utils.lock(path, timeout, create=True)

Lock the file at path. Raise a LockTimeout error if the lock cannot be acquired within timeout seconds.

Return a lock object that should be passed unchanged to the gc3libs.utils.unlock function.

If no path points to a non-existent location, an empty file is created before attempting to lock (unless create is False). An attempt is made to remove the file in case an error happens.

See also: gc3libs.utils.unlock()

gc3libs.utils.lookup(obj, name)

Return attribute or item with the given name in collection obj.

Raises:LookupError – If obj has no attribute nor item with the given name.

This is meant for cases when different versions of an API may either return a dictionary (hence, key/__getitem__-based lookup) or an object/namespace (hence, ./getattr-style lookup) and you want to handle them in a uniform way.

The following examples demo it:

>>> D = {'a':1, 'b':2}
>>> lookup(D, 'a')
1
>>> lookup(D, 'c')
Traceback (most recent call last):
  ...
LookupError: Object ... has no attribute nor key named `c`

>>> class X(object):
...   a = 1
...   b = 2
>>> x = X()
>>> lookup(x, 'a')
1
>>> lookup(x, 'c')
Traceback (most recent call last):
  ...
LookupError: Object ... has no attribute nor key named `c`
gc3libs.utils.mkdir(path, mode=511)

Like os.makedirs, but does not throw an exception if PATH already exists.

gc3libs.utils.mkdir_with_backup(path, mode=511)

Like os.makedirs, but if path already exists and is not empty, rename the existing one to a backup name (see the backup function).

Unlike os.makedirs, no exception is thrown if the directory already exists and is empty, but the target directory permissions are not altered to reflect mode.

gc3libs.utils.move_recursively(src, dst, overwrite=False, changed_only=True)

Move src to dst, descending it recursively if necessary.

The overwrite and changed_only optional arguments have the same effect as in copytree() (which see).

gc3libs.utils.movefile(src, dst, overwrite=False, changed_only=True, link=False)

Move a file from src to dst; return True if the move was actually made.

The overwrite and changed_only optional arguments have the same effect as in copyfile() (which see).

If dst is a directory, a file with the same basename as src is created (or overwritten) in the directory specified.

Return True or False, depending on whether the source file was actually moved to the destination.

See also: copyfile()

gc3libs.utils.movetree(src, dst, overwrite=False, changed_only=True)

Recursively move an entire directory tree rooted at src.

The overwrite and changed_only optional arguments have the same effect as in copytree() (which see).

See also: copytree().

gc3libs.utils.occurs(pattern, filename, match=<function grep>)

Return True if a line in filename matches pattern.

The match argument selects how exactly pattern is searched for in the contents of filename:

  • when match=grep (default), then pattern is a regular expression that is searched for (unanchored) in every line;
  • when match=fgrep, then pattern is a string that is searched for literally in every line;
  • more in general, the match function should return an iterator over matches of pattern within the contents of filename: if at least one match is found, occurs will return True.
Parameters:
  • pattern (str) – Pattern to search for
  • filename (str) – Path name of the file to search into
  • match – Function returning iterator over matches
gc3libs.utils.parse_linux_proc_limits(data=None)

Return dictionary mapping limit name to corresponding value. In case the actual limit is ‘unlimited’, value is set to None.

gc3libs.utils.parse_range(spec)

Return minimum, maximum, and stepping value for a range.

Argument spec must be a string of the form LOW:HIGH:STEP, where LOW, HIGH and STEP are (integer or floating-point) numbers. Example:

>>> parse_range('1:10:2')
(1, 10, 2)

>>> parse_range('1.0:3.5:0.5')
(1.0, 3.5, 0.5)

Note that, as soon as any one of LOW, HIGH, STEP is not an integer, all of them are parsed as Python floats:

>>> parse_range('1:3:0.5')
(1.0, 3.0, 0.5)

>>> parse_range('1.0:3:1')
(1.0, 3.0, 1.0)

>>> parse_range('1:3.0:1')
(1.0, 3.0, 1.0)

The final part :STEP can be omitted if the step is 1:

>>> parse_range('2:5')
(2, 5, 1)

>>> parse_range('1.0:3.0')
(1.0, 3.0, 1.0)

Finally, note that parse_range does not perform any kind of check on the validity of the resulting range; so it is possible to parse a string into an empty range or range specification with stepping 0:

>>> parse_range('1:-5:10')
(1, -5, 10)

>>> parse_range('1:2:0')
(1, 2, 0)

As a special case to simplify user interfaces, a single number is accepted as a degenerate range: it will be parsed as a range whose content is just the given number:

>>> parse_range('42')
(42, 43, 1)
gc3libs.utils.prettyprint(D, indent=0, width=0, maxdepth=None, step=4, only_keys=None, output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>, _key_prefix='', _exclude=None)

Print dictionary instance D in a YAML-like format. Each output line consists of:

  • indent spaces,
  • the key name,
  • a colon character :,
  • the associated value.

If the total line length exceeds width, the value is printed on the next line, indented by further step spaces; a value of 0 for width disables this line wrapping.

Optional argument only_keys can be a callable that must return True when called with keys that should be printed, or a list of key names to print.

Dictionary instances appearing as values are processed recursively (up to maxdepth nesting). Each nested instance is printed indented step spaces from the enclosing dictionary.

gc3libs.utils.progressive_number(qty=None, id_filename=None)

Return a positive integer, whose value is guaranteed to be monotonically increasing across different invocations of this function, and also across separate instances of the calling program.

This is accomplished by using a system-wide file which holds the “next available” ID. The location of this file can be set using the GC3PIE_ID_FILE environment variable, or programmatically using the id_filename argument. By default, the “next ID” file is located at ~/.gc3/next_id.txt:file:

Example:

>>> # create "next ID" file in a temporary location
>>> import tempfile, os
>>> (fd, tmp) = tempfile.mkstemp()

>>> n = progressive_number(id_filename=tmp)
>>> m = progressive_number(id_filename=tmp)
>>> m > n
True

If you specify a positive integer as argument, then a list of monotonically increasing numbers is returned. For example:

>>> ls = progressive_number(5, id_filename=tmp)
>>> len(ls)
5
(clean up test environment)
>>> os.remove(tmp)

In other words, progressive_number(N) is equivalent to:

nums = [ progressive_number() for n in range(N) ]

only more efficient, because it has to obtain and release the lock only once.

After every invocation of this function, the last returned number is stored into the file passed as argument id_filename. If the file does not exist, an attempt to create it is made before allocating an id; the method can raise an IOError or OSError if id_filename cannot be opened for writing.

Note: as file-level locking is used to serialize access to the counter file, this function may block (default timeout: 30 seconds) while trying to acquire the lock, or raise a LockTimeout exception if this fails.

Raise:LockTimeout, IOError, OSError
Returns:A positive integer number, monotonically increasing with every call. A list of such numbers if argument qty is a positive integer.
gc3libs.utils.read_contents(path)

Return the whole contents of the file at path as a single string.

Example:

>>> read_contents('/dev/null')
''

>>> import tempfile
>>> (fd, tmpfile) = tempfile.mkstemp()
>>> w = open(tmpfile, 'w')
>>> w.write('hey') and None  # make doctest compatible with Py2 and Py3
>>> w.close()
>>> read_contents(tmpfile)
'hey'

(If you run this test, remember to do cleanup afterwards)

>>> os.remove(tmpfile)
gc3libs.utils.remove(path)

Remove a file, but raise no exception if it does not exist.

gc3libs.utils.safe_repr(obj)

Return a string describing Python object obj.

Avoids calling any Python magic methods, so should be safe to use as a ‘last resort’ in implementation of __str__ and __repr__.

gc3libs.utils.same_docstring_as(referenced_fn)

Function decorator: sets the docstring of the following function to the one of referenced_fn.

Intended usage is for setting docstrings on methods redefined in derived classes, so that they inherit the docstring from the corresponding abstract method in the base class.

gc3libs.utils.samefile(path1, path2)

Like os.path.samefile but return False if either one of the paths does not exist.

gc3libs.utils.sh_quote_safe(arg)

Escape a string for safely passing as argument to a shell command.

Return a single-quoted string that expands to the exact literal contents of text when used as an argument to a shell command. Examples (note that backslashes are doubled because of Python’s string read syntax):

>>> print(sh_quote_safe("arg"))
'arg'
>>> print(sh_quote_safe("'arg'"))
''\''arg'\'''
gc3libs.utils.sh_quote_safe_cmdline(args)

Single-quote a list of strings for passing to the shell as a command. Return string comprised of the quoted arguments, concatenated and separated by spaces.

Examples:

>>> print(sh_quote_safe_cmdline(['sh', '-c', 'echo c(1,2,3)']))
'sh' '-c' 'echo c(1,2,3)'
gc3libs.utils.sh_quote_unsafe(arg)

Double-quote a string for passing as argument to a shell command.

Return a double-quoted string that expands to the contents of text but still allows variable expansion and \-escapes processing by the UNIX shell. Examples (note that backslashes are doubled because of Python’s string read syntax):

>>> print(sh_quote_unsafe("arg"))
"arg"
>>> print(sh_quote_unsafe('"arg"'))
"\"arg\""
>>> print(sh_quote_unsafe(r'"\"arg\""'))
"\"\\\"arg\\\"\""
gc3libs.utils.sh_quote_unsafe_cmdline(args)

Double-quote a list of strings for passing to the shell as a command. Return string comprised of the quoted arguments, concatenated and separated by spaces.

Examples:

>>> print(sh_quote_unsafe_cmdline(['sh', '-c', 'echo $HOME']))
"sh" "-c" "echo $HOME"
gc3libs.utils.string_to_boolean(word)

Convert word to a Python boolean value and return it. The strings true, yes, on, 1 (with any capitalization and any amount of leading and trailing spaces) are recognized as meaning Python True:

>>> string_to_boolean('yes')
True
>>> string_to_boolean('Yes')
True
>>> string_to_boolean('YES')
True
>>> string_to_boolean(' 1 ')
True
>>> string_to_boolean('True')
True
>>> string_to_boolean('on')
True

Any other word is considered as boolean False:

>>> string_to_boolean('no')
False
>>> string_to_boolean('No')
False
>>> string_to_boolean('Nay!')
False
>>> string_to_boolean('woo-hoo')
False

This includes also the empty string and whitespace-only:

>>> string_to_boolean('')
False
>>> string_to_boolean('  ')
False
gc3libs.utils.tempdir(**kwargs)

A context manager for creating and then deleting a temporary directory.

All arguments are passed unchanged to the tempfile.mkdtemp standand library function.

(Original source and credits: http://stackoverflow.com/a/10965572/459543)

gc3libs.utils.to_bytes(s)

Convert string s to an integer number of bytes. Suffixes like ‘KB’, ‘MB’, ‘GB’ (up to ‘YB’), with or without the trailing ‘B’, are allowed and properly accounted for. Case is ignored in suffixes.

Examples:

>>> to_bytes('12')
12
>>> to_bytes('12B')
12
>>> to_bytes('12KB')
12000
>>> to_bytes('1G')
1000000000

Binary units ‘KiB’, ‘MiB’ etc. are also accepted:

>>> to_bytes('1KiB')
1024
>>> to_bytes('1MiB')
1048576
gc3libs.utils.to_str(arg, origin='ascii')

Convert arg to a Python text string.

If arg is already a text string (i.e., a unicode object in Python 2, and a str object in Python 3), then return it unchanged. As an exception, if arg is None, return None (unchanged).

Second argument origin determines the handling of arg when arg is a byte-string:

  • if origin is 'filesystem' then arg is converted to a text string using from_filesystem_bytes() (which see);
  • if origin is 'terminal' then conversion of arg is attempted using from_terminal_bytes() (which see);
  • otherwise, origin is interpreted as an encoding name, and byte string arg is decoded using that encoding’s rules.

If arg is neither a text string nor a byte string, then conversion to string is attempted using Python’s built-in str() function.

gc3libs.utils.touch(path)

Ensure a regular file exists at path.

If the file already exists, its access and modification time are updated.

(This is a very limited and stripped down version of the touch POSIX utility.)

gc3libs.utils.unlock(lock)

Release a previously-acquired lock.

Argument lock should be the return value of a previous gc3libs.utils.lock call.

See also: gc3libs.utils.lock()

gc3libs.utils.update_parameter_in_file(path, var_in, new_val, regex_in)

Updates a parameter value in a parameter file using predefined regular expressions in _loop_regexps.

Parameters:
  • path – Full path to the parameter file.
  • var_in – The variable to modify.
  • new_val – The updated parameter value.
  • regex – Name of the regular expression that describes the format of the parameter file.
gc3libs.utils.write_contents(path, data)

Overwrite the contents of the file at path with the given data. If the file does not exist, it is created.

Example:

>>> import tempfile
>>> (fd, tmpfile) = tempfile.mkstemp()
>>> write_contents(tmpfile, 'big data here') and None # discard return value on Py3
>>> read_contents(tmpfile)
'big data here'

(If you run this test, remember to clean up afterwards)

>>> os.remove(tmpfile)