gc3libs.poller

This module implements “pollers”. A “Poller” is an object that monitors a given URL and returns events whenever a new object is created inside that URL.

class gc3libs.poller.FilePoller(url, recurse=False, **kw)

Track events on the filesystem using Python’s standard os module.

Params recurse:When True, automatically track events in any (already-existing or newly-created) subdirectory.

Warning

In order to issue ‘modified’ events, this class relies on checking an inode’s st_mtime field, which only provides 1-second resolution. Modification events that happen too close will not be told apart as distinct; in particular, modifying a file less than 1s after creating it will not be detected.

This implementation is only used to track Url with file schema whenever the inotify_simple module is not available.

get_new_events()

Iterate over events that happened since last call to this method.

Returns a list of tuples (subject, event).

A subject is a unique identifier for a watched “thing”: the exact form and type depends on the actual concrete class; pollers that watch the filesystem or HTTP-accessible resources will use a URL (gc3libs.url.Url) as a subject instance, but e.g. pollers that watch a database table might use a row ID instead.

The associated event is one or more of the following strings:

  • created: the subject has been created since the last call to get_new_events();
  • modified: the subject has changed since the last call to get_new_events();
  • deleted: the subject has been deleted since the last call to get_new_events();

Depending on the concrete poller class, some events might never occur, or cannot be detected. Most notably, only filesystem-watching pollers might be able to generate meaningful modified events.

recurse

Whether the poller is watching the entire directory tree pointed to by self.url, or only the directory at its top level.

class gc3libs.poller.INotifyPoller(url, recurse=False, **kw)

Use Linux’ inofity to track new events on the specified filesystem location.

Params recurse:When True, automatically track events in any (already-existing or newly-created) subdirectory.

This poller is used by default when the inotify_simple Python package is available and the URL has a file schema.

Warning

On Linux, the maximum number of inotify descriptors that a user can open is limited by the kernel parameters:

  • fs.inotify.max_user_instances
  • fs.inotify.max_user_watches
  • fs.inotify.max_queued_events

See also the inotify(7) manpage

get_new_events()

Iterate over events that happened since last call to this method.

Returns a list of tuples (subject, event).

A subject is a unique identifier for a watched “thing”: the exact form and type depends on the actual concrete class; pollers that watch the filesystem or HTTP-accessible resources will use a URL (gc3libs.url.Url) as a subject instance, but e.g. pollers that watch a database table might use a row ID instead.

The associated event is one or more of the following strings:

  • created: the subject has been created since the last call to get_new_events();
  • modified: the subject has changed since the last call to get_new_events();
  • deleted: the subject has been deleted since the last call to get_new_events();

Depending on the concrete poller class, some events might never occur, or cannot be detected. Most notably, only filesystem-watching pollers might be able to generate meaningful modified events.

recurse

Whether the poller is watching the entire directory tree pointed to by self.url, or only the directory at its top level.

class gc3libs.poller.Poller(url, **kw)

Abstract class for an URL Poller.

A Poller is a class that tracks new events on a specific Url. When calling the get_events() it will return a list of tuples (Url, mask) containing the events occurred for each one of the underlying URLs.

get_new_events()

Iterate over events that happened since last call to this method.

Returns a list of tuples (subject, event).

A subject is a unique identifier for a watched “thing”: the exact form and type depends on the actual concrete class; pollers that watch the filesystem or HTTP-accessible resources will use a URL (gc3libs.url.Url) as a subject instance, but e.g. pollers that watch a database table might use a row ID instead.

The associated event is one or more of the following strings:

  • created: the subject has been created since the last call to get_new_events();
  • modified: the subject has changed since the last call to get_new_events();
  • deleted: the subject has been deleted since the last call to get_new_events();

Depending on the concrete poller class, some events might never occur, or cannot be detected. Most notably, only filesystem-watching pollers might be able to generate meaningful modified events.

class gc3libs.poller.SwiftPoller(url, **kw)

Periodically check a SWIFT bucket and trigger events when new objects are created.

Right now, a valid URL can be one of the following form:

  • If the keystone endpoint is reachable via HTTP, either one of:
    • swift://<user>+<tenant>:<password>@<keystone-url>?container
    • swt://<user>+<tenant>:<password>@<keystone-url>?container
  • If the keystone endpoint is reachable via HTTPS, either one of:
    • swifts://<user>+<tenant>:<password>@<keystone-url>?container
    • swts://<user>+<tenant>:<password>@<keystone-url>?container

We assume that keystone auth version 2 is used.

get_new_events()

Iterate over events that happened since last call to this method.

Returns a list of tuples (subject, event).

A subject is a unique identifier for a watched “thing”: the exact form and type depends on the actual concrete class; pollers that watch the filesystem or HTTP-accessible resources will use a URL (gc3libs.url.Url) as a subject instance, but e.g. pollers that watch a database table might use a row ID instead.

The associated event is one or more of the following strings:

  • created: the subject has been created since the last call to get_new_events();
  • modified: the subject has changed since the last call to get_new_events();
  • deleted: the subject has been deleted since the last call to get_new_events();

Depending on the concrete poller class, some events might never occur, or cannot be detected. Most notably, only filesystem-watching pollers might be able to generate meaningful modified events.

gc3libs.poller.make_poller(url, **extra)

Factory method that returns the registered poller for the specified gc3libs.url.Url.