CHAP.TaskManager

Python thread pool, see http://code.activestate.com/recipes/577187-python-thread-pool/ Author: Valentin Kuznetsov <vkuznet [AT] gmail [DOT] com>

Module Contents

Classes

StoppableThread	Thread class with a stop() method. The thread itself has to check regularly for the stopped() condition.
UidSet	UID holder keeps track of uid frequency
Worker	Thread executing worker from a given tasks queue
TaskManager	Task manager class based on thread module which executes assigned tasks concurently. It uses a pool of thread workers, queue of tasks and pid set to monitor jobs execution.

Functions

genkey	Generate a new key-hash for a given query. We use md5 hash for the query and key is just hex representation of this hash.
set_thread_name	Set thread name for given identified
start_new_thread	Wrapper wroung standard thread.strart_new_thread call

API

CHAP.TaskManager.genkey(query)

Generate a new key-hash for a given query. We use md5 hash for the query and key is just hex representation of this hash.

CHAP.TaskManager.set_thread_name(ident, name)

Set thread name for given identified

class CHAP.TaskManager.StoppableThread(target, name, args)

Bases: threading.Thread

Thread class with a stop() method. The thread itself has to check regularly for the stopped() condition.

Initialization

This constructor should always be called with keyword arguments. Arguments are:

group should be None; reserved for future extension when a ThreadGroup class is implemented.

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number.

args is the argument tuple for the target invocation. Defaults to ().

kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.init()) before doing anything else to the thread.

stop()

Set event to stop the thread

stopped()

Return stopped status of the thread

running()

Return running status of the thread

CHAP.TaskManager.start_new_thread(name, func, args, unique=False)

Wrapper wroung standard thread.strart_new_thread call

class CHAP.TaskManager.UidSet

Bases: object

UID holder keeps track of uid frequency

Initialization

add(uid)

Add given uid or increment uid occurence in a set

discard(uid)

Either discard or downgrade uid occurence in a set

__contains__(uid)

Check if uid present in a set

get(uid)

Get value for given uid

class CHAP.TaskManager.Worker(name, taskq, pidq, uidq, logger=None)

Bases: threading.Thread

Thread executing worker from a given tasks queue

Initialization

This constructor should always be called with keyword arguments. Arguments are:

group should be None; reserved for future extension when a ThreadGroup class is implemented.

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number.

args is the argument tuple for the target invocation. Defaults to ().

kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.init()) before doing anything else to the thread.

force_exit()

Force run loop to exit in a hard way

run()

Run thread loop

class CHAP.TaskManager.TaskManager(nworkers=10, name='TaskManager', logger=None)

Bases: object

Task manager class based on thread module which executes assigned tasks concurently. It uses a pool of thread workers, queue of tasks and pid set to monitor jobs execution.

.. doctest::

Use case:
mgr  = TaskManager()
jobs = []
jobs.append(mgr.spawn(func, args))
mgr.joinall(jobs)

Initialization

status()

Return status of task manager queue

nworkers()

Return number of workers associated with this manager

spawn(func, *args, **kwargs)

Spawn new process for given function

remove(pid)

Remove pid and associative process from the queue

is_alive(pid)

Check worker queue if given pid of the process is still running

clear(tasks)

Clear all tasks in a queue. It allows current jobs to run, but will block all new requests till workers event flag is set again

joinall(tasks)

Join all tasks in a queue and quit

quit()

Put None task to all workers and let them quit