- class yaook.op.tasks.ExponentialBackOff(base: int, start: float, max_: int)
Iterable to represent the sleep intervals used for an exponential back-off.
base – The factor by which the back-off value is multiplied on each iteration.
start – The initial value.
max – The maximum value for the back-off.
This object is iterable. Iterating it will yield values starting at start, increasing by a factor of base until they reach max_. At that point, only max_ is returned indefinitely.
The iterator can be reset to its original value by calling
- class yaook.op.tasks.TaskQueue(*, backoff_base: int = 2, backoff_start: float = 0.5, backoff_max: int = 120, jitter: float = 0.1, logger: Optional[logging.Logger] = None)
De-duplicating and re-trying queue of tasks.
backoff_base – The base of the exponential back-off for failed tasks.
backoff_start – The start value of the exponential back-off for failed tasks.
backoff_max – The cutoff value for the exponential back-off for failed tasks.
jitter – The jitter ratio to use for rescheduling failed tasks.
logger – The logger to use for logging information about the queue itself.
The queue does not do anything unless
Task are initially run in the order they are enqueued.
If a task is run and fails, it will be rescheduled for a later attempt, without blocking other enqueued tasks. Hence, the order of the tasks can not be used to enforce dependencies between tasks. Repeatedly failing tasks will be rescheduled with exponential back-off, but never removed from the queue completely.
Tasks are rescheduled not in a strict exponential back-off, but are slightly jittered based on the jitter argument. This is to spread out the load on consumed resources even more if several tasks fail at the same time.
- async run() None
Run the queue, forever.
- class yaook.op.tasks.RestartingTask(coroutine_function: Callable, logger: Optional[logging.Logger] = None, loop: Optional[asyncio.events.AbstractEventLoop] = None)
Background task which gets restarted with exponential back-off if it crashes.
coroutine_function – The coroutine function implementing the task.
logger – A logger used to log internals of the object.
loop – An optional asyncio event loop to use.
If the coroutine raises an exception, it will be restarted with an exponential back-off (up to 120 seconds). If the coroutine exits without error or with a cancellation, it will not be restarted automatically.
stop()is called and the task is currently running, it will be cancelled (and not automatically restarted).
- start() None
Start the task to run it in the background.
This resets the internal exponential back-off for crashes and starts the task immediately, if it is not runnnig. If the task is already running, this method only resets the back-off.
- restart() None
Stop and start the task.
If the task has been stopped before, this is equivalent to calling
- stop() None
Stop the task.
If the task is not running, this does nothing. Otherwise, the task is cancelled.
- async wait_for_termination() None
Stop the task and wait for it to terminate.
Exceptions from the task are not propagated but are logged as usual.