Skip to content

Thread workers

Workers run a long task in another thread, so that the GUI stays responsive, and report back with signals (such as started, returned, errored, finished, and, for generators, yielded), which can safely update the GUI from the main thread. Except for new_worker_qthread, they use Qt's QRunnable and global QThreadPool.

The simplest way to use them is the @thread_worker decorator. Calling a decorated function returns a worker, which runs the function when it's started (or at once, if callbacks are given with connect=):

import time

from superqt.utils import thread_worker


@thread_worker
def long_task(n):
    time.sleep(n)
    return n


worker = long_task(2)
worker.returned.connect(print)
worker.start()

If the function is a generator, the worker is a GeneratorWorker: it emits yielded for each value, and can be paused, resumed, and aborted between them. Otherwise, it's a FunctionWorker.

These tools were written for napari, and moved to superqt. The napari guide to threading describes them in more depth.

WorkerBase

WorkerBase(
    func: Callable[_P, _R] | None = None,
    SignalsClass: type[
        WorkerBaseSignals
    ] = WorkerBaseSignals,
)

Base class for creating a Worker that can run in another thread.

Parameters:

  • SignalsClass (type, default: WorkerBaseSignals ) –

    A QObject subclass that contains signals, by default WorkerBaseSignals

Attributes:

  • signals (WorkerBaseSignals) –

    signal emitter object. To allow identify which worker thread emitted signal.

abort_requested

abort_requested: bool

Whether the worker has been requested to stop.

is_running

is_running: bool

Whether the worker has been started.

quit

quit() -> None

Send a request to abort the worker.

Note

It is entirely up to subclasses to honor this method by checking self.abort_requested periodically in their worker.work method, and exiting if True.

run

run() -> None

Start the worker.

The end-user should never need to call this function. But it cannot be made private or renamed, since it is called by Qt.

The order of method calls when starting a worker is:

   calls QThreadPool.globalInstance().start(worker)
   |               triggered by the QThreadPool.start() method
   |               |             called by worker.run
   |               |             |
   V               V             V
   worker.start -> worker.run -> worker.work

This is the function that actually gets called when calling QThreadPool.start(worker). It simply wraps the work() method, and emits a few signals. Subclasses should NOT override this method (except with good reason), and instead should implement work().

work

work() -> Exception | _R

Main method to execute the worker.

The end-user should never need to call this function. But subclasses must implement this method (See GeneratorFunction.work for an example implementation). Minimally, it should check self.abort_requested periodically and exit if True.

Examples:

class MyWorker(WorkerBase):
    def work(self):
        i = 0
        while True:
            if self.abort_requested:
                self.aborted.emit()
                break
            i += 1
            if i > max_iters:
                break
            time.sleep(0.5)

start

start() -> None

Start this worker in a thread and add it to the global threadpool.

The order of method calls when starting a worker is:

   calls QThreadPool.globalInstance().start(worker)
   |               triggered by the QThreadPool.start() method
   |               |             called by worker.run
   |               |             |
   V               V             V
   worker.start -> worker.run -> worker.work

await_workers

await_workers(msecs: int | None = None) -> None

Ask all workers to quit, and wait up to msec for quit.

Attempts to clean up all running workers by calling worker.quit() method. Any workers in the WorkerBase._worker_set set will have this method.

By default, this function will block indefinitely, until worker threads finish. If a timeout is provided, a RuntimeError will be raised if the workers do not gracefully exit in the time requests, but the threads will NOT be killed. It is (currently) left to the user to use their OS to force-quit rogue threads.

Important

If the user does not put any yields in their function, and the function is super long, it will just hang... For instance, there's no graceful way to kill this thread in python:

@thread_worker
def ZZZzzz():
    time.sleep(10000000)

This is why it's always advisable to use a generator that periodically yields for long-running computations in another thread.

See this stack-overflow post for a good discussion on the difficulty of killing a rogue python thread:

Parameters:

  • msecs (int, default: None ) –

    Waits up to msecs milliseconds for all threads to exit and removes all threads from the thread pool. If msecs is None (the default), the timeout is ignored (waits for the last thread to exit).

Raises:

  • RuntimeError –

    If a timeout is provided and workers do not quit successfully within the time allotted.

FunctionWorker

FunctionWorker(func: Callable[_P, _R], *args, **kwargs)

QRunnable with signals that wraps a simple long-running function.

Note

FunctionWorker does not provide a way to stop a very long-running function (e.g. time.sleep(10000)). So whenever possible, it is better to implement your long running function as a generator that yields periodically, and use the GeneratorWorker instead.

Parameters:

  • func (Callable) –

    A function to call in another thread

  • *args –

    will be passed to the function

  • **kwargs –

    will be passed to the function

Raises:

  • TypeError –

    If func is a generator function and not a regular function.

GeneratorWorker

GeneratorWorker(
    func: Callable[_P, Generator[_Y, _S | None, _R]],
    *args,
    SignalsClass: type[
        WorkerBaseSignals
    ] = GeneratorWorkerSignals,
    **kwargs,
)

QRunnable with signals that wraps a long-running generator.

Provides a convenient way to run a generator function in another thread, while allowing 2-way communication between threads, using plain-python generator syntax in the original function.

Parameters:

  • func (callable) –

    The function being run in another thread. May be a generator function.

  • SignalsClass (type, default: GeneratorWorkerSignals ) –

    A QObject subclass that contains signals, by default GeneratorWorkerSignals

  • *args –

    Will be passed to func on instantiation

  • **kwargs –

    Will be passed to func on instantiation

is_paused

is_paused: bool

Whether the worker is currently paused.

work

work() -> _R | Exception | None

Core event loop that calls the original function.

Enters a continual loop, yielding and returning from the original function. Checks for various events (quit, pause, resume, etc...). (To clarify: we are creating a rudimentary event loop here because there IS NO Qt event loop running in the other thread to hook into)

send

send(value: _S)

Send a value into the function (if a generator was used).

toggle_pause

toggle_pause() -> None

Request to pause the worker if playing or resume if paused.

pause

pause() -> None

Request to pause the worker.

resume

resume() -> None

Send a request to resume the worker.

Convenience functions

thread_worker

thread_worker(
    function: Callable[_P, Generator[_Y, _S, _R]],
    start_thread: bool | None = None,
    connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    worker_class: type[WorkerBase] | None = None,
    ignore_errors: bool = False,
) -> Callable[_P, GeneratorWorker[_Y, _S, _R]]
thread_worker(
    function: Callable[_P, _R],
    start_thread: bool | None = None,
    connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    worker_class: type[WorkerBase] | None = None,
    ignore_errors: bool = False,
) -> Callable[_P, FunctionWorker[_R]]
thread_worker(
    function: Literal[None] = None,
    start_thread: bool | None = None,
    connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    worker_class: type[WorkerBase] | None = None,
    ignore_errors: bool = False,
) -> Callable[
    [Callable],
    Callable[_P, FunctionWorker | GeneratorWorker],
]
thread_worker(
    function: Callable | None = None,
    start_thread: bool | None = None,
    connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    worker_class: type[WorkerBase] | None = None,
    ignore_errors: bool = False,
)

Decorator that runs a function in a separate thread when called.

When called, the decorated function returns a WorkerBase. See create_worker for additional keyword arguments that can be used when calling the function.

The returned worker will have these signals:

  • started: emitted when the work is started
  • finished: emitted when the work is finished
  • returned: emitted with return value
  • errored: emitted with error object on Exception

It will also have a worker.start() method that can be used to start execution of the function in another thread. (useful if you need to connect callbacks to signals prior to execution)

If the decorated function is a generator, the returned worker will also provide these signals:

  • yielded: emitted with yielded values
  • paused: emitted when a running job has successfully paused
  • resumed: emitted when a paused job has successfully resumed
  • aborted: emitted when a running job is successfully aborted

And these methods:

  • quit: ask the thread to quit
  • toggle_paused: toggle the running state of the thread.
  • send: send a value into the generator. (This requires that your decorator function uses the value = yield syntax)

Parameters:

  • function (callable, default: None ) –

    Function to call in another thread. For communication between threads may be a generator function.

  • start_thread (bool, default: None ) –

    Whether to immediaetly start the thread. If False, the returned worker must be manually started with worker.start(). by default it will be False if the _connect argument is None, otherwise True.

  • connect (Dict[str, Union[Callable, Sequence]], default: None ) –

    A mapping of "signal_name" -> callable or list of callable: callback functions to connect to the various signals offered by the worker class. by default None

  • worker_class (Type[WorkerBase], default: None ) –

    The WorkerBase to instantiate, by default FunctionWorker will be used if func is a regular function, and GeneratorWorker will be used if it is a generator.

  • ignore_errors (bool, default: False ) –

    If False (the default), errors raised in the other thread will be reraised in the main thread (makes debugging significantly easier).

Returns:

  • callable –

    function that creates a worker, puts it in a new thread and returns the worker instance.

Examples:

@thread_worker
def long_function(start, end):
    # do work, periodically yielding
    i = start
    while i <= end:
        time.sleep(0.1)
        yield i

    # do teardown
    return "anything"


# call the function to start running in another thread.
worker = long_function()

# connect signals here if desired... or they may be added using the
# `connect` argument in the `@thread_worker` decorator... in which
# case the worker will start immediately when long_function() is called
worker.start()

create_worker

create_worker(
    func: Callable[_P, Generator[_Y, _S, _R]],
    *args,
    _start_thread: bool | None = None,
    _connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    _worker_class: type[GeneratorWorker]
    | type[FunctionWorker]
    | None = None,
    _ignore_errors: bool = False,
    **kwargs,
) -> GeneratorWorker[_Y, _S, _R]
create_worker(
    func: Callable[_P, _R],
    *args,
    _start_thread: bool | None = None,
    _connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    _worker_class: type[GeneratorWorker]
    | type[FunctionWorker]
    | None = None,
    _ignore_errors: bool = False,
    **kwargs,
) -> FunctionWorker[_R]
create_worker(
    func: Callable,
    *args,
    _start_thread: bool | None = None,
    _connect: dict[str, Callable | Sequence[Callable]]
    | None = None,
    _worker_class: type[GeneratorWorker]
    | type[FunctionWorker]
    | None = None,
    _ignore_errors: bool = False,
    **kwargs,
) -> FunctionWorker | GeneratorWorker

Convenience function to start a function in another thread.

By default, uses FunctionWorker for functions and GeneratorWorker for generators, but a custom WorkerBase subclass may be provided. If so, it must be a subclass of WorkerBase, which defines a standard set of signals and a run method.

Parameters:

  • func (Callable) –

    The function to call in another thread.

  • _start_thread (bool, default: None ) –

    Whether to immediaetly start the thread. If False, the returned worker must be manually started with worker.start(). by default it will be False if the _connect argument is None, otherwise True.

  • _connect (Dict[str, Union[Callable, Sequence]], default: None ) –

    A mapping of "signal_name" -> callable or list of callable: callback functions to connect to the various signals offered by the worker class. by default None

  • _worker_class (type of `GeneratorWorker` or `FunctionWorker`, default: None ) –

    The WorkerBase to instantiate, by default FunctionWorker will be used if func is a regular function, and GeneratorWorker will be used if it is a generator.

  • _ignore_errors (bool, default: False ) –

    If False (the default), errors raised in the other thread will be reraised in the main thread (makes debugging significantly easier).

  • *args –

    will be passed to func

  • **kwargs –

    will be passed to func

Returns:

  • worker ( WorkerBase ) –

    An instantiated worker. If _start_thread was False, the worker will have a .start() method that can be used to start the thread.

Raises:

  • TypeError –

    If a worker_class is provided that is not a subclass of WorkerBase.

  • TypeError –

    If _connect is provided and is not a dict of {str: callable}

Examples:

def long_function(duration):
    import time

    time.sleep(duration)


worker = create_worker(long_function, 10)

new_worker_qthread

new_worker_qthread(
    Worker: type[WorkerProtocol],
    *args,
    _start_thread: bool = False,
    _connect: dict[str, Callable] | None = None,
    **kwargs,
)

Convenience function to start a worker in a QThread.

thread, not as the actual code or object that runs in that thread. The QThread object is created on the main thread and lives there.

Worker objects which derive from QObject are the things that actually do the work. They can be moved to a QThread as is done here.

Mostly ignorable detail

While the signals/slots syntax of the worker looks very similar to standard "single-threaded" signals & slots, note that inter-thread signals and slots (automatically) use an event-based QueuedConnection, while intra-thread signals use a DirectConnection. See Signals and Slots Across Threads

Parameters:

  • Worker (QObject) –

    QObject type that implements a work() method. The Worker should also emit a finished signal when the work is done.

  • _start_thread (bool, default: False ) –

    If True, thread will be started immediately, otherwise, thread must be manually started with thread.start().

  • _connect (dict, default: None ) –

    Optional dictionary of {signal: function} to connect to the new worker. for instance: _connect = {'incremented': myfunc} will result in: worker.incremented.connect(myfunc)

  • *args –

    will be passed to the Worker class on instantiation.

  • **kwargs –

    will be passed to the Worker class on instantiation.

Returns:

  • worker ( WorkerBase ) –

    The created worker.

  • thread ( QThread ) –

    The thread on which the worker is running.

Examples:

Create some QObject that has a long-running work method:

class Worker(QObject):
    finished = Signal()
    increment = Signal(int)

    def __init__(self, argument):
        super().__init__()
        self.argument = argument

    @Slot()
    def work(self):
        # some long running task...
        import time

        for i in range(10):
            time.sleep(1)
            self.increment.emit(i)
        self.finished.emit()


worker, thread = new_worker_qthread(
    Worker,
    "argument",
    _start_thread=True,
    _connect={"increment": print},
)