Does there exist a synchronous executor for client.submit?

[aa1371]

Does there exist an synchronous executor for tasks along these lines?

class DummyFuture(Future):
    def __init__(res=None, exc=None):
        self.res = res
        self.exc = exc

    def result(self):
        if self.exc:
            raise self.exc
        return self.res


def dummy_submit(func, *args, **kwargs):
    try:
        res = func(*args, **kwargs)
    except Exception as e:
        return DummyFuture(exc=e)
    else:
        return DummyFuture(res)


client = Client(synchronous=True)  # effectively replaces Client.submit with dummy_submit above

Haven't been able to find anything like this so far, but would be extremely helpful for debugging. The closest thing I have found is an example that requires using a tornado dummy executor, posted by @mrocklin in the SO answer below, however that solution requires quite a bit of overhead/boilerplate code.
https://stackoverflow.com/a/44193980/5428668

[fjetter]

It depends on what you are actually looking for.

There is no single-threaded, synchronous distributed implementation. Distributed itself is by definition built of multiple entities and is managing how these work together (e.g. worker and scheduler). Recreating the logic performed by this complex system in a single-threaded machine would be, maybe counter-intuitively, not easy, many feature may even be impossible. At least a partial answer to why this is can be found in the SO comment https://stackoverflow.com/questions/44193979/how-do-i-run-a-dask-distributed-cluster-in-a-single-thread/44193980#comment91406782_44193980
Imho, if you are relying on any distributed features, you should go for the LocalCluster, possibly with only one worker if concurrency is an issue. If you want to inspect the cluster itself (scheduler, worker) for debugging, I can recommend running the cluster asynchronously as we are doing in many of our tests (I know this is the opposite of what you were asking for but it is quite useful depending on your usecase :) ).

If you are only relying on executing futures/tasks, there is a way in dask itself, see https://docs.dask.org/en/latest/scheduling.html#single-thread
The easiest way for this is probably to use dask.delayed instead of plain futures. Delayed is a more high level, arguably easier to use interface where you can simply call dask.compute(collection, scheduler="sync") without need of a distributed cluster.
If you want to peak behind the curtain, what happens if you call dask.compute is that we call get_sync, see https://github.com/dask/dask/blob/23c93d72c8fbd194b44614c29dd584ff93fc1eef/dask/local.py#L539-L566
which runs your entire graph sequentially in a single thread. Notice, that there is no client. If you are relying on a client to be there, you should probably go for a LocalCluster instead

[aa1371]

Thanks for the insight. I think the single worker LocalCluster in conjunction with client.recreate_error_locally, would likely be just as convenient for debugging issues.