Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache doesn't sync for in-memory archives across parallel map #210

Open
mmckerns opened this issue Aug 28, 2023 · 0 comments
Open

cache doesn't sync for in-memory archives across parallel map #210

mmckerns opened this issue Aug 28, 2023 · 0 comments

Comments

@mmckerns
Copy link
Member

A mystic.cache fails to cache when used in a parallel map if cached=True, however succeeds if cached=False unless it's a dict_archive. This indicates that the in-memory cache is passed to the child process, but the in-memory cached results are not linked back to the main process. Which, makes sense, but maybe isn't what is expected. Is it reasonable to expect the in-memory cache in the main process to be able to fetch the results from the in-memory caches of the child processes?

Current behavior:

Python 3.8.17 (default, Jun 11 2023, 01:54:00) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mystic.cache as mc
>>> import mystic.models as mm
>>> model = mc.cached(archive=mc.archive.dict_archive('foo'))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dict_archive('foo', {(1, 2, 3): 201.0}, cached=True)
>>> import pathos as pa
>>> pmap = pa.maps.Map(pa.pools.ProcessPool)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dict_archive('foo', {(1, 2, 3): 201.0}, cached=True)
>>> 
>>> model = mc.cached(archive=mc.archive.dict_archive('bar', cached=False))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0}, cached=False)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0}, cached=False)
>>> 
>>> model = mc.cached(archive=mc.archive.dir_archive('baz', cached=False))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dir_archive('baz', {(1, 2, 3): 201.0}, cached=False)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dir_archive('baz', {(7, 8, 9): 470685.0, (1, 2, 3): 201.0, (4, 5, 6): 48225.0}, cached=False)
>>> 
>>> model = mc.cached(archive=mc.archive.dir_archive('zap'))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dir_archive('zap', {(1, 2, 3): 201.0}, cached=True)
>>> pmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dir_archive('zap', {(1, 2, 3): 201.0}, cached=True)
>>> 
>>> mmap = pa.maps.Map()
>>> mmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dir_archive('zap', {(1, 2, 3): 201.0, (4, 5, 6): 48225.0, (7, 8, 9): 470685.0}, cached=True)
>>> 
>>> model = mc.cached(archive=mc.archive.dict_archive('bar', cached=False))(mm.rosen)
>>> model([1,2,3])
201.0
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0}, cached=False)
>>> mmap(model, [[4,5,6],[7,8,9]])
[48225.0, 470685.0]
>>> model.__cache__()
dict_archive({(1, 2, 3): 201.0, (4, 5, 6): 48225.0, (7, 8, 9): 470685.0}, cached=False)
>>> 
@mmckerns mmckerns changed the title cache fails to cache for in-memory archives within parallel map cache doesn't sync for in-memory archives across parallel map Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant