Feat/store paths #2272
Feat/store paths #2272
Conversation
…nto feat/store-paths
…nto feat/store-paths
|
Edit: whoops, never mind. I missed that this only affects |
| @@ -320,6 +336,13 @@ async def _get_many( | |||
| for req in requests: | |||
| yield (req[0], await self.get(*req)) | |||
|
|
|||
| def with_path(self, path: str) -> Self: | |||
There was a problem hiding this comment.
| def with_path(self, path: str) -> Self: | |
| @abstractmethod | |
| def with_path(self, path: str) -> Self: |
Or do you think think we can do this in a generic way that applies to all stores?
There was a problem hiding this comment.
I don't think we can make it generic unfortunately, because each store class might have its own extra attributes. maybe we could use __getstate__ to get around this?
| if path.is_dir(): # TODO: support deleting directories? shutil.rmtree? | ||
| shutil.rmtree(path) | ||
| else: | ||
| await to_thread(path.unlink, True) # Q: we may want to raise if path is missing | ||
|
|
||
| async def exists(self, key: str) -> bool: | ||
| path = self.root / key | ||
| path = Path(self.path) / key |
There was a problem hiding this comment.
You do this a lot. I wonder if the store could have a private attr like self._ppath = Path(self.path) that does this once.
There was a problem hiding this comment.
I had a similar thought when I found myself doing this over and over again 😅
| @@ -28,12 +29,15 @@ def __init__( | |||
| self, | |||
| store_dict: MutableMapping[str, Buffer] | None = None, | |||
| *, | |||
| path: str = "", | |||
There was a problem hiding this comment.
this store highlights the fact that not all stores have a meaningful path.
There was a problem hiding this comment.
I'm not sure I agree -- if store_dict really is an arbitrary mutable mapping, there's no guarantee that we want all IO operations to be scoped to the top-level key space, and that's exactly what the path attribute is for.
| file_path: Path | str, | ||
| *, | ||
| path: str = "", |
There was a problem hiding this comment.
This feels quite confusing. IIUC, you are allowing the caller to specify a path within the zipfile as opposed to putting the root of the store at the root of the zip, is that right?
There was a problem hiding this comment.
exactly! this is required to make ZipStore congruent to RemoteStore
There was a problem hiding this comment.
though this may have some impact (simplifying) zarr-developers/zarr-specs#311
Adds a
pathattribute to all the store classes. Key operations likeget,set,list,list_prefixetc are now scoped to the subtree rooted by thepathattribute of the store, which is howRemoteStoreworks in the v3 branch. This change is motivated by the desire for a consistent, scalable store API.Key changes:
LocalStore.roottoLocalStore.path, and changed the type of the attribute frompathlib.Pathtostrpathattribute toMemoryStorepathattribute ofZipStore, and introduced a new attributefile_paththat does whatZipStore.pathused to do (point to the location of the Zipfile in the parent file system).TODO: