Unable to list files with service account auth for GDriveFilesystem

[chandrasekharan-zipstack]

Hello, I'm using the latest version of the library PyDrive2[fsspec]==1.17.0 and I'm trying to authenticate a service account (the SA email is shared with the drive I wish to test and the keys were generated, I doubt if this order matters).

I followed the steps provided in the docs for the jsonkey file as a string. Its authenticated without errors however I'm unable to list files from the shared drive

1. GDrive API is hit
   
2. Listing of files with GDriveFilesystem returns empty (screenshot while debugging)
   

Digging further, I followed steps mentioned in this comment and was able to list files with the GDrive object however. In fact I noticed the GDrive object being created here and the files get listed (while debugging) when I used

file_list = self.client.ListFile().GetList()

Code snippet:

from oauth2client.service_account import ServiceAccountCredentials
from pydrive2.auth import GoogleAuth
from pydrive2.fs import GDriveFileSystem
from pydrive2.drive import GoogleDrive

def _init_with_service_key(self, service_key_settings: str) -> GDriveFileSystem:
        """Creates a GDriveFileSystem with a service account key."""
        # Attempt from issue #21 
        # scope = ["https://www.googleapis.com/auth/drive"]
        # gauth = GoogleAuth()
        # gauth.auth_method = 'service'
        # gauth.credentials = ServiceAccountCredentials.from_json_keyfile_dict(service_key_settings, scopes=scope)
        # drive = GoogleDrive(gauth)

        # about = drive.GetAbout()
        # file_list = drive.ListFile().GetList()
        # gdrive = GDriveFileSystem(
        #     path="root", 
        #     google_auth=gauth,
        #     use_service_account=True
        # )
        
        gdrive = GDriveFileSystem(
            path="root",
            use_service_account=True,
            client_json=service_key_settings,
            skip_instance_cache=True,
            use_listings_cache=False
        )
        return gdrive

Any ideas on how I can proceed? Thanks in advance! :)

[shcheklein]

@chandrasekharan-zipstack it might be a bug in the fsspec implementation. It was done to address needs of DVC as product in the first place, and I think is has a lot of edge cases that are not solved to be used in general. I would recommend to use the older API for now.

[chandrasekharan-zipstack]

I would recommend to use the older API for now.

@shcheklein thanks for the information, I did go through the docs a bit but can you elaborate on what you mean be the older API here? Is it related to using the GoogleDrive object mentioned here?

Also it might be a good idea to mention such limitations/edge cases in the docs

[shcheklein]

Yes, I meant this for example: https://docs.iterative.ai/PyDrive2/filelist/