trigger a warning when trying to append to a file on google cloud storage

[Courvoisier13]

When trying to append to a file on GCS using pandas, the file was being overwritten. That is because GCS files are immutable. But there was no warning and it took a while to debug.

The mode = 'a' in pandas.to_csv is passed on to fsspec.open. It would be nice if we have a warning or error in this case.
See pandas-dev/pandas#51821

import pandas as pd
df = pd.DataFrame({
    'account-start': ['2017-02-03', '2017-03-03', '2017-01-01'],
    'client': ['Alice Anders', 'Bob Baker', 'Charlie Chaplin'],
    'balance': [-1432.32, 10.43, 30000.00],
    'db-id': [1234, 2424, 251],
    'proxy-id': [525, 1525, 2542],
    'rank': [52, 525, 32],
    ...
})
header = True
to_csv_mode = 'w'
with pd.read_csv(gs_path, chunksize=1) as reader:
    for r in reader:
        r.to_csv(temp_gs_path, index=False, header=header, mode=to_csv_mode)
        header = False
        to_csv_mode = 'a'

[martindurant]

(transferred to gcsfs)

[martindurant]

Append in GCS is possible, via compose (currently used by the merge() method), where we upload the extra data to some scratch file, and when close/committing, do a merge to rewrite the final destination. S3fs does something similar, except that in that case there is no need to write the new data to an actual key which then needs to be removed when done.

@Courvoisier13 , are you interested in working on this?

[Courvoisier13]

I've built indeed my own solution using compose. I'm a bit swamped right now. Maybe I can come back when i have more time.

[martindurant]

It would be very helpful!