Change remote globally in Git history

[dmpetrov]

Changing buckets and cloud easily for a single project is a very compelling feature.

But in fact, when I need to transfer a data-remote from one bucket to another (or another cloud) I can do that properly only on HEAD of the repo. All the old commit will have old remote (in .dvc/config). As a result, when I checkout back in Git history an old remote will be used.

So, I need to keep old data remote (bucket) or I'll have troubles using my old Git commits.

Is it possible to make remote settings "global"? A single remote change should change it everywhere in the Git history. How Git does that and can it work for DVC? Are there other options?

All ideas are welcome!

[efiop]

@dmpetrov This is mostly about properly maintaining your project. E.g. using tags or branches for released versions, which can be easilly updated to point to the new bucket.

[efiop]

E.g. say some project died and closed its remote, but someone has the cache somwhere. In that case, you fork it and put your own remote and rely on it. Or, you could specify -r remote to specific commands, like import/get(which we don't have support for right now, but we do have a ticket opened).

[dmpetrov]

@efiop could you please elaborate? How should the team properly maintain the project to minimize this amount of work?

Use case: A team has a long-running project (year+), hundreds of commits, dozens of releases (with tags). All Git history is important. Some clients might still use old ML models, all changes in data sources have to be fully tracked from the inception. Not released commits (without tags) might be also important (we decided to release an old but a promising experiment).

Suddenly the team decided to change a cloud provider. It looks like a tremendous amount of work has to be done to make it happen. It would be great if DVC can make it in a few commands.

[efiop]

Oops, sorry for the delay.

Not released commits (without tags) might be also important (we decided to release an old but a promising experiment).

Well, giving that to your customers was a bad idea from beginning 🙂

So say you have some tag like v1 that was using s3. Now you are migrating to gs, so what you do is you go to v1, create a branch from it, adjust the remote to point to the new gs location, commit, move v1 to this new commit. Then you'll have to make your users update.

But if you, as a maintainer, would take it more seriously earlier, you would create some kind of proxy remote. E.g. http remote (like we do in dvc core project) that you will be able to trivially switch from s3 to gs without needing to adjust anything in the projects themselves.

[dmpetrov]

@efiop these are good workarounds. I urge everyone to think about a holistic solution instead.

Ideally, we need the same solution as Git has - I can easily move a repo with the entire history from, for example, GitHub to GitLab by a couple of commands just by changing remote, pushing and removing the old one. Is there a way to implement something similar in DVC?

[ghost]

@dmpetrov , @efiop , what about using external tools like the AWS CLI or the Google Cloud Platform CLI to sync the cache?

For example, if you are migrating to GCP, it would be gsutil cp -r .dvc/cache gs://my-bucket

[ghost]

If you don't care about modifying your Git history, git rebase -i -root and modifying the commit who added the remote, pointing to the new one.

[dmpetrov]

@MrOutis sure, you can use the tools. But it is not clear how to change the links in Git history - you will still have .dvc/config files in history pointing to the old repo. This is the major question.

Modifying the commit - yes, it is a possibility. Is there any better way to define and change data remotes? (most likely it should not be committed to config)

[shcheklein]

👍 on my end, it feels that there should be a better solution to this.

[efiop]

Ideally, we need the same solution as Git has - I can easily move a repo with the entire history from, for example, GitHub to GitLab by a couple of commands just by changing remote, pushing and removing the old one. Is there a way to implement something similar in DVC?

Sure:

dvc pull (maybe with --all-commits or --all-tags or --all-branches)
dvc remote add gs gs://bucket/path
dvc push -r gs (maybe with --all-commits or --all-tags or --all-branches)
dvc remote remove s3
dvc remote default gs
git add .dvc/config
git commit
git push

Maybe I don't quite understand how you propose to "modify git history". The workarounds I've provided initially update the branches/tags that are used or propose to use some type of proxy to route your pulls. Currently I can't think of any other way to achieve this.

[shcheklein]

How about we always rely on the latest commit in a branch to determine the actual remote? No matter what is committed in the history.

[efiop]

@shcheklein sounds fragile and non-obvious. Plus it again won't work until the user dvc updates or something, to update the git repo cache. You might also want to move to another remote, while dropping the old one, which would also break this.

[dmpetrov]

@efiop The code you provided looks like another workaround, not a holistic solution for changing remotes "globally". The problem - if a user checks out an old revision (clones or imports v2 tag for example) it will point to an outdated remote.

@shcheklein thank you! It is definitely a global solution that might work. There are some issues with this approach (thanks @efiop to pointing to this) but at least we have something to consider or/and improve.

[Suor]

It looks like you simply want to rewrite the history for config file.

EDIT. Another option is completely separating config from git history, which we already support in the form of global/user/local configs.

[dimitry12]

But if you, as a maintainer, would take it more seriously earlier, you would create some kind of proxy remote. E.g. http remote (like we do in dvc core project) that you will be able to trivially switch from s3 to gs without needing to adjust anything in the projects themselves.

@efiop can you please elaborate or point to the details of proxy-remote implementation?

Also to summarize possible solutions I see in the thread as I am facing the same inconvenience (I want to import older model from a github-based registry using --rev parameter; and at the time of that old revision registry used a now gone dvc-remote):

• Rewrite git history by editing .dvc/config in all commits so that it contains the currently active dvc-remote. That breaks many things outside of the dvc as commit hashes change.
• (yet unsupported?) --remote parameter for dvc import which would use one of the dvc-remotes specified inmaster/main's .dvc/config?
• (unsupported for good) When using dvc import automatically use the default remote from master/main's .dvc/config instead of that revision's .dvc/config. I agree that's fragile and breaking change, and may not work for the teams who don't dvc push --all-commits
• Using proxy/alias for dvc-remote (see the first part of this comment). Seems like a nice solution.
• Use global .dvc/config when dvc is in the role of data-registry. How would that work if I want to use different default dvc-remotes for different registries?

[dberenbaum]

* Use global `.dvc/config` when `dvc` is in the role of data-registry. How would that work if I want to use different default dvc-remotes for different registries?

Hi @dimitry12, for this option, would dvc config --local (.dvc/config.local) work? It would give you a local config that's ignored by Git.

[Suor]

@dberenbaum yes local config will also work, as long as you manage it properly, i.e. update it over all your working copies.

[dberenbaum]

Thanks, @Suor! I'm wondering if this issue can be closed then since it seems that the introduction of the local config makes changing remotes globally possible and on par with Git.

[dimitry12]

Local config works for me.

[shcheklein]

@dberenbaum my 2cs on this: local config can be a good temporary solution, but it breaks a bit the point of repos being self descriptive/self contained. Why it can be important? For two reasons (and may be I'm missing something else):

1. Asking everyone to do local config means an additional step people will need to remember to do. Not the biggest problem, but makes everything a bit more fragile.
2. It complicates automation - CI (e.g. CML), Viewer - those environment where we are accessing repos from code. Local config is a possibility but will require additional steps from users. In the Viewer it'll require additional UI/UX to manage this.

[dberenbaum]

See discussion at https://discord.com/channels/485586884165107732/563406153334128681/818978209944961045 and https://discord.com/channels/485586884165107732/563406153334128681/819212764282748928

[dberenbaum]

Another argument for why local config is insufficient: for data registry repos where the data is being fetched from outside the project via get/import, this won't work. See https://discord.com/channels/485586884165107732/563406153334128681/826365388400754728.

[adamliter]

Is there a good solution for this? Does the -r option of dvc fetch/dvc pull/dvc push work as intended?

If I have a series of commits with one remote, but then switch to another remote, I thought the following would work to transfer everything to the new remote:

git checkout <COMMIT_WITH_OLD_REMOTE>
dvc fetch -A
git checkout <COMMIT_WITH_NEW_REMOTE>
dvc push -A -r new-remote

It does not work as I would've expected. Instead, I get this:

WARNING: failed to collect '27ac1c6a4340160a2021d924894219e7450f5a4b', skipping
WARNING: failed to collect '748ef9610c82e01de8b8af536fc816a37c4994fb', skipping
WARNING: failed to collect '8df24cec941f303c2ad49c2c85e91c3a93b007df', skipping
[...]

[adamliter]

Yes, it succeeds, and no it doesn't show the same warnings. But that's expected, right?

If I understand correctly, DVC is fetching from whatever the remote was in .dvc/config at the time the hash was committed into the repo. For example, commit A has S3 as a remote, and dvc.lock has model.pkl with hash abc123 at commit A. Then commit B introduces a new version of model.pkl with hash def456. Commit C changes .dvc/config to use GS instead of S3. Because dvc.lock still has model.pkl with hash def456 in commit C, I think you can dvc push and it'll go to the new remote. But if I dvc fetch -A it seems it fetches from whatever remote was in .dvc/config at the time, so doing dvc fetch -A succeeds, but if's fetching from the old S3 remote (if I didn't already have it cached locally).

What I'm trying to achieve is also sending model.pkl with hash abc123 to the new remote introduced in commit C.

So I was expecting that git checkout <COMMIT_B> would revive .dvc/config with the S3 remote. Then I could dvc fetch -A to get everything from that S3 remote in case my local cache doesn't have everything. Then I could git checkout <COMMIT_C> to get the .dvc/config with the new remote. Then I could dvc push -A to send everything to the new GS remote. However, that doesn't seem to work, and neither does dvc push -A -r <NEW_GS_REMOTE>.

Is there a viable workaround for this, other than rewriting the git history?

[dberenbaum]

Yes, it succeeds, and no it doesn't show the same warnings. But that's expected, right?

Yes. I was asking because those warnings seem like they come from fetching rather than pushing, so I'm not sure why they show only while pushing.

Have you tried the suggestion above to override via your local config? That is the simplest way to push everything to a new remote. For example:

$ dvc remote add -d --local new gs://...
$ dvc push -A

[adamliter]

FWIW, here is a way to reproduce those warnings (done with dvc==3.50.2).

cd /tmp
mkdir -p dvc-test
mkdir -p dvcstore1
mkdir -p dvcstore2
cd dvc-test
git init
dvc init
dvc remote add -d myremote1 /tmp/dvcstore1
touch model.pkl
dvc add model.pkl
git add .
git commit -m "my first model"
dvc push
echo "2" >> model.pkl
dvc add model.pkl
git add .
git commit -m "my second model"
dvc push
find ../dvcstore1/files/md5 -type f | wc -l # this returns 2
dvc remote remove myremote1
dvc remote add -d myremote2 /tmp/dvcstore2
dvc push -A # this just pushes the second version of the model only
find ../dvcstore2/files/md5 -type f | wc -l # this returns 1
dvc push -A -r myremote2 # this returns the following
# WARNING: failed to collect '617a3042e91548b0bf7b905e550789180dd4d25e', skipping
# WARNING: failed to collect '0c7d92d110ed68424d43a72d6de5e37e4104a3d4', skipping
# Collecting                                                                                                                   |1.00 [00:00,  324entry/s]
# Pushing
# Everything is up to date.
find ../dvcstore2/files/md5 -type f | wc -l # this *still* returns 1

It looks like the local suggestion does work, at least with my minimal reproduction above. For some reason when I was trying this out the other day, I didn't think it was working:

dvc remote add -d --local myremote2 /tmp/dvcstore2
dvc push -A
find ../dvcstore2/files/md5 -type f | wc -l # this *now* returns 2

This could maybe work, but it does mean continuing to use a .dvc/config.local file if the old remote goes away, something everyone on the team would have to do. Continuing with the minimal example:

rm .dvc/config.local
rm -rf .dvc/cache
rm -rf /tmp/dvcstore1
dvc fetch -A # this only fetches 1 file from dvcstore2
# Collecting                                                                                                                   |3.00 [00:00,  410entry/s]
# WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
# md5: d41d8cd98f00b204e9800998ecf8427e
# Fetching
# 1 file fetched
find .dvc/cache/files/md5 -type f | wc -l # this returns 1

If I bring back the local config it will fetch the second file from /tmp/dvsctore2:

dvc remote add -d --local myremote2 /tmp/dvcstore2
dvc fetch -A
find .dvc/cache/files/md5 -type f | wc -l # this *now* returns 2

I agree with these sentiments from above:

Ideally, we need the same solution as Git has - I can easily move a repo with the entire history from, for example, GitHub to GitLab by a couple of commands just by changing remote, pushing and removing the old one. Is there a way to implement something similar in DVC?

and:

my 2cs on this: local config can be a good temporary solution, but it breaks a bit the point of repos being self descriptive/self contained. Why it can be important? For two reasons (and may be I'm missing something else):

1. Asking everyone to do local config means an additional step people will need to remember to do. Not the biggest problem, but makes everything a bit more fragile.
2. It complicates automation - CI (e.g. CML), Viewer - those environment where we are accessing repos from code. Local config is a possibility but will require additional steps from users. In the Viewer it'll require additional UI/UX to manage this.

[dberenbaum]

Keep in mind that if you changed the location of your Git remote, everyone on the team will need to remember to update it also, so the DVC local remote is no different from how Git works. The only difference between configuring DVC and Git remotes are that DVC usually keeps the remote info in your repo history. The upside of that choice is that not everyone has to configure/clone the remote individually, but the downside is this situation where you want to change the remote in your history.

The only solution I can see is making the remote a proxy to some other endpoint that can be changed on the backend without requiring user-facing changes on the frontend. This is an idea we have discussed and I would love to see, but it's unfortunately not something that is likely to be prioritized, so you would probably need to setup your own proxy if that's important to you.

[adamliter]

Yes, ironically, we're in the process of trying to move to a remote that is a proxy.

It seems like there's an additional difference between git and DVC. I'd imagine in a hypothetical world where you specified the git remote in a file tracked by git that git would not try to fetch/push half of the commits to the old remote and half to the new. I'm not familiar with the reason for this being the desired behavior in DVC with .dvc/config but not with .dvc/config.local. Assuming there's good reason to not change it for .dvc/config, is it fair to say these are the only two options:

1. Rewrite the git history
2. Ensure everyone creates a .dvc/config.local pointing to the right remote (and just delete .dvc/config to avoid confusion ...?)

The second option seems like a bit of a bummer, especially for stuff you might want to do in CI. I guess a different version of the first option would be to go create branching commits off of all of the old commits that introduced new files tracked by DVC where you update .dvc/config to point to the new remote, but that could get tedious. Would at least make CI more straightforward though, since you wouldn't need to create a .dvc/config.local file as part of the CI process, assuming you're just doing CI stuff off of all of those new commits.

Lastly, FWIW, is dvc push -A -r <NEW_REMOTE> failing with the warnings I mentioned above a bug? This thread seems to suggest it should actually push all files to the new remote, rather than fail to do so.

Thanks for engaging on this topic. Very much appreciate all of your comments and suggestions! 😄