Problem: using symlinks breaks Windows dev environments

[jhsimpson]

Archivematica cannot be deployed on Windows, but this PR from @minusdavid artefactual/deploy-pub#39 makes it possible to deploy a development environment on Windows, using vagrant to deploy to a linux vm.

That PR is working great, but there is a problem with checking out a git repo that contains symlinks into a windows filesystem (google it, lots of links). Windows doesn't properly support symlinks, and so checking out a repo with symlinks is difficult, ansible roles choke, you get weird git errors, etc.

In this repo, there are only a few symlinks being used - it would not be hard to remove them altogether. I think the only place left is in the osdeps folders. Removing those symlinks and creating duplicate files for now would allow osdeps to differ for each platform, which is fine, and would make developing in a Windows environment much easier, which is a bonus.

[jhsimpson]

This is also an issue in the storage service (same osdeps files).

[minusdavid]

Alas, it looks like the symlinks in archivematica-storage-service stable/0.10.x are what's preventing the checkout to qa/0.x:

TASK [artefactual.archivematica-src : Checkout archivematica-storage-service code] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x", "failed": true, "msg": "Failed to checkout qa/0.x", "rc": 1, "stderr": "error: Your local changes to the following files would be overwritten by checkout:\n\t
osdeps/RedHat-7.json\n\tosdeps/Ubuntu-16.json\n\tstorage_service/static/js/vendor/base64.js\nPlease, commit your changes or stash them before you can switch branches.\nAborting\n", "stderr_lines": ["error: Your local changes to the following files would be overwritten by
checkout:", "\tosdeps/RedHat-7.json", "\tosdeps/Ubuntu-16.json", "\tstorage_service/static/js/vendor/base64.js", "Please, commit your changes or stash them before you can switch branches.", "Aborting"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/vagrant/singlenode.retry

Because the working directory is dirty, it won't checkout. And even though we specify "force: yes" in Ansible, I think the Ansible git module has a bug where it doesn't force checkouts if the branch being checked out is a remote branch.

[minusdavid]

The same issue will apply to archivematica stable/1.6.x as well. Ansible just fails before it gets that far.

[minusdavid]

Here are the files in question:

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   osdeps/RedHat-7.json
    modified:   osdeps/Ubuntu-16.json
    modified:   storage_service/static/js/vendor/base64.js

[minusdavid]

^ for the archivematica-storage-service

For archivematica, I think fpr (which is a submodule?) will also be an issue.

[minusdavid]

I imagine removing the symlinks for osdeps would be easy enough, but I'm guessing not so much with the submodules?

[minusdavid]

I haven't used submodules myself so I'm not really sure how they're used in Archivematica.

[minusdavid]

Trying to reproduce the git problem but struggling...

[minusdavid]

Ahh... I was struggling to reproduce the problem on a system running Ansible 1.8.2, but I see in Ansible devel (and probably Ansible 2.3.2) that there is a difference in how the git clone is performed!

In older versions of Ansible, they would use the "version" to git clone with a specific branch. (Using syntax like @sevein described at #700).

But in newer versions of Ansible, they only clone with a specific branch if the depth parameters is used :S.
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/source_control/git.py#L448

We can confirm this for 2.3 as well:
https://github.com/ansible/ansible/blob/stable-2.3/lib/ansible/modules/source_control/git.py#L430

I'll try it to see if I can reproduce it manually on a system with Ansible 2.3.2...

[minusdavid]

First proof:
After running a Ansible playbook with just one task to clone archivematica-storage-service, this is what I get:

vagrant@am-local:/opt/archivematica/archivematica-storage-service$ git branch -vv

• qa/0.x 610e4eb [origin/qa/0.x] move db dependencies out of base requirements
  stable/0.10.x 1afb22a [origin/stable/0.10.x] Re-ingest: don't delete pointer file, refs #11356

If I run it on an older system, I just see the "qa/0.x" branch because it uses that branch option with the git clone.

[minusdavid]

So I deleted qa/0.x, made some modifications to stable/0.10.x and re-ran the simplified Ansible playbook and it did seem to force the checkout... which is surprising.

[minusdavid]

I'm guessing it must have done a reset first... and that must have worked whereas that wouldn't work on on the Windows/Linux Virtualbox shared folder.

I can confirm that if I hack Ansible a bit...

[minusdavid]

Ah just looked through the code instead and yeah it does a reset if there are local mods. So naturally it would only fail if the reset failed and if the checkout didn't use the force option properly applied, and I've already discovered that it doesn't have the force option properly applied...

[minusdavid]

I'm going to try adding the depth option because that should cause the git clone to use the branch parameter...

[minusdavid]

Edited artefactual.archivematica-src/tasks/ss-main.yml to include "depth: 1" and waiting for Ansible to run...

I think that should make the clone faster, and I think developers can fetch the rest of the history after the fact if they need it.

[minusdavid]

Ah looks like my edit might not have been included..

[minusdavid]

Ah I'll just copy my own role and define it... let's try that again.

[minusdavid]

Success for archivematica-storage-service. Looks like unexpected behaviour in Ansible is creating most of the problems here...

[minusdavid]

Looking at src/archivematica-storage-service, it just has the latest commit when using "depth: 1", but if I use "git fetch --unshallow" manually after the fact I get the full history for the qa/0.x branch. It also seems to have fetched the upstream repo's tags. However, I can't access any other remote branches...

Also, I'm facing this error once again:

TASK [artefactual.archivematica-src2 : Run SS django collectstatic] ************
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "./manage.py collectstatic --noinput", "failed": true, "msg": "\n:stderr: Traceback (most recent call last):\n File "./man
age.py", line 10, in \n execute_from_command_line(sys.argv)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/core/mana
gement/init.py", line 354, in execute_from_command_line\n utility.execute()\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/djan
go/core/management/init.py", line 303, in execute\n settings.INSTALLED_APPS\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/djan
go/conf/init.py", line 48, in getattr\n self._setup(name)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/__init
__.py", line 44, in setup\n self.wrapped = Settings(settings_module)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/
init.py", line 92, in init\n mod = importlib.import_module(self.SETTINGS_MODULE)\n File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module\n i
mport(name)\n File "/vagrant/src/archivematica-storage-service/storage_service/storage_service/settings/production.py", line 7, in \n import dj_database_url\nImport
Error: No module named dj_database_url\n", "path": "/usr/share/python/archivematica-storage-service/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "st
ate": "absent", "syspath": ["/tmp/ansible_EIX2bR", "/tmp/ansible_EIX2bR/ansible_modlib.zip", "/tmp/ansible_EIX2bR/ansible_modlib.zip", "/usr/lib/python2.7", "/usr/lib/python2.7/pla
t-x86_64-linux-gnu", "/usr/lib/python2.7/lib-tk", "/usr/lib/python2.7/lib-old", "/usr/lib/python2.7/lib-dynload", "/usr/local/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist
-packages"]}

[minusdavid]

Right, so if I update roles/artefactual.archivematica-src/tasks/ss-main.yml to use "depth: 1", I'm able to clone the archivematica-storage-service repository without running into that symlink/checkout issue.

But then - as a developer - I need to manually run the following in order to get all the remote branches and untruncated history for doing development:

git config remote.origin.fetch "+refs/heads/:refs/remotes/origin/"
git fetch --unshallow

That seems sub-optimal to me. Ansible's git module seems buggy to me in that regard.

[minusdavid]

As for the SS django collectstatic error, it's not the same error from artefactual/deploy-pub#40 but it's the same task. I see something about "No module named dj_database_url".

I'll leave that one up to you folk with more control and understanding of Archivematica internals.

[minusdavid]

Raised an issue with Ansible but even if they do think it's a bug, it wouldn't really help us in the short-term, unless they release a new version that can be targeted with pip and then we specify the use of pip in the Vagrantfile to get that particular version (or newer) of Ansible.

ansible/ansible#27991

[minusdavid]

Despite my many posts, I suppose this issue isn't really the right issue to raise all of this...

But I suppose it's mostly only an issue because of the symlinks in the stable branches.

[sromkey]

It looks like there are changes requested in artefactual/deploy-pub#39 . I'm untagging this from the 1.7 release and bumping to 1.8 for the time being.