Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve GitHub API usage #57

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

improve GitHub API usage #57

wants to merge 9 commits into from

Conversation

line-o
Copy link
Member

@line-o line-o commented Apr 2, 2024

  • incremental update reports ignored files
    Ignored files are expath-pkg.xml, repo.xml and all .xconf files.
    Files that would be ignored was already reported by the dry-run but not for the actual run.
    This allows for clients to act on that information and can, for instance, report that a full installation is necessary or that the index configuration has changed.

  • handle edge-case when a file is modified and then renamed
    That would have left the old file in the connected collection dangling.
    Now every renamed file will be deleted.
    As another edge-case where a file is added and then renamed would try to remove an non-existing file, github:incremental-delete

  • incremental updates handles pages
    When a single commit changes more than 300 changed files than the request is paged. The github module will now handle these.

  • download of files larger than 1MB
    Files larger than 1 megabyte are not returned in the query for metadata directly. That means we cannot rely on this query alone.
    If the content is empty we use the 'download_url' property instead.

For GitHub.com we can construct the file download URL to point to raw.githubusercontent.com and do not even need to query for metadata of a changed file first.

When a single commit changes more than 300 changed files than the request is paged. The github module will now handle these.
That would have left the old file in the connected collection dangling.
Now every renamed file will be deleted.
As another edge-case where a file is _added_ and then renamed would try to remove an non-existing file, `github:incremental-delete`
will now treat a not found resource as a successful deletion.
Ignored files are expath-pkg.xml, repo.xml and all .xconf files.
Files that would be ignored was already reported by the dry-run but not for the actual run.
This allows for clients to act on that information and can, for instance, report that a full installation is necessary or that the index configuration has changed.
When determining the changeset for incremental uploads but the local commit hash cannot be found on the remote
the error github:commit-not-found is raised.
- added case is now handled separately as the old entry must be filtered from the del sequence
- get rid of github:remove-or-ignore
  by always filtering lists
  This should improve readability.
@line-o line-o changed the title fix(github): download of files larger than 1MB improve GitHub API usage Oct 18, 2024
An existing file can be removed and then added by **renaming** another file to the previously removed one.
This will now be handled gracefully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant