Skip to content

Commit

Permalink
Dimitrovr/test permissions (#14)
Browse files Browse the repository at this point in the history
* Update supported specification version to 1.0.28

I looked into all changes between our current version 1.0.19 and the
current version of the specification 1.0.28 and I agree with Jussi that
the only one not fully resolved is:
"8dafd00 (tag: v1.0.24) Clarify optional attributes" and more precisely
the changes from commit:
https://github.com/theupdateframework/specification/pull/165/commits/4dd279bc318afaea9c069b265c0468e235df0192

It doesn't make sense to have a target file without "paths" or
"path_hash_prefixes", so our `python-tuf requirement to have at least
one of them set makes sense.

Both with Jussi we agreed that we can easily loosen this requirement if
when solving https://github.com/theupdateframework/specification/issues/200
it's decided that both of them can be omitted,
but for now, we decided it's better to stick to our current requirement
to have one of them set.

Signed-off-by: Martin Vrachev <[email protected]>

* Revert "github: disable pip caching temporarily"

This reverts commit 55d6cb47da9d2eb3bd6ebdbc3b93a5b8884d9454.

According to changelog setup-python v2.3.2 should include a workaround
for the issue.

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump pynacl from 1.4.0 to 1.5.0

Bumps [pynacl](https://github.com/pyca/pynacl) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/pyca/pynacl/releases)
- [Changelog](https://github.com/pyca/pynacl/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/pynacl/compare/1.4.0...1.5.0)

---
updated-dependencies:
- dependency-name: pynacl
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump requests from 2.26.0 to 2.27.1

Bumps [requests](https://github.com/psf/requests) from 2.26.0 to 2.27.1.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.26.0...v2.27.1)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump urllib3 from 1.26.7 to 1.26.8

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.7 to 1.26.8.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.7...1.26.8)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump charset-normalizer from 2.0.7 to 2.0.11

Bumps [charset-normalizer](https://github.com/ousret/charset_normalizer) from 2.0.7 to 2.0.11.
- [Release notes](https://github.com/ousret/charset_normalizer/releases)
- [Changelog](https://github.com/Ousret/charset_normalizer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ousret/charset_normalizer/compare/2.0.7...2.0.11)

---
updated-dependencies:
- dependency-name: charset-normalizer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump cryptography from 35.0.0 to 36.0.1

Bumps [cryptography](https://github.com/pyca/cryptography) from 35.0.0 to 36.0.1.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/35.0.0...36.0.1)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Document serialization "repro" issue

It's not obvious to casual reader that reading metadata and then
writing it might not always produce the same file. It's also not
immediately obvious why this matters.

Document both concepts.

Fixes #1392

Signed-off-by: Jussi Kukkonen <[email protected]>

* doc: render tuf logo and favicon on rtd

Configure docs to display
- tuf icon as favicon
- tuf horizontal logo (white) in navbar

Signed-off-by: Lukas Puehringer <[email protected]>

* Exceptions docs for __init__ and from_dict()

Document ValueError, KeyError and TypeError exceptions for __init__ and
from_dict() methods in Metadata API.

Signed-off-by: Martin Vrachev <[email protected]>

* github: Pin actions hashes

This allows us to control when our workflows change.
Dependabot should now open PRs when the actions update.

This still leaves the actual OS image as a variable but Github does not
support pinning that: we'd have to start using our own containers (and
installing our own pythons, etc) to do that -- not worth the trouble.

Fixes #1826

Signed-off-by: Jussi Kukkonen <[email protected]>

* Add small missing tests

Add a test triggering the MetaFile version validation and a TargetFile
test accessing custom.

Signed-off-by: Martin Vrachev <[email protected]>

* build: Re-add setup.py

The Python build tools are fine without a setup.py but Dependabot
chokes: https://github.com/dependabot/dependabot-core/issues/4483

Add a setup.py to keep Dependabot happy.

Fixes #1828

Signed-off-by: Jussi Kukkonen <[email protected]>

* build: Remove pylintrc from MANIFEST

pylint config lives in pyproject.toml nowadays.

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump pycparser from 2.20 to 2.21

Bumps [pycparser](https://github.com/eliben/pycparser) from 2.20 to 2.21.
- [Release notes](https://github.com/eliben/pycparser/releases)
- [Changelog](https://github.com/eliben/pycparser/blob/master/CHANGES)
- [Commits](https://github.com/eliben/pycparser/compare/release_v2.20...release_v2.21)

---
updated-dependencies:
- dependency-name: pycparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* verify_signature(): handle SerializationError

We should handle the possible SerializationError inside
Key.verify_signature(), because the user of this API is not interested
in SerializationError when he is trying to verify his signature.

Note that the SerializationError can be thrown when calling
signed_serializer.serialize() on the metadata signed part.

Signed-off-by: Martin Vrachev <[email protected]>

* build: add license field to setup.cfg

List our licenses in the license field of setup.cfg

While the PyPA packaging documentation states that the license field is
optional[1] and that classifiers should be the main way to indicate
license, this field is used to populate the License printed by pip show.

1. https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#license

Fixes #1833

Signed-off-by: Joshua Lock <[email protected]>

* Improve docstrings language in Metadata API - quotes

This change unifies quotes to double backtick across docs in the
Metadata API in order to provide better visualisation

Signed-off-by: Ivana Atanasova <[email protected]>

* Improve docstrings language in Metadata API - wording

This change unifies wording across docs in the Metadata API, like
Args vs. Arguments and same repetitive descriptions written
differently in different classes/methods

Signed-off-by: Ivana Atanasova <[email protected]>

* Improve docstrings language in Metadata API - article

This change unifies as mush as the context allows and improves the
use of definite vs. indefinite vs. no article across docs in the
Metadata API. It sticks to no article in most cases for simplisity
and readability, but leaves definite article where it's strictly
necessary

Signed-off-by: Ivana Atanasova <[email protected]>

* Update repetitive docstrings language in Metadata API

This change updates some obvious and unnecessary fields docs in the
Metadata API with more despriptive details

Signed-off-by: Ivana Atanasova <[email protected]>

* verify_delegate() doc ValueError and TypeError

Add missing documentation for ValueError and TypeError inside
Metadata.verify_delegate().

Signed-off-by: Martin Vrachev <[email protected]>

* build: Remove docs build requirement version pin

docutils is a sphinx-rtd-theme requirement: pinning was done
to workaround a bug that seems to now be fixed.

Signed-off-by: Jussi Kukkonen <[email protected]>

* from_securesystemslib_key() raise ValueError

If a securesystemslib.FormatError is raised inside
Key.from_securesystemslib_key() then reraise ValueError.
This is done so that our users don't have to import securesystemslib
in order to handle the error and because the securesystemslib error
itself is securesystemslib implementation-specific.

Signed-off-by: Martin Vrachev <[email protected]>

* doc: remove obsolete docs/images dir

Remove obsolete docs/images directory which contains unused
variants of the logo. The canonical location of TUF logos is
theupdateframework/artwork, which has high-resolution formats (png
and svg) for all variants of the logo.

Also see https://github.com/theupdateframework/artwork/pull/3.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: rename contribution instructions

Rename CONTRIBUTORS.rst -> CONTRIBUTING.rst. The new name is what
GitHub expects and will make the document more discoverable, e.g.
on https://github.com/theupdateframework/python-tuf/contribute.

More details under:
https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors

Note: I searched all repositories in theupdateframework GitHub
organization for references (there were none) and will update the
links in the CII Best Practice badge app for tuf.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: update installation documentation

Update severely outdated installation documentation.

- Simplify "Simple Installation" section
- Update "Release Verification" section to actually verify a tuf
  release and with a key of an active maintainer
- Update and simplify section about non-python dependencies
  (just point to installation instructions for underlying crypto
  backends, they are up-to-date and have become a lot easier)
- Add "Development installation" section

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: update install section in contributing doc

Replace custom installation section in contribution docs with
pointer to updated installation documentation.

Also configure sphinx autosectionlabel for cross-document refs.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: minor title changes for rtd navbar

Make contributing document header sentence case for consistency
with other docs and shorten menu name in side navbar to stand out
less.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: fix link to tox docs

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: move verify section in install docs

Move release signature verification instructions to bottom of
install docs. The doc is short, so the section is still prominent
enough for promoting verification, but does not break the reading
flow as much anymore.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: cross-doc absolute link hack

This is an ugly hack to also resolve the link when the document is
rendered in GitHub, where it is likely to be browsed, because it is
the community standard location for a GitHub repo's contributing
docs.

Coordinate with #1849 to better separate RTD docs with GitHub docs
in the future.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: Remove inactive maintainers from MAINTAINERS

As discussed in detail in #1793, maintainer-level (GitHub)
permissions should be granted to those who need them, i.e. who
actively maintain the project at the moment.
The MAINTAINERS.txt file should reflect that state.

It will be reviewed regularly (#1803), and can be changed (e.g.
reverted to a prior state) at any time as need arises.

To express our appreciation for past efforts, we might use the
Acknowledgement section of the README, and also update it
regularly.

In the case of this update: Big kudos to @awwad, @SantiagoTorres
and @sechkova for all their valuable contributions to python-tuf!

Signed-off-by: Lukas Puehringer <[email protected]>

Signed-off-by: Lukas Puehringer <[email protected]>

* Update securesystemslib version to 0.22.0

Signed-off-by: Martin Vrachev <[email protected]>

* Add signatures serialization tests

Signed-off-by: Martin Vrachev <[email protected]>

* Move nonunique sigs test to serialization tests

Move the duplicating signatures tests from test_metadata_base function
in test_api.py into test_metadata_serialization.py.
This is a more logical place to store this test case as
test_metadata_base is actually focused on testing
Metadata.signed.is_expired.
That also is the reason why I renamed test_metadata_base to
test_metadata_signed_is_expired.

Signed-off-by: Martin Vrachev <[email protected]>

* Remove unnecessary copy operations

There is no need to copy "case_dict" inside serialization test
functions in test_metadata_serialization.py when we are testing
invalid arguments.
These dictionaries are not be used after calling "from_dict" and
it doesn't matter if they are empty afterward.

Signed-off-by: Martin Vrachev <[email protected]>

* doc: add emeritus section to maintainers file

Signed-off-by: Lukas Puehringer <[email protected]>

* Unify quoting in ngclient docstrings

This change updates the docstrings library of ngclient with
unified double backtick quoting for better readability

Signed-off-by: Ivana Atanasova <[email protected]>

* Unify article in ngclient docstrings

This change updates the docstrings library of ngclient with no
article for all Args in order to be unified amongst all python-tuf
docstrings

Signed-off-by: Ivana Atanasova <[email protected]>

* Unify wording of docstrings language in ngclient

This change unifies common wording in the docstrings library of
ngclient, like "Args" vs. "Arguments"

Signed-off-by: Ivana Atanasova <[email protected]>

* Fix line lengths in ngclient docstrings

This change shortens line lengths that exceed the requiremets and
adds more clarification on methods where the short message is not
complete enough

Signed-off-by: Ivana Atanasova <[email protected]>

* Metadata test full serialization cycle

Replace the usage of Metadata.to_dict inside
test_valid_metadata_serialization and instead use Metadata.to_bytes()
in order to test that the full serialization cycle is working as
expected:
Metadata.from_bytes -> Metadata.to_bytes

Signed-off-by: Martin Vrachev <[email protected]>

* build(deps): bump charset-normalizer from 2.0.11 to 2.0.12

Bumps [charset-normalizer](https://github.com/ousret/charset_normalizer) from 2.0.11 to 2.0.12.
- [Release notes](https://github.com/ousret/charset_normalizer/releases)
- [Changelog](https://github.com/Ousret/charset_normalizer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ousret/charset_normalizer/compare/2.0.11...2.0.12)

---
updated-dependencies:
- dependency-name: charset-normalizer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Add support for unrecognized fields in Metadata

The Document formats section (chapter 4) of the
specification says the following:

"All of the formats described below include the ability to add more
attribute-value fields to objects for backward-compatible format
changes. Implementers who encounter undefined attribute-value pairs in
the format must include the data when calculating hashes or verifying
signatures and must preserve the data when re-serializing."

I initially thought it's applicable only to the SIGNED fields as
"undefined attribute-value pairs in the format must include the data
when calculating hashes or verifying signatures"
This doesn't mean that the sentence before that excludes "Metadata" as a
possible place for additional fields.
The other maintainers agreed with me and we are going to add support for
'unrecognized_fields" inside "Metadata".

Signed-off-by: Martin Vrachev <[email protected]>

* dep: update pinned requirements

As described in #1249 requirements-pinned.txt is automatically
updated by Dependabot on version updates, but not if transitive
dependencies are added or removed.

This patch removes the no longer required transient dependency six,
following a run of pip-compile for all supported Python versions.

No other dependency changes were detected, nor were there different
dependencies in different Python versions, requiring env markers
in the requirements file.

Signed-off-by: Lukas Puehringer <[email protected]>

* build: pin test requirements for deterministic CI

Configures tox to use a pinned requirements file for deterministic
CI builds, i.e. our CI shouldn't start failing because of an
incompatible upstream release of any of our testing tools:

NOTE: pinned tuf runtime requirements were already were already
used for test builds before (included via `-r
requirements-pinned.txt` in 'requirements-test.txt'). Now they are
explicitly listed in 'requirements-test-pinnned.txt'.

'requirements-test-pinnned.txt' was generated semi-automatically by
running pip-compile over 'requirements-test.txt' for each
supported/tested Python version (see snippet below) and manually
merging the resulting per-Python version requirements files into
one, adding environment markers as needed.

```
for ver in 3.7.12 3.8.12 3.9.9 3.10.0; do
  pyenv virtualenv ${ver} tuf-env-${ver}
  pyenv activate tuf-env-${ver}
  python3 -m pip install -U pip pip-tools
  pip-compile --no-header --annotation-style line \
      -o requirements-test-pinned-${ver}.txt \
      requirements-test.txt
  pyenv deactivate
  pyenv uninstall -f tuf-env-${ver}
done
```

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: minor readme updates

- Add generic opening sentence that says what TUF actually does.
- Add link to #tuf channel on CNCF slack to contact section

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: remove note about unstable API in RTD docs

The API is no longer unstable.

Signed-off-by: Lukas Puehringer <[email protected]>

* github: Update github-script to 6.0.0

The big change is runtime update from nodejs 12 to nodejs 16: does not
seem to affect us.

Dependabot got confused so this update is done manually to v6.0.0
release commit:
https://github.com/actions/github-script/releases/tag/v6.0.0

Signed-off-by: Jussi Kukkonen <[email protected]>

* doc: update acknowledgements and rm AUHTORS.txt

Update README.md#Acknowledgements
- Reword to acknowledge maintainer contributions as well
- Remove names that are mentioned in maintainers document
- Remove duplicate Konstantin Andrianov
  Santiago Torres-Arias, Sebastien Awwad, Trishank Kuppusamy,
  Vladimir Diaz)
- Add new significant contributors
  (Ivana Atanasova, Kairo de Araujo, Martin Vrachev)

Remove unmaintained AUTHORS.txt, which lists many individuals and
organisations that are/were not affiliated with 'python-tuf', but
other projects in the TUF ecosystem (Thandy, Notary, etc.) and
thus is not suited for this repository.
-> theupdateframework.io#38

Caveats:
- Significant contributors means  top ~20 committers sorted by
  commit count (`git shortlog -s`).
- The Acknowledgements section might miss significant contributors,
  if they contributed by other means than git commits in this repo.

Signed-off-by: Lukas Puehringer <[email protected]>

* CI: remind to update contributor acknowledgement

Add optional task to  maintainer permission review reminder
checklist that suggests to also update the list of significant
contributors in README.md#acknowledgements.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: reword announcement about upcoming 1.0.0

Change docs in preparation of close v1.0.0 release.

- Remove important notice about upcoming 1.0.0 release from README
- Reword 1.0.0-ANNOUNCEMENT.md to not sound outdated after release

Co-authored-by: Joshua Lock <[email protected]>
Signed-off-by: Jussi Kukkonen <[email protected]>

* __init__.py: Remove unused constant

Metadata API defines a specification version it supports already,
and that one is updated to the actual specification version we
produce.

Signed-off-by: Jussi Kukkonen <[email protected]>

* python-tuf version 1.0.0 \o/

* Update Changelog
* Update version numbers

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Add messages to serialization errors

We can't really add any details but this at least means
printing the error works.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Log details of verify error

We don't want to error out from the whole verify_delegate() process if
e.g. a single key fails to load but we do want to provide details for
debugging in the unexpected failure cases.

This means "example_client -vv  download file1.txt" fails like this:

    Found trusted root in /home/jku/.local/share/python-tuf-client-example
    INFO:tuf.api.metadata:Key
4e777de0d275f9d28588dd9a1606cc748e548f9e22b6795b7cb3f63f98035fcb failed
to verify sig: Failed to load PEM key bogus-key-content-here
    INFO:tuf.api.metadata:Key 4e777de0d275f9d28588dd9a1606cc748e548f9e22b6795b7cb3f63f98035fcb failed to verify root
    Failed to download target x: root was signed by 0/1 keys

Fixes #1875

Signed-off-by: Jussi Kukkonen <[email protected]>

* docs: Add blog configuration

Add config for GitHub Pages so that we can use it as a project blog.
* _config.yml is jekyll configuration
* index.md contains description and title for the blog main page.
* Any files matching "_posts/YYYY-MM-DD-TITLE.md" are considered posts

The Github Pages configuration only allows "/" or "/docs/" as the Jekyll
root directory: The clutter in docs/ is annoying but otherwise this is a
very easy setup.

Signed-off-by: Jussi Kukkonen <[email protected]>

* docs: Add a blog post

This is https://ssl.engineering.nyu.edu/blog/2022-02-21-tuf-1_0_0
only slightly modified (the logo would break the excerpts in the index
page so I moved it a bit).

Signed-off-by: Jussi Kukkonen <[email protected]>

* docs: Add a new 200px icon

Also rename the existing icon so differences are obvious.

Signed-off-by: Jussi Kukkonen <[email protected]>

* docs: Clean up blog header

Minima theme by default adds all files in blog root (docs/) as links in
the header. This looks ridiculous in our case: let's just have a link to
blog front page.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Add __eq__ to classes in Metadata API

By adding __eq__ we can compare that two objects are equal.
That will be useful when adding validation API call.

One bug I have found during testing is that I don't check if the type
of "other" in the __eq__ implementations are the expected ones.
I assumed that when comparing "root == obj" if "obj" is None that
automatically the result will be false.
Later after a mypy warning, I realized we should implement the __eq__
methods to accept "Any" type as other and we should check manually
that "other" is the expected type.

Signed-off-by: Martin Vrachev <[email protected]>

* Test __eq__ implementation for all classes

Test the "__eq__" implementation for all classes defined in
tuf/api/metadata.py
The tests are many but simple. The idea is to test each of the metadata
classes one by one and with this to make sure there are no possible
cases missed.

Signed-off-by: Martin Vrachev <[email protected]>

* Add "validation" arg in JSONSerializer

If the "validation" argument is set then when
serializing the metadata object will be validated.

Signed-off-by: Martin Vrachev <[email protected]>

* Take order into account for certain cases

After we have dropped OrderedDict in https://github.com/theupdateframework/python-tuf/pull/1783/commits/e3b267e2e0799673ac99ccfccd3631628013201c
we are relying on python3.7+ default behavior to preserve the insertion
order, but there is one caveat.
When comparing dictionaries the order is still irrelevant compared to
OrderedDict. For example:
>>> OrderedDict([(1,1), (2,2)]) == OrderedDict([(2,2), (1,1)])
False
>>> dict([(1,1), (2,2)]) == dict([(2,2), (1,1)])
True

There are two special attributes, defined in the specification, where
the order makes a difference when comparing two objects:
- Metadata.signatures
- Targets.delegations.roles.
We want to make sure that the order in those two cases makes a
difference when comparing two objects and that's why those changes
are required inside two __eq__ implementations.

Signed-off-by: Martin Vrachev <[email protected]>

* build(deps): bump actions/setup-python from 2.3.2 to 3

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2.3.2 to 3.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/7f80679172b057fc5e90d70d197929d454754a5a...0ebf233433c08fb9061af664d501c3f3ff0e9e20)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump actions/checkout from 2.4.0 to 3

Bumps [actions/checkout](https://github.com/actions/checkout) from 2.4.0 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/ec3a7ce113134d7a93b817d10a8272cb61118579...a12a3943b4bdde767164f792f33f40b04645d846)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* build: Single source version number

As of setuptools 46.4.0, one can accomplish single source version
number with
    version = attr: package.__version__
in setup.cfg: As long as setuptools simplified AST parser is able to
read the file, this works without actually importing anything.

Signed-off-by: Jussi Kukkonen <[email protected]>

* docs: Update release docs

* version number is single sourced now
* Mention that using pip against test.pypi.org is unsafe
* Fix some filenames in the examples

Signed-off-by: Jussi Kukkonen <[email protected]>

* Remove redundant comment about version

The version is no longer duplicated in setup.cfg (since 5155ba74), so remove
redundant TODO suggesting folks update in two places.

Co-authored-by: Ofek Lev <[email protected]>
Signed-off-by: Joshua Lock <[email protected]>

* test: use tox isolated environments

Enable tox isolated environments to perform build operations in a virtual
environment.
See https://tox.wiki/en/latest/config.html#conf-isolated_build

Co-Authored-By: Ofek Lev <[email protected]>
Signed-off-by: Joshua Lock <[email protected]>

* setup: remove upper bound limit on python_requires

Setting upper bound version constraints in libraries is a source of
problems for users of those libraries, see:
https://iscinumpy.dev/post/bound-version-constraints/

The intent of the python-tuf version constraint is to ensure we're
using a version of Python which supports all the features we rely
on, this is a better fit for a lower limit.

Suggested-by: Ofek Lev <[email protected]>
Signed-off-by: Joshua Lock <[email protected]>

* gitignore: fix directory patterns

Fix the directory ignore patterns to ignore the entire directories,
including child directories.
https://git-scm.com/docs/gitignore#_pattern_format

Co-authored-by: Ofek Lev <[email protected]>
Signed-off-by: Joshua Lock <[email protected]>

* Use spec version from tuf/api/metadata in examples

Replace the hardcoded specification version with the one defined inside
tuf/api/metadata.py

Signed-off-by: Martin Vrachev <[email protected]>

* Update package metadata

Signed-off-by: Ofek Lev <[email protected]>

* Improve docstrings in Metadata API to be more descritpive

This change updates some parts of the Metadata API docstrings
that did not give enough details and context

Fixes #1600

Signed-off-by: Ivana Atanasova <[email protected]>

* Improve Signer docstrings in Metadata API

Change to @lukpueh proposal with more clarification on why and how
the `securesystemslib.signer.Signer` interface is used

Co-authored-by: Lukas Pühringer <[email protected]>

Signed-off-by: Ivana Atanasova <[email protected]>

* Add tests for Updater input validation

This test covers `targetinfo`, `target_path`, `target_base_url`,
`metadata_dir` and `filepath` input validation of the `Updater`
methods

Signed-off-by: Ivana Atanasova <[email protected]>

* Test expired metadata from cache

This tests that an expired timestamp/snapshot/targets when loaded
from cache is not stored as final but is used to verify the new
timestamp

Fixes #1681

Signed-off-by: Ivana Atanasova <[email protected]>

* Verify validation is performed from local metadata

This change verifies that when local metadata has expired, it is
still used to verify new metadata that's pulled from remote

Signed-off-by: Ivana Atanasova <[email protected]>

* Fix expired metadata tests

This change fixes the expired metadata tests to mock `datetime`
as previously they mocked `time` incorrectly, which did not affect
update methods, as they use `datetime.datetime.utcnow()` to
calculate now

Signed-off-by: Ivana Atanasova <[email protected]>

* Update expired metadata tests logic

This change improves the logic of expired metadata tests, so that
it is explicitly visible what the expiry time and the versions are
and when update/refresh is called in that period

Signed-off-by: Ivana Atanasova <[email protected]>

* build: Add verify-release script

verify-release
* Builds a release from current commit
* Notifies if git describe does not match built version
* Notifies if built version is not the latest GitHub or PyPI version
* Asserts that the GitHub and PyPI release artifacts match the built
  release artifacts

This should be useful after release as any developer (or a CI job) can
easily verify that the release matches the sources in git.

Note that the last checks currently fail as the 1.0 build was not
reproducible. They should succeed after next release.

Signed-off-by: Jussi Kukkonen <[email protected]>

* gitattributes: make all JSON files end with LF

A really specific bug occurred on CI runs on all Windows machines
https://github.com/theupdateframework/python-tuf/runs/5467473050?check_suite_focus=true
where we weren't able to verify that what was generated is the same
as the stored on Git.

After research with Jussi, we found out that the problem comes not
from the content of the file that was generated, but because on Windows
Git proactively replaced all line endings for text files with CRLF symbol
("\r") this made the locally stored JSON files different from the one
generated.

We want to make sure such bugs doesn't occur again and that's why we
disable this behavior for all JSON files.

Signed-off-by: Martin Vrachev <[email protected]>

* Provide a way to generate a simple repository

I created a new script called "generate_md.py" which can be used
to easily generate a repository. Additionally, I created a new
test file making sure that the locally stored metadata files and
the newly generated metadata roles are the same.
This will allow us to test that we are not changing the metadata
file structure when making changes.

Signed-off-by: Martin Vrachev <[email protected]>

* Revert "build: pin test requirements for deterministic CI"

This reverts commit 5643cecf688876c1bca78dd60d13ba94d4c98cc0.

Signed-off-by: Lukas Puehringer <[email protected]>

* build: pin direct test dependencies

Fixes #1899
Reverts #1867

In #1867 we started pinning direct and transitive test
dependencies for stable test results, i.e. to not have an unnoticed
update of a used test tool (or their dependencies) break our tests.

This resulted in a dependabot updates inundating our PR tracker,
potentially obfuscating updates, which we care to address with
higher priority.

As a compromise we now only pin direct test dependencies, which
should still give us relatively stable test runs, while reducing
the spam.

Signed-off-by: Lukas Puehringer <[email protected]>

* build(deps): bump cryptography from 36.0.1 to 36.0.2

Bumps [cryptography](https://github.com/pyca/cryptography) from 36.0.1 to 36.0.2.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/36.0.1...36.0.2)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* verify_release: Be explicit about PyPI version

We are interested in what pip thinks is the current tuf version: make
that explicit in method naming and comments.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Add default constructor arguments

This allows creating new metadata with less boilerplate:
    root = Metadata(Root())
    targets = Metadata(Targets())

Set reasonable default values for all the arguments -- version to
1, spec_version to current supported version, etc.

Expires does not have a good default value and my original plan was
to require expires argument to be set. That would mean an
incompatible API change though as arguments before expires would be
now optional... So expires now defaults to an arbitrary value of 1
day from moment of creation.

One noteworthy special case is consistent_snapshot where the default
value is True (since that's what we want people to use for new
metadata) but None is also used to imply that metadata does not contain
the field at all.

Signed-off-by: Jussi Kukkonen <[email protected]>

* tests: Use the default Metadata constructor args

Signed-off-by: Jussi Kukkonen <[email protected]>

* examples: Use the constructor default arguments

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump urllib3 from 1.26.8 to 1.26.9

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.8 to 1.26.9.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.9/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.8...1.26.9)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* tests: Remove unused variables from generate_md

Signed-off-by: Jussi Kukkonen <[email protected]>

* verify_release: Be specific about expected artifacts

Use a hard-coded list of artifacts that we expect to find in a
release. Specifically check that each of those files matches
the corresponding file in locally built release.

Also add two missing annotations.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Be more careful with container args

If argument is an empty container, we want to use the given empty
container. Only create a new container if argument is None.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Set default expires to utcnow()

This means the metadata is by default expired: this seems like a fine
default since we only allow a default value for practical reasons (not
allowing it would mean backwards incompatible API change).

Signed-off-by: Jussi Kukkonen <[email protected]>

* verify_release: Warn about missing requirements

This is mostly useful for build module as it's not imported otherwise:
we explicitly call "python -m build" so everything works like in a
real release build.

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump pylint from 2.12.2 to 2.13.2

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.12.2 to 2.13.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.12.2...v2.13.2)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump types-requests from 2.27.14 to 2.27.15

Bumps [types-requests](https://github.com/python/typeshed) from 2.27.14 to 2.27.15.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump mypy from 0.941 to 0.942

Bumps [mypy](https://github.com/python/mypy) from 0.941 to 0.942.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.941...v0.942)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump black from 22.1.0 to 22.3.0

Bumps [black](https://github.com/psf/black) from 22.1.0 to 22.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.1.0...22.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* import requests.exceptions is not necessary

All calls use requests.* and importing requests.exceptions is not
necessary.

Signed-off-by: Kairo de Araujo <[email protected]>

* build(deps): bump pylint from 2.13.2 to 2.13.4

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.2 to 2.13.4.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.2...v2.13.4)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Add test coverage for delegated hash bins

This change adds tests coverage for `path_hash_prefixes` and
verifies that role names matching specific prefixed successfully
find and download the corresponding metadata files

Signed-off-by: Ivana Atanasova <[email protected]>

* tests: Small refactor of a test

Test was supposed to test a threshold that is higher than number of
signatures, but it actually was just using completely unsigned metadata.

This still doesn't test the case where _trusted_ metadata defines a
threshold that new metadata does not reach: only the case where new
metadata defines threshold that it does not meet (this case is covered
in updater tests though).

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump types-requests from 2.27.15 to 2.27.16

Bumps [types-requests](https://github.com/python/typeshed) from 2.27.15 to 2.27.16.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump actions/setup-python from 3.0.0 to 3.1.0

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3.0.0 to 3.1.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/0ebf233433c08fb9061af664d501c3f3ff0e9e20...9c644ca2ab8e57ea0a487b5ec2f8290740378bfd)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Metadata API: Document constructor default arguments

Signed-off-by: Jussi Kukkonen <[email protected]>

* build: add GH workflow to build + release on PyPI

Add workflow with two jobs to build and publish on PyPI.  The
release job waits for the build job and uses a custom release
environment, which can be configured to require review.

To share the build artifacts between the jobs and to make them
available for intermediate review, they are stored using
'actions/upload-artifact' and 'actions/download-artifact'.
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts

To upload the build artifacts to PyPI, the PyPA recommended
'pypa/gh-action-pypi-publish' is used.
https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

**Caveat**
The URL to grab the artifacts, e.g. for review, requires knowledge
of action ID and artifact ID, and a login token (no special
permissions). This makes it a bit cumbersome to fetch the artifacts
with a script and compare them to a local build.
https://docs.github.com/en/actions/managing-workflow-runs/downloading-workflow-artifacts

Signed-off-by: Lukas Puehringer <[email protected]>

* build: update CD workflow to create GH release

- Create preliminary GitHub release (X.Y.Z-rc) in 'build' job,
  using popular 3rd-party 'softprops/action-gh-release'.
- Finalize GH release in 'release' job using custom GH script.

Signed-off-by: Lukas Puehringer <[email protected]>

* build(deps): bump pylint from 2.13.4 to 2.13.5

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.4 to 2.13.5.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.4...v2.13.5)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* build: update CI/CD workflow to run in series

- Change CI workflow to also run on push to (release) tag
- Change CD workflow to run on successful CI run, and only if a
  (release) tag push triggered the CI

NOTE: Unfortunately the setup is not very robust
      (see code comment in cd.yml)

Signed-off-by: Lukas Puehringer <[email protected]>

* build: lint 'verify_release' with tox

Enable tox to lint 'verify_release' script and fix:
- whitespace
- unused import (we only import here to see if the module is
  available for use in a subprocess)
- unfound import (same as unused import)

Signed-off-by: Lukas Puehringer <[email protected]>

* build: add skip-pypi flag to verify_release script

Add '--skip-pypi' flag to 'verify_release' script to allow for
pre-release checks, when the automatic build job has uploaded the
build assets to GitHub and is awaiting review/approval in order to
upload it to PyPI eventually.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: describe auto release workflow in RELEASE.md

Change RELEASE.md to include instructions to trigger and review
auto release workflow (CI/CD).

Signed-off-by: Lukas Puehringer <[email protected]>

* build(deps): bump actions/setup-python from 3.1.0 to 3.1.1

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3.1.0 to 3.1.1.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/9c644ca2ab8e57ea0a487b5ec2f8290740378bfd...21c0493ecfd34b1217f0a90ec19a327f3cc0a048)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* verify_release: Build from git sources only

Make a new (local) git clone to build from. This ensures uncommitted
files do not affect the build.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Annotate 'unrecognized_fields' as Dict

Fixes #1938

Description of the changes being introduced by the pull request:

Annotating as Mapping seems wrong as further changes to the content might
be added in the code base. Hence, annotation changed to Dict.

Signed-off-by: Abhisman Sarkar <[email protected]>

* Tests: restore objects to initial state after test

Inside test_metadata_eq_.py we test the __eq__ implementations of all
classes. In order to do this, we change the attribute of the object and
then compare them to the unchanged version of those objects.
Usually, we do it in the following steps:
1. create an initial version "a"
2. create a copy of "a" called "b"
3. iterate all attributes inside "b" and change them to a given value
4. check that "a" and "b" are different

We do however forget to restore the object `b` to its initial state
which means we don't check the `__eq__` correctly as we stop on the
first, the found difference which could be of an older attribute changed
in one of the past iterations.

Signed-off-by: Martin Vrachev <[email protected]>

* tests: Test client max metadata length config

Fixes #1730

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump actions/checkout from 3.0.0 to 3.0.1

Bumps [actions/checkout](https://github.com/actions/checkout) from 3.0.0 to 3.0.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/a12a3943b4bdde767164f792f33f40b04645d846...dcd71f646680f2efd8db4afa5ad64fdcba30e748)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump types-requests from 2.27.16 to 2.27.19

Bumps [types-requests](https://github.com/python/typeshed) from 2.27.16 to 2.27.19.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Replaced manual path construction with os.path.join

Signed-off-by: Ari <[email protected]>

* Reverted URL construction back to f-strings

Signed-off-by: Ari <[email protected]>

* Update supported spec version

Signed-off-by: Marina Moore <[email protected]>

* Regenerate tests with new spec version

Signed-off-by: Marina Moore <[email protected]>

* build(deps): bump actions/setup-python from 3.1.1 to 3.1.2

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3.1.1 to 3.1.2.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/21c0493ecfd34b1217f0a90ec19a327f3cc0a048...98f2ad02fd48d057ee3b4d4f66525b231c3e52b6)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* doc: describe repo setup in RELEASE.md + typos fix

Signed-off-by: Lukas Puehringer <[email protected]>

* build: minor updates in CI/CD workflow files

- polish code comments
- wrap long lines

Signed-off-by: Lukas Puehringer <[email protected]>

* build(deps): bump pylint from 2.13.5 to 2.13.7

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.5 to 2.13.7.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.5...v2.13.7)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump types-requests from 2.27.19 to 2.27.20

Bumps [types-requests](https://github.com/python/typeshed) from 2.27.19 to 2.27.20.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update build config to include examples dir

Explicitly include `examples` dir in sdist.

The same would be achieved, by removing explicit includes, which
currently would also add these files/dirs:

```
/gitattributes
/github
/mypy_cache
/readthedocs.yaml
/verify_release
```

Maybe we should instead of defining includes, explicitly exclude
(some of) these files? The advantage of a blacklist approach is
that it becomes less likely to forget including files that should
be in included.

See hatch docs for:
- what files should be in sdist
https://ofek.dev/hatch/latest/plugins/builder/#source-distribution

- what files get into sdist by default:
https://ofek.dev/hatch/latest/plugins/builder/#default-file-selection_1

- how to configure what files get into sdist:
https://ofek.dev/hatch/latest/config/build/#file-selection

Fixes #1901

Signed-off-by: Lukas Puehringer <[email protected]>

* build(deps): bump actions/checkout from 3.0.0 to 3.0.2

Bumps [actions/checkout](https://github.com/actions/checkout) from 3.0.0 to 3.0.2.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...2541b1294d2704b0964813337f33b291d3f8596b)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Refactor ci/cd workflows (WIP)

This is an intermediate commit for easier review. See subsequent
commit for details.

Signed-off-by: Lukas Puehringer <[email protected]>

* Refactor ci/cd workflows

Prior to this change, ci triggered cd, depending on the event that
triggered ci. Due to the vague information about that event
available to cd, the workflow pipeline was a bit brittle.

This change disassociates ci and cd workflows to allow for an
independent configuration of trigger events.

The test jobs, which used to be defined in ci, are now in a
separate workflow file _test.yml that can be included in both ci
and cd workflows.

**Changes in ci**
- Only defines trigger events and permissions, the "meat" of ci is
  defined in the called _test.yml now.
- No longer triggers on tag pushes, this was only needed for cd.

**Changes in cd**
- Now triggers directly on tag pushes instead of (cd)-workflow_run.
- Calls _test.yml, and require successful run before build/release.
  (`needs: test` replaces `if: ...`)
- Changes variable names about pushed tag that triggered the event.

Signed-off-by: Lukas Puehringer <[email protected]>

* Restrict cd permissions to contents: write

This is the minimum permission needed to create/modify GH releases.

Signed-off-by: Lukas Puehringer <[email protected]>

* Adopt recent ci/cd changes in release docs

Since #1971 ci and cd workflows run independently of each other,
each of them also calling the test workflow.

This patch updates RELEASE.md to match the new setup.

It also fixes a (twice) broken link.

Signed-off-by: Lukas Puehringer <[email protected]>

* build(deps): bump securesystemslib[crypto,pynacl] from 0.22.0 to 0.23.0

Bumps [securesystemslib[crypto,pynacl]](https://github.com/secure-systems-lab/securesystemslib) from 0.22.0 to 0.23.0.
- [Release notes](https://github.com/secure-systems-lab/securesystemslib/releases)
- [Changelog](https://github.com/secure-systems-lab/securesystemslib/blob/master/CHANGELOG.md)
- [Commits](https://github.com/secure-systems-lab/securesystemslib/compare/v0.22.0...v0.23.0)

---
updated-dependencies:
- dependency-name: securesystemslib[crypto,pynacl]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* build: fix success message in verify_release

Prior to #1946 the verify_release script was successful if both PyPI
and GitHub release artifacts matched the local build.

Now, if the `--skip-pypi` option is provided, the script can also
be successful if only the GitHub release artifacts match the local
build.

This commit splits the final success message in two separate
success messages, one for PyPI and one for GitHub.

Signed-off-by: Lukas Puehringer <[email protected]>

* build: add 'gpg sign' option to verify_release

Add option to sign locally built release artifacts with gpg,
if they match the downloaded artifacts from GitHub, PyPI.

Signed-off-by: Lukas Puehringer <[email protected]>

* doc: describe signatures creation in RELEASE.md

Mention how to use verify_release with the recently added --sign
option to create signatures for a verified release.

Signed-off-by: Lukas Puehringer <[email protected]>

* build: minor style/wording fixes in verify_release

Co-authored-by: Joshua Lock <[email protected]>
Signed-off-by: Lukas Puehringer <[email protected]>

* python-tuf 1.1.0

* Update Changelog
* bump version

Signed-off-by: Jussi Kukkonen <[email protected]>

* verify_release: Tweak pip download

It seems --no-deps does not work as it used to (and actually installs
all build dependencies). This is very bad because verify_release also
uses "--no-binary :all:" leading to actually _building_ all build
dependencies from source.

Use "--no-binary tuf" instead: build dependencies will still be
installed (into a working environment) but at least they won't be built
from source.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Update maintainers permission checklist

* Release permissions are now controlled in GitHub release environment
* It is no longer required for a releasing maintainer to have PyPI
  permissions

Signed-off-by: Jussi Kukkonen <[email protected]>

* build: Pin hatchling version

Building a specific release with specific build tools feels like correct
choice for reproducibility in general. It's also practically required
as the hatchling version is embedded in the WHEEL file: this means
updating the build tool modifies the resulting build artifact.

Pin hatchling version. This version should be kept up-to-date: my
working assumption is that Dependabot will handle it.

Signed-off-by: Jussi Kukkonen <[email protected]>

* build(deps): bump cryptography from 36.0.2 to 37.0.1

Bumps [cryptography](https://github.com/pyca/cryptography) from 36.0.2 to 37.0.1.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/36.0.2...37.0.1)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* build(deps): bump mypy from 0.942 to 0.950

Bumps [mypy](https://github.com/python/mypy) from 0.942 to 0.950.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.942...v0.950)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* lint: Stop using requests annotations

requests project does not maintain annotations: typeshed project tries
to do it for them, and releases the annotations as "types-requests".

There's two main problems:
* typeshed releases constantly: this means a lot of test dependency
  updates
* typeshed releases are not tagged in git: updates are impossible to
  review

The benefit we get from types-requests is minimal as there is very
little requests-related code and it does not change often.

Remove annotations to lower the test dependency update churn.

Signed-off-by: Jussi Kukkonen <[email protected]>

* Metadata API: Checking for None instead of falsyness

Fixes #1937

Initialization of unrecognized_fields acts surprisingly when the input
container is empty. Hence, We're checking for None instead of falsyness.

Signed-off-by: Abhisman Sarkar <[email protected]>

* build(deps): bump pylint from 2.13.7 to 2.13.8

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.7 to 2.13.8.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.7...v2.13.8)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* docs: Add a blog post about ngclient design

Try to explain some decisions made in ngclient.

Signed-off-by: Jussi Kukkonen <[email protected]>

* docs: Add doc links to ngclient blog post

Signed-off-by: Jussi Kukkonen <[email protected]>

* blog: Update post date, update sloccount

Signed-off-by: Jussi Kukkonen <[email protected]>

* chore: update the workflow responsible for notifying of new TUF spec release

Signed-off-by: Radoslav Dimitrov <[email protected]>

* chore: limit the permissions for the job calling the version check workflow

Signed-off-by: Radoslav Dimitrov <[email protected]>

* chore: test with issues:read permission

Signed-off-by: Radoslav Dimitrov <[email protected]>

Co-authored-by: Martin Vrachev <[email protected]>
Co-authored-by: Jussi Kukkonen <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Lukas Pühringer <[email protected]>
Co-authored-by: Jussi Kukkonen <[email protected]>
Co-authored-by: Joshua Lock <[email protected]>
Co-authored-by: Ivana Atanasova <[email protected]>
Co-authored-by: Ofek Lev <[email protected]>
Co-authored-by: Ivana Atanasova <[email protected]>
Co-authored-by: Kairo de Araujo <[email protected]>
Co-authored-by: Abhisman Sarkar <[email protected]>
Co-authored-by: Ari <[email protected]>
Co-authored-by: Marina Moore <[email protected]>
Co-authored-by: Lukas Pühringer <[email protected]>
  • Loading branch information
15 people authored May 10, 2022
1 parent e6b41d5 commit 7de325d
Show file tree
Hide file tree
Showing 199 changed files with 9,245 additions and 42,131 deletions.
17 changes: 0 additions & 17 deletions .fossa.yml

This file was deleted.

6 changes: 3 additions & 3 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Files that will always have LF line endings on checkout.
tests/repository_data/** text eol=lf

# All JSON files will always have LF line endings on checkout.
# This prevents git replacing line endings with CRLF on Windows.
*.json text eol=lf
12 changes: 10 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,16 @@
version: 2
updates:
- package-ecosystem: pip

- package-ecosystem: "pip"
directory: "/"
schedule:
interval: daily
interval: "daily"
time: "10:00"
open-pull-requests-limit: 10

- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
time: "10:00"
open-pull-requests-limit: 10
90 changes: 90 additions & 0 deletions .github/workflows/_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
on:
workflow_call:
# Permissions inherited from caller workflow


jobs:
tests:
name: Tests
strategy:
fail-fast: false
# Run regular TUF tests on each OS/Python combination, plus special tests
# (sslib master) and linters on Linux/Python3.x only.
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
os: [ubuntu-latest, macos-latest, windows-latest]
toxenv: [py]
include:
- python-version: 3.x
os: ubuntu-latest
toxenv: with-sslib-master
experimental: true
- python-version: 3.x
os: ubuntu-latest
toxenv: lint

env:
# Set TOXENV env var to tell tox which testenv (see tox.ini) to use
# NOTE: The Python 2.7 runner has two Python versions on the path (see
# setup-python below), so we tell tox explicitly to use the 'py27'
# testenv. For all other runners the toxenv configured above suffices.
TOXENV: ${{ matrix.toxenv }}

runs-on: ${{ matrix.os }}

steps:
- name: Checkout TUF
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@98f2ad02fd48d057ee3b4d4f66525b231c3e52b6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements*.txt'

- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade tox coveralls
- name: Run tox (${{ env.TOXENV }})
# See TOXENV environment variable for the testenv to be executed here
run: tox

- name: Publish on coveralls.io
# A failure to publish coverage results on coveralls should not
# be a reason for a job failure.
continue-on-error: true
# TODO: Maybe make 'lint' a separate job instead of case handling here
if: ${{ env.TOXENV != 'lint' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COVERALLS_FLAG_NAME: ${{ runner.os }} / Python ${{ matrix.python-version }} / ${{ env.TOXENV }}
COVERALLS_PARALLEL: true
# Use cp workaround to publish coverage reports with relative paths
# FIXME: Consider refactoring the tests to not require the test
# aggregation script being invoked from the `tests` directory, so
# that `.coverage` is written to and .coveragrc can also reside in
# the project root directory as is the convention.
run: |
cp tests/.coverage .
coveralls --service=github --rcfile=tests/.coveragerc
coveralls-fin:
# Always run when all 'tests' jobs have finished even if they failed
# TODO: Replace always() with a 'at least one job succeeded' expression
if: always()
needs: tests
runs-on: ubuntu-latest
container: python:3-slim
steps:
- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade coveralls
- name: Finalize publishing on coveralls.io
continue-on-error: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: coveralls --finish
87 changes: 87 additions & 0 deletions .github/workflows/cd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: CD
concurrency: cd

on:
push:
tags:
- v*

permissions:
contents: write

jobs:
test:
uses: ./.github/workflows/_test.yml

build:
name: Build
runs-on: ubuntu-latest
needs: test
outputs:
release_id: ${{ steps.gh-release.outputs.id }}
steps:
- name: Checkout release tag
uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
with:
ref: ${{ github.event.workflow_run.head_branch }}

- name: Set up Python
uses: actions/setup-python@0ebf233433c08fb9061af664d501c3f3ff0e9e20
with:
python-version: '3.x'

- name: Install build dependency
run: python3 -m pip install --upgrade pip build

- name: Build binary wheel and source tarball
run: python3 -m build --sdist --wheel --outdir dist/ .

- id: gh-release
name: Publish GitHub release candiate
uses: softprops/action-gh-release@1e07f4398721186383de40550babbdf2b84acfc5
with:
name: ${{ github.ref_name }}-rc
tag_name: ${{ github.ref }}
body: "Release waiting for review..."
files: dist/*

- name: Store build artifacts
uses: actions/upload-artifact@6673cd052c4cd6fcf4b4e6e60ea986c889389535
# NOTE: The GitHub release page contains the release artifacts too, but using
# GitHub upload/download actions seems robuster: there is no need to compute
# download URLs and tampering with artifacts between jobs is more limited.
with:
name: build-artifacts
path: dist

release:
name: Release
runs-on: ubuntu-latest
needs: build
environment: release
steps:
- name: Fetch build artifacts
uses: actions/download-artifact@fb598a63ae348fa914e94cd0ff38f362e927b741
with:
name: build-artifacts
path: dist

- name: Publish binary wheel and source tarball on PyPI
uses: pypa/gh-action-pypi-publish@717ba43cfbb0387f6ce311b169a825772f54d295
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}

- name: Finalize GitHub release
uses: actions/github-script@9ac08808f993958e9de277fe43a64532a609130e
with:
script: |
await github.rest.repos.updateRelease({
owner: context.repo.owner,
repo: context.repo.repo,
release_id: '${{ needs.build.outputs.release_id }}',
name: '${{ github.ref_name }}',
body: 'See [CHANGELOG.md](https://github.com/' +
context.repo.owner + '/' + context.repo.repo +
'/blob/${{ github.ref_name }}/docs/CHANGELOG.md) for details.'
})
102 changes: 7 additions & 95 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,104 +1,16 @@
name: Run TUF tests and linter
name: CI

on:
push:
branches:
- develop

pull_request:
workflow_dispatch:

jobs:
build:
strategy:
fail-fast: false
# Run regular TUF tests on each OS/Python combination, plus special tests
# (sslib master) and linters on Linux/Python3.x only.
matrix:
python-version: [3.6, 3.7, 3.8, 3.9]
os: [ubuntu-latest, macos-latest, windows-latest]
toxenv: [py]
include:
- python-version: 3.x
os: ubuntu-latest
toxenv: with-sslib-master
experimental: true
# TODO: Change to 3.x once pylint fully supports Python 3.9
- python-version: 3.8
os: ubuntu-latest
toxenv: lint

env:
# Set TOXENV env var to tell tox which testenv (see tox.ini) to use
# NOTE: The Python 2.7 runner has two Python versions on the path (see
# setup-python below), so we tell tox explicitly to use the 'py27'
# testenv. For all other runners the toxenv configured above suffices.
TOXENV: ${{ matrix.toxenv }}

runs-on: ${{ matrix.os }}

steps:
- name: Checkout TUF
uses: actions/checkout@v2

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
permissions:
contents: read

- name: Find pip cache dir
id: pip-cache
run: echo "::set-output name=dir::$(pip cache dir)"

- name: pip cache
uses: actions/cache@v2
with:
# Use the os dependent pip cache directory found above
path: ${{ steps.pip-cache.outputs.dir }}
# A match with 'key' counts as cache hit
key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}
# A match with 'restore-keys' is used as fallback
restore-keys: ${{ runner.os }}-pip-

- name: Install dependencies
run: |
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade tox coveralls
- name: Run tox (${{ env.TOXENV }})
# See TOXENV environment variable for the testenv to be executed here
run: tox

- name: Publish on coveralls.io
# A failure to publish coverage results on coveralls should not
# be a reason for a job failure.
continue-on-error: true
# TODO: Maybe make 'lint' a separate job instead of case handling here
if: ${{ env.TOXENV != 'lint' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COVERALLS_FLAG_NAME: ${{ runner.os }} / Python ${{ matrix.python-version }} / ${{ env.TOXENV }}
COVERALLS_PARALLEL: true
# Use cp workaround to publish coverage reports with relative paths
# FIXME: Consider refactoring the tests to not require the test
# aggregation script being invoked from the `tests` directory, so
# that `.coverage` is written to and .coveragrc can also reside in
# the project root directory as is the convention.
run: |
cp tests/.coverage .
coveralls --service=github --rcfile=tests/.coveragerc
coveralls-fin:
# Always run when all 'build' jobs have finished even if they failed
# TODO: Replace always() with a 'at least one job succeeded' expression
if: always()
needs: build
runs-on: ubuntu-latest
container: python:3-slim
steps:
- name: Finalize publishing on coveralls.io
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade coveralls
coveralls --finish
jobs:
test:
uses: ./.github/workflows/_test.yml
55 changes: 55 additions & 0 deletions .github/workflows/maintainer-permissions-reminder.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: Maintainer review reminder

on:
schedule:
- cron: '10 10 10 2 *'
workflow_dispatch:

permissions:
issues: write

jobs:
file-reminder-issue:
name: File issue to review maintainer permissions
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@9ac08808f993958e9de277fe43a64532a609130e
with:
script: |
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: "Yearly maintainer permissions review",
body: `
This is a checklist for evaluating python-tuf maintainer accounts and permissions. This issue is automatically opened once a year.
### Tasks
1. Update this list to include any new services
2. Evaluate the accounts and permissions for each service on the list. Some rules of thumb:
* Critical services should have a minimum of 3 _active_ maintainers/admins to prevent project lockout
* Each additional maintainer/admin increases the risk of project compromise: for this reason permissions should be removed if they are no longer used
* For services that are not frequently used, each maintainer/admin should check that they really are still able to authenticate to the service and confirm this in the comments
3. Update MAINTAINERS.txt to reflect current permissions
4. (Bonus) Update significant contributors in README.md#acknowledgements
### Critical services
* [ ] **PyPI**: maintainer list is visible to everyone at https://pypi.org/project/tuf/
* Only enough maintainers and org admins to prevent locking the project out
* [ ] **GitHub**: release environment reviewers listed in https://github.com/theupdateframework/python-tuf/settings/environments
* Maintainers who can approve releases to PyPI
* [ ] **GitHub**: permissions visible to admins at https://github.com/theupdateframework/python-tuf/settings/access
* "admin" permission: Only for maintainers and org admins who do project administration
* "push/maintain" permission: Maintainers who actively approve and merge PRs (+admins)
* "triage" permission: All contributors trusted to manage issues
### Other
* [ ] **ReadTheDocs**: admin list is visible to everyone at https://readthedocs.org/projects/theupdateframework/
* [ ] **Coveralls**: everyone with github "admin" permissions is a Coveralls admin: https://coveralls.io/github/theupdateframework/python-tuf
`
})
console.log("New issue created.")
Loading

0 comments on commit 7de325d

Please sign in to comment.