Added the KeyPoints TVTensor

[Alexandre-SCHOEPP]

Description

Adds and integrates the KeyPoints TVTensor (requested in #8728), which is a representation of picture-attached points (or vertices) attached point

Details

Inner workings

The KeyPoints represent a tensor of shape [..., 2], which allow for arbitrarily complex structures to be represented (polygons, skeletons, or even SAM-like points prompts). Whenever the __new__ is called, the shape of the source tensor is checked.

Tensors of shape [2] are reshaped to [1, 2], similarly to BoundingBoxes.

KeyPoints, like BoundingBoxes, carry arround a canvas_size attribute which represents the scale of a batch-typical picture.

Kernels

Kernels for all operations should be supported (if I missed one, I will fix this). It merely consists of an adaptation of the code of BoundingBoxes.

Particularities

Maintainers may notice that a TYPE_CHECKING section was added that differs significantly from the implementation:

class KeyPoints(TVTensors)


    if TYPE_CHECKING:
        # EVIL: Just so that MYPY+PYLANCE+others stop shouting that everything is wrong when initializeing the TVTensor
        # Not read or defined at Runtime (only at linting time).
        # TODO: BOUNDING BOXES needs something similar
        def __init__(
            self,
            data: Any,
            *,
            dtype: Optional[torch.dtype] = None,
            device: Optional[Union[torch.device, str, int]] = None,
            requires_grad: Optional[bool] = None,
            canvas_size: Tuple[int, int],
        ):
            ...

I marked this section as EVIL since it is a trick, but it cannot generate vulnerabilities: TYPE_CHECKING is always False at runtime, and only ever True for the linter.

For the last few months, I had issues in my weird PyLance + Mypy mix with BoundingBoxes initialization. No overload is ever detected to match it. By "re-defining" it, I got to it solved on my machine.

Convertors

Added a convertor convert_box_to_points in torchvision.transorfms.v2.functional._meta exported in torchvision.transforms.v2 which (as its name states) converts a [N, 4] BoundingBoxes TVTensor into a [N, 4, 2] KeyPoints TVTensor.

Other changes

For the purposes of my custom type checking, I also changed tv_tensors.wrap to be 3.8-compatible generics.

Since wrap only ever outputs a subclass of its like argument, I used a TypeVar bound to TVTensor to ensure that type-checking passes no matter the checker used.

Methodology

• Formated using ufmt
• Flake8 compliance with line-length 120 enforced by editor
• Documented the classes

Discussion

Since many converters of BoundingBoxes are based on chaning the bboxes to polygons, then operating on the points, I believe that there is a possibility to lower line count and increase reliability for negligeable computational latency cost by using KeyPoints kernels and converting using the method described in the details above

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/8817

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

Hi @Alexandre-SCHOEPP!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!