Implement new endpoints required by htsget

[aaperis]

Endpoint needed:

• Index file endpoint (should exist, just make sure it works)

  • ([see GET("/s3/*path"] --GET to https://<endpoint-index>/id.bam.bai)

• Encrypted file size (should it contain the size of the crypt4gh header too?)

  • make sure it has the correct endpoint (HEAD to https://<endpoint-file>/id.bam.c4gh)
  • HEAD("/s3/*path") gives decrypted filesize in set in Content-Length in the headers here and below
  • add HEAD("/s3-encrypted/*path"), which shares code, but use ArchiveSize instead
  • add http response header server-additional-bytes, stating the size of the crypt4gh header
  • [optional] add http response header client-additional-bytes, stating the size of the crypt4gh header that the client will get
  • the size should be set in the Content-Length

• [see "/header/*path"] return header size

  • currently we are returning the whole header - change to return only the size
  • <endpoint-crypt4gh-header-size> in ARCHITECHURE.md
  • see code but looks like it returns the actual file
  • make sure that the public keys that from user vs server (htsget) and repository (us) create headers of the same size (since we're using the size of the crypt4gh header in the db) (probably: talk to jocke and stefan).

• (For the underlying BAM/VCF header) Initially full file - implement encrypted header: partial file. starting from scratch, if time allows. [see "/s3-encrypted/*path"]

  • possible to return the whole file, encrypted with the repo's key
  • check if possible to return partial from beginning of the file
  • [have a full file version in branch, test. start with the partial file (start: 0, end: n)].
  • htsget will send a http header range: bytes=0-x, use this rather than the params. The range can be expected to match the cryp4gh block boundaries.

• Get client and server key (they will be used for the re-encryption service)

  • the server-public-key is the one coming from htsget and should be used from the requests that come from htsget
  • the client-public-key is the one used for the final request. it should again be used for the re-encryption, but only for the last request, coming from the user, not the htsget-server
  • extract from http headers
  • make sure we know which requests come from user vs htsget and use correct key
  • send to re-encryption server, get file encrypted with that key
  • usage: when htsget makes the request /s3-encrypted/*path". (implemented later: when client makes the final call, for now use s3/*path)