Add WebAPI for fetching torrent metadata

[Piccirello]

This PR implements two new APIs for fetching a torrent's metadata. The APIs accept a magnet URI, torrent hash, .torrent URL, or uploaded .torrent file, and return the torrent's associated metadata. This PR also modifies the /torrents/add API to support downloading a torrent whose metadata has been previously fetched. The ultimate goal is for the WebUI to provide an Add Torrent experience equivalent to that of the GUI, where content can be reprioritized/unchecked before the torrent is added.

/fetchMetadata API

HTTP request

To request metadata for a torrent, specify the torrent in the source query parameter of a GET request to /api/v2/torrents/fetchMetadata. Torrents are supported in the following formats:

• magnet URI (e.g. magnet:?xt=urn:btih:a8eeefc8a0dc402b24686ddfd775a409fe4b00e0&dn=example)
• hash (e.g. a8eeefc8a0dc402b24686ddfd775a409fe4b00e0)
• .torrent URL (e.g. https://example.com/example.torrent)

HTTP response

Given the asynchronous nature of retrieving metadata, there are two successful HTTP status codes used.

When metadata is requested for a torrent that requires asynchronous background work (i.e. connecting to DHT/peers), the client will receive a 202. A 202 indicates that the request was successful, but additional background work must be completed before a meaningful response can be provided. The response will contain the info hashes, if available, or an empty object.

GET /api/v2/torrents/fetchMetadata?source=abc
HTTP/1.1 202 OK
{
  "hash": string,
  "infohash_v1": string,
  "infohash_v2": string
} | {}

When metadata is available for the torrent, either because the torrent exists in the transfer list or because the metadata has been retrieved from a prior request, the client will receive a 200.

GET /api/v2/torrents/fetchMetadata?source=abc
HTTP/1.1 200 OK
{
  "comment": string,
  "created_by": string,
  "creation_date": int,
  "files": [],
  "hash": string,
  "infohash_v1": string,
  "infohash_v2": string,
  "name": string,
  "piece_size": int,
  "pieces_num": int,
  "private": bool,
  "total_size": int,
  "trackers": [],
  "webseeds": []
}

Retrieved metadata will be cached in the current web session. Subsequent requests performed within the same web session will return the metadata immediately, while other web sessions will be required to reretrieve the torrent's metadata from peers. Once a torrent is added using the cached metadata, the metadata is removed from the cache.

/parseMetadata API

HTTP request

To request metadata for one to several .torrent file(s), you may upload the files to the /parseMetadata API. To do so, submit the file(s) as multipart MIME data. You may use any key for the uploaded value.

To test file upload using curl, specify the -F flag (e.g. curl https://127.0.0.1:8080/api/v2/torrents/parseMetadata --get -F file=@"/root/example.torrent").

HTTP response

This API always responds to successful requests (i.e. valid torrent file(s)) with a 200. The response will contain the full metadata for the uploaded torrent(s). The response object is keyed off of the uploaded file's name.

GET /api/v2/torrents/parseMetadata
HTTP/1.1 200 OK
{
  "example.torrent": {
    "comment": string,
    "created_by": string,
    "creation_date": int,
    "files": [],
    "hash": string,
    "infohash_v1": string,
    "infohash_v2": string,
    "name": string,
    "piece_size": int,
    "pieces_num": int,
    "private": bool,
    "total_size": int,
    "trackers": [],
    "webseeds": []
  }
}

/add API

The existing /add API now supports using the metadata cache that's populated by the new metadata APIs. When specifying a url and/or torrent file to download, the metadata cache is first checked for the torrent. If found, the metadata is used directly from the cache, rather than needing to re-retrieve it. Note that when specifying the name of a .torrent file uploaded via the parseMetadata API, you must prepend file: to the file name. For example, if you uploaded example.torrent to the parseMetadata API, you can add this torrent via the /add API by specifying a url of file:example.torrent.

When metadata is retrieved directly from the cache, you may also specify a new filePriorities parameter. This parameter allows for specifying the file priority of each file in the torrent. This parameter may only be specified when adding a single torrent.

Alternatives:

I explored having the metadata API leave the request open until the metadata was available. Once the metadata was fetched, it would be returned directly in the response of the original request. One downside of this approach is that metadata retrieval can take an arbitrary long amount of time. This could result in torrents whose metadata could never be retrieved via this API (e.g. due to the retrieval taking longer than the client's/reverse proxy's request timeout). This approach would also require some further modification of qBittorrent's web application layer to suppress the default behavior of returning a blank response.

Future work:

• Modify the WebUI to make use of the new /fetchMetadata API. This will likely mean splitting the current Add/Upload Torrent dialog into two dialogs. The first dialog will support specifying the URL(s)/.torrent file(s) to submit, while the second dialog will display the torrent's metadata and allow for modification of file priorities.
• Support downloading the retrieved metadata as a .torrent file (as supported in the GUI)

Closes #20966.

[glassez]

Only preliminary comments after brief review.

[glassez]

@Piccirello
I wonder if it's difficult to implement client side bencode parser in JS? It is very possible that one (or more) already exists.
I believe that if we had used one, we would have got a more universal solution. Since users often add local (for the WebUI) .torrent files, we would not have to send it to the server first for parsing, and then again to add the torrent. As for magnet links, it would also look easier if you sent raw metadata to the client and forget about it. Otherwise, how do you propose to add such torrents (after WebUI user selects file priorities etc.)? Of course, you can store all this metadata in a session... But this seems to be a more cumbersome implementation.

In any case, this part of the API should be thought out, implying the subsequent addition of torrents. Otherwise, we may end up with something little useful in practice.

[Piccirello]

Since users often add local (for the WebUI) .torrent files, we would not have to send it to the server first for parsing, and then again to add the torrent.

I'm not convinced that's the approach we'll eventually take. I can imagine sending the .torrent file once, returning the metadata to the client, and then allowing the torrent to be added without needing to re-send the file (likely by transmitting the info hash).

As for magnet links, it would also look easier if you sent raw metadata to the client and forget about it. Otherwise, how do you propose to add such torrents (after WebUI user selects file priorities etc.)? Of course, you can store all this metadata in a session... But this seems to be a more cumbersome implementation.

To me the session seems like an appropriate place to store this. I don't think the client should be responsible for parsing this data. It would also mean each client (official and unofficial) would need to implement it.

In any case, this part of the API should be thought out, implying the subsequent addition of torrents. Otherwise, we may end up with something little useful in practice.

I agree with this. I'll try to sketch out what the next steps would look like and how this API would be used.

[NikcN22]

@Piccirello I wonder if it's difficult to implement client side bencode parser in JS? It is very possible that one (or more) already exists. I believe that if we had used one, we would have got a more universal solution. Since users often add local (for the WebUI) .torrent files, we would not have to send it to the server first for parsing, and then again to add the torrent. As for magnet links, it would also look easier if you sent raw metadata to the client and forget about it. Otherwise, how do you propose to add such torrents (after WebUI user selects file priorities etc.)? Of course, you can store all this metadata in a session... But this seems to be a more cumbersome implementation.

In any case, this part of the API should be thought out, implying the subsequent addition of torrents. Otherwise, we may end up with something little useful in practice.

I think there is no need to embed the Bencode decoder directly into the client API. This is quite easy to do directly in the “graphical” part. More interesting is the need to provide the ability to assign priority to files in the add method.

[Chocobo1]

I wonder if it's difficult to implement client side bencode parser in JS? It is very possible that one (or more) already exists.

FYI, there certainly exists bencode encoder/decoder library in JS however I'm not aware that they are compatible with bittorrent v2. In the past, I had to mod one to suit my need.

[glassez]

FYI, there certainly exists bencode encoder/decoder library in JS however I'm not aware that they are compatible with bittorrent v2.

"bencode" format is independent from BitTorrent so generic "bencode" decoder should not depend on it too. Or do you refer to torrent file specific parsers?

[Chocobo1]

"bencode" format is independent from BitTorrent so generic "bencode" decoder should not depend on it too.

They had deficiencies in their implementations. Not fully conform with the spec.

[glassez]

"bencode" format is independent from BitTorrent so generic "bencode" decoder should not depend on it too.

They had deficiencies in their implementations. Not fully conform with the spec.

It seems to be the same problem as with bencode editors. I couldn't find BitTorrent independent editor for Linux.

[Piccirello]

I ended up exploring how retrieved metadata would tie into the /add API, resulting in some changes to the /metadata API. Namely, the /metadata API now supports processing multiple sources at once. I've also made the necessary changes to the /add API to support downloading a torrent whose metadata has been previously retrieved via /metadata. This allows for adding the torrent with custom file priorities, which will enable a future PR to modify the WebUI's Add Torrent experience to mimic that of the GUI. PR description has been modified with the full changes.

[glassez]

sources can be delineated by a (non-url encoded) comma

I suppose you're talking about percent-encoding, right?
I don't believe it is valid requirement. IIRC, in URL some of characters MUST be percent-encoded and other CAN be percent-encoded.

[glassez]

When metadata is requested for multiple torrents, a 202 will be returned if any of the torrents requires asycnhronous background work. This means that a 202 may include data for some of the request torrents.

It would look much simpler and more convenient if "metadata" endpoint accepted only single source.

[Piccirello]

sources can be delineated by a (non-url encoded) comma

I suppose you're talking about percent-encoding, right? I don't believe it is valid requirement. IIRC, in URL some of characters MUST be percent-encoded and other CAN be percent-encoded.

, is a reserved character and so it must be percent encoded.

It would look much simpler and more convenient if "metadata" endpoint accepted only single source.

This was my initial approach but I changed it because of how it will integrate with the Add Torrent dialogs in the WebUI. With proper documentation and status codes, I think this API will be easy for consumers/clients to understand.

[glassez]

, is a reserved character and so it must be percent encoded.

Then why do you suggest using it non-encoded?

I changed it because of how it will integrate with the Add Torrent dialogs in the WebUI.

Do "Add Torrent dialogs in the WebUI" really require API endpoint to allow accepting multiple sources at once?

With proper documentation and status codes, I think this API will be easy for consumers/clients to understand.

If "metadata" accepts only single source then 202 status would definitely mean that the response does not contain metadata, and 200 - on the contrary, that the metadata is available in the response.
If "metadata" accepts multiple sources then 202 status is ambiguous and you will still have to check the response body to understand which metadata is immediately available and which is not. In addition, it is possible that one of the sources is invalid. What should you do in this case? In single source approach you could just return error status...

[Piccirello]

@Chocobo1 please review when you have a chance.

[Piccirello]

For folks that would like to test this - I have a branch with a nearly complete WebUI implementation. Lmk and I'll push it up.

[Piccirello]

@qbittorrent/bug-handlers this PR has been in a ready state for several weeks. Can I get a thorough review and/or an approval?

[glassez]

@qbittorrent/bug-handlers this PR has been in a ready state for several weeks. Can I get a thorough review and/or an approval?

It's still on my to-do list. Unfortunately it's hard to review such changes using the mobile web version of GitHub, which I usually use to review the PRs. And there is still not enough time at the computer for it, since there are many other tasks.

[Piccirello]

@qbittorrent/bug-handlers this PR has been in a ready state for several weeks. Can I get a thorough review and/or an approval?

It's still on my to-do list. Unfortunately it's hard to review such changes using the mobile web version of GitHub, which I usually use to review the PRs. And there is still not enough time at the computer for it, since there are many other tasks.

Understood. You've already given several rounds of reviews, so I invite other folks to take a look too.

[Chocobo1]

I only commented on coding style.

[Chocobo1]

Last two comments for coding style.