feat: add findex impl - client and server sides #15

Manuthor · 2024-10-18T08:00:30Z

Prerequisites:

cloudproof_findex: add a basic REST client Findex implementation (old Findex Cloud not reusable): feat(findex): add basic REST client instanciation - without Findex Cloud cloudproof_rust#72
- Adapt cloudproof_rust for Findex v7.
  - A POC has been done for Findex v7 slightly changing the rest-interface for the new REST basic interface (without the custom Findex Cloud AuthorizationToken). Old work on Findex Cloud is kept for non regression.
- Replace rust macro with classical Rust impl (vscode does not support macros)
- Make Findex REST Backend instantiable (and usable) in other bindings (FFI, Wasm, pyo3)
- Adapt other cloudproof_* langages repos and make the CI OK
findex repository:
- Release the findex 7.0.0to be used incloudproof_rust`
- [Findex v6] - Add missing serialization impls for structs Tokens, TokenWithEncryptedValueList and TokenToEncryptedValueMap: fix: Add missing serialization impls for structs Tokens, TokenWithEncryptedValueList and TokenToEncryptedValueMap. findex#88
- Eventually release the 6.0.1 to be used in cloudproof_rust
Auth0:
- Create a proper findex-server Auth0 tenant
- Configure it, create a test JWT token for user [email protected]
- Use this token in tests

Client:

Server:

Closes #1
Closes #4
Closes #6
Closes #9
Closes #3

crate/client/src/findex_rest_client.rs

crate/cli/src/lib.rs

crate/client/src/findex_rest_client.rs

.rustfmt.toml

tbrezot · 2024-10-22T06:44:52Z

crate/cli/src/actions/findex/mod.rs

+    #[clap(long, short = 'k')]
+    pub key: String,


Since we don't want the key to be client-side, this should be replaced by a JWT, or at least an enum Key | JWT

Do you mean the Findex key only persist in a KMS?

Exactly. We need to decide whether we want to allow both flows or only the KMS-as-crypto-oracle one. Supporting both will need to switch to Findex-v7 which is more modular.

crate/cli/src/actions/findex/mod.rs

tbrezot · 2024-10-22T07:13:16Z

crate/cli/src/actions/findex/index.rs

+        let config = Configuration::Rest(
+            findex_rest_client.client.clone(),
+            findex_rest_client.server_url.clone(),
+            findex_rest_client.server_url.clone(),
+        );


This seems strange to me.

If I follow you well, upon calling a CLI command, you call this function process with the client object (idk what it is). But since I doubt you serialize and store this client upon returning from the CLI, it means it is instantiated each time the CLI is called. Moreover, it seems to me that only one Findex operation can be performed per CLI invocation. Therefore I conclude that this clone is useless since the caller of process will not reuse this FindexClient.

Please, point to me where my reasoning is flawed.

More generally, I do not understand where the necessity of this FindexClient object and this action mechanism comes from. Imho, the ideally simplistic flow is:

user invokes the CLI with some arguments

the CLI parses its conf/ENV variables to findex the server URL. If it cannot be found, it becomes a required argument.

the CLI parses the provided arguments, returning an error upon reading an invalid one. There are 3 (or 4) possible commands: search, insert, delete (additionally compact). The validity of the rest of the argument depends on the identity of the command, and thus the main body of the CLI is just a match on the parsed command.

after parsing all arguments, if no error was thrown, then it means the parsed command will be performed: the CLI therefore instantiates a Findex object.

The CLI runs the appropriate Findex command with its arguments, and returns the result to the user.

As I said, this is rather simplistic and may need to be developed, for example in order to add a non-Findex command like it seems you are planning to. However, I don't think we need dedicated types with their own proceed functions and to pass around some FindexClient object.

The FindexClient is just a HTTPS Client which is initialized once and then pass to clap actions. About this clone, your reasoning is correct, I've removed the shared reference and passed directly the object to be consumed by actions. Yes the CLI instantiates a HTTPS client each time and does not chain Findex operations (that's what we want).
About the CLI flow, since the 4 operations search, add, delete and compact have different arguments, the 4 corresponding process functions (which are basic functions) are used:

to parse those arguments

to give them properly to Findex API

to display to stdout the Findex output

tbrezot · 2024-10-22T07:23:45Z

crate/client/src/findex_rest_client.rs

@@ -21,11 +21,13 @@ use crate::{
 #[derive(Clone)]
 pub struct FindexClient {
    pub server_url: String,
-    client: Client,
+    pub client: Client,


It seems this is just an HTTPS client. I would suggest not to store it in some FindexClient. This would make everything simpler to read (this HTTPS client has nothing to do with Findex, and could be used by any other protocol built over HTTPS).

FindexClient could be renamed at least I suppose

tbrezot · 2024-10-22T07:36:43Z

crate/server/src/config/command_line/db.rs


    /// The url of the database for findex-redis
    #[clap(
        long,
        env = "FINDEX_SERVER_DATABASE_URL",
-        required_if_eq_any([("database_type", "redis-findex")])
+        required_if_eq_any([("database_type", "redis-findex")]),
+        default_value = "redis://localhost:6379"


I wouldn't use a default value here, but return an error upon failure to parse this information.

the idea was to give the format of the URL redis://XXXX:6379

crate/server/src/core/implementation.rs

crate/server/src/database/redis/mod.rs

tbrezot · 2024-10-22T08:08:00Z

Prerequisites:

* `cloudproof_findex`: add a basic REST client Findex implementation (old Findex Cloud not reusable): [feat(findex): add basic REST client instanciation - without Findex Cloud cloudproof_rust#72](https://github.com/Cosmian/cloudproof_rust/pull/72)
  
  * [x]  Create the `findex-cloud` feature and keep `rest-interface` for the new REST basic interface (without the custom Findex Cloud `AuthorizationToken`). Old work on Findex Cloud is kept for non regression.
  * [x]  Replace rust macro with classical Rust impl (vscode does not support macros)
  * [x]  Make Findex REST Backend instantiable (and usable) in other bindings (FFI, Wasm, pyo3)
  * [ ]  Adapt other `cloudproof_*` langages repos and make the CI OK
  * [ ]  Release the `cloudproof_findex` 6.1.0`to be used in`findex-server`

* `findex` repository:
  
  * [x]  Add missing serialization impls for structs `Tokens`, `TokenWithEncryptedValueList` and `TokenToEncryptedValueMap`: [fix: Add missing serialization impls for structs `Tokens`, `TokenWithEncryptedValueList` and `TokenToEncryptedValueMap`. findex#88](https://github.com/Cosmian/findex/pull/88)
  * [ ]  Release the 6.0.1 to be used in `cloudproof_rust`

* `Auth0`:
  
  * [x]  Create a proper `findex-server` Auth0 tenant
  * [ ]  Configure it, create a test JWT token for user `[email protected]`
  * [ ]  Use this token in tests

Client:

* [x]  Authentication JWT + PKCS12. Clap action `login` and `logout`

* [x]  Clap action `server-version`

* Findex
  
  * Indexing using REST Findex client interface:
    
    * [x]  with an arbitrary dataset
    * [ ]  with any dataset on clap input
  * [ ]  Searching words

* Encrypt indexed data using KMS:
  
  * [ ]  hybrid encryption KEM - DEM with:
    KEM: RFC5649 (done by KMS with ckms)
    DEM: AES GCM  (done locally by ckms)

* [ ]  Merge CLI in `ckms`

Server:

* [x]  Authentication JWT + PKCS12. Reuse KMS Auth middleware

* [x]  Add enpoints for indexing/searching with findex (server "just" INSERT and SELECT data, does not know the Findex master key)

* [x]  Database trait for all databases impls

* [ ]  Database support:
  
  * [x]  Redis
  * [ ]  Sqlite

* [ ]  Authorization: Multiple clients to index and search on the same datasets. Adapt endpoints to index and search using an `ID`

* [ ]  Convert TOML conf in JSON

UI

* User interface should allowed at least:
  
  * [ ]  Authenticating to the server (JWT at least)
  * [ ]  Indexing arbitrary datasets
  * [ ]  Searching words
  * [ ]  Save encrypted either on a S3 storage or on a managed database (another Redis, ...)

Great summary. Some questions:

how are you planning to introduce the KMS in the decryption process? It seems to me that it would require forking Findex v6 since it hard-codes the use of AES/SHA3 in the implementation of the EncryptedValue. The v7 is more flexible since the crypto component is abstracted behind a trait and can therefore be selected at compile-time.
do you really want to "merge the cli in ckms"?
do you really want a JSON config file ? JSON is overly complex for the task imho
I didn't get your authorization bullet: is it the separation between different indexes?

HatemMn · 2024-10-24T13:01:54Z

@Manuthor some updates of the cosmian all big :

We will keep the findex v6 of cloudproof for this PR
According to Bruno UI is not a priority

HatemMn

The following points were done with my commits, edit them according to nesecssituy

Create a proper findex-server Auth0 tenant
Configure it, create a test JWT token for user [email protected]
Use this token in tests

.pre-commit-config.yaml

crate/test_server/src/test_server.rs

Co-authored-by: Hatem M'naouer <[email protected]>

Manuthor · 2024-10-27T19:40:30Z

The following points were done with my commits, edit them according to nesecssituy
* Create a proper findex-server Auth0 tenant

* Configure it, create a test JWT token for user [[email protected]](mailto:[email protected])

* Use this token in tests

done

HatemMn

Test

* ci: fix * test: authentication failing * test: ignore auth tests * test: re-enable test server auth tests * test: re-enable client auth tests * test: disable client auth tests * test: do not clear db * chore: rename access to permission * refactor: check_permission * test: try mt tests * chore: remove all lints * test: remove dependency_on_unit_never_type_fallback * test: make default server port to 6666 * fix: lint

Manuthor force-pushed the findex_v6 branch 3 times, most recently from 3fbfcb8 to f8a13c0 Compare October 18, 2024 08:19

Manuthor commented Oct 18, 2024

View reviewed changes

crate/client/src/findex_rest_client.rs Outdated Show resolved Hide resolved

Manuthor commented Oct 18, 2024

View reviewed changes

crate/cli/src/lib.rs Outdated Show resolved Hide resolved

feat: add findex impl - client and server sides

9027235

Manuthor force-pushed the findex_v6 branch from f8a13c0 to 9027235 Compare October 18, 2024 08:25

Manuthor commented Oct 18, 2024

View reviewed changes

crate/client/src/findex_rest_client.rs Show resolved Hide resolved

Manuthor requested review from bgrieder, HatemMn, ccorsin and tbrezot October 18, 2024 10:24

feat: add CSV datasets support in CLI

7302140

tbrezot reviewed Oct 22, 2024

View reviewed changes

HatemMn requested changes Oct 25, 2024

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

crate/test_server/src/test_server.rs Show resolved Hide resolved

crate/test_server/src/test_server.rs Show resolved Hide resolved

Manuthor force-pushed the findex_v6 branch from 2c6f243 to 3271a1a Compare October 27, 2024 04:47

fix: rename IndexAction to AddAction and flatten Findex actions

f45e876

Manuthor force-pushed the findex_v6 branch from 3271a1a to f45e876 Compare October 27, 2024 05:18

Manuthor and others added 2 commits October 27, 2024 06:19

Update .pre-commit-config.yaml

fcf2bed

Co-authored-by: Hatem M'naouer <[email protected]>

fix(auth0): use proper findex-server token

fb190a6

test: add JWT authentication tests

7fa5dd2

HatemMn approved these changes Oct 28, 2024

View reviewed changes

Manuthor added 5 commits October 30, 2024 10:55

feat: add Delete Findex operation

f0afc9d

fix: remove Secret for now

818c300

fix: prepare sqlite impl

248ef34

feat: add authorization access mechanism

98936b5

feat: revoke role

0b367d6

Manuthor and others added 3 commits November 3, 2024 08:01

docs: cli access

147efd1

chore: rename FindexClient to RestClient

e7dc054

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add findex impl - client and server sides #15

feat: add findex impl - client and server sides #15

Manuthor commented Oct 18, 2024 •

edited

Loading

tbrezot Oct 22, 2024

Manuthor Oct 27, 2024 •

edited

Loading

tbrezot Oct 28, 2024

tbrezot Oct 22, 2024

Manuthor Oct 27, 2024

tbrezot Oct 22, 2024

Manuthor Oct 27, 2024

tbrezot Oct 22, 2024

Manuthor Nov 5, 2024

tbrezot commented Oct 22, 2024

HatemMn commented Oct 24, 2024

HatemMn left a comment

Manuthor commented Oct 27, 2024

HatemMn left a comment

feat: add findex impl - client and server sides #15

Are you sure you want to change the base?

feat: add findex impl - client and server sides #15

Conversation

Manuthor commented Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

Manuthor Oct 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tbrezot commented Oct 22, 2024

HatemMn commented Oct 24, 2024

HatemMn left a comment

Choose a reason for hiding this comment

Manuthor commented Oct 27, 2024

HatemMn left a comment

Choose a reason for hiding this comment

Manuthor commented Oct 18, 2024 •

edited

Loading

Manuthor Oct 27, 2024 •

edited

Loading