Add events to QueryBuilder class

[summersab]

• Would provide the functionality to resolve: Add support for database encryption #13103

Summary

This provides events before and after the execution of QueryBuilder objects in order to modify QueryBuilder elements, the query results, or both. While it may seem like this is only useful for niche applications (or potentially dangerous), there are valid instances where intercepting DB queries and results are extremely useful (or even necessary).

Being able to intercept queries and/or results in order to restrict access, redact information, or even generate query-level logs would require the ability to access the QueryBuilder class before and/or after execution.

One application that I have implemented for my organization is adding column-level encryption on a per-user basis. Without being able to access the QueryBuilder class immediately before execution, this integration would be impossible. Once this PR is accepted and my integration is tested more thoroughly in production, I plan to make my code publicly available so administrators may encrypt information in the NC database such as contact and calendar data.

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[nickvergessen]

or even generate query-level logs

This is already available by setting the following config:

server/config/config.sample.php

Lines 2225 to 2232 in dde5c46

	/**
	* Log all queries into a file
	*
	* Warning: This heavily decreases the performance of the server and is only
	* meant to debug/profile the query interaction manually.
	* Also, it might log sensitive data into a plain text file.
	*/
	'query_log_file' => '',

Regarding the events:From my POV this looks like an interesting idea, but it has the option to fail very hard performance-wise.

Regarding column encryption: Also any user that has access to the database mostlikely also has access to the source code, so the encryption is not helping a lot. Which use/attack case are you trying to prevent?

[summersab]

Regarding the events:From my POV this looks like an interesting idea, but it has the option to fail very hard performance-wise.

Yes, this is true - if used incorrectly, these events can be a huge performance failure. However, based on my early usage of this code, there really isn't a noticible performance hit when done well. (Also, let's be honest. Plenty of integrations have the potential for bad performance unless you build them right. Some might be even worse than this).

Regarding column encryption: Also any user that has access to the database mostlikely also has access to the source code, so the encryption is not helping a lot. Which use/attack case are you trying to prevent?

The integration I wrote uses AES_ENCRYPT/AES_DECRYPT (MariaDB) or pgcrypto (PostgreSQL) in combination with the IProvideUserSecretBackend class to encrypt user data in the DB using a user-specific secret. Based on the tables and columns present in a given query, my code adds the required functions to the QueryBuilder object in order to encrypt and decrypt user data. It may seem questionable, but it works quite well, and I have not noticed a performance hit. There really is no other way to accomplish this sort of integration without modifying core files or using events like these.

That said, yes - this is a very unique implementation with very specific use-cases, and it could lead to negative effects. However, those use-cases DO exist. Also, there are other NC APIs that can impact performance even worse when used incorrectly.

Thank you for your feedback. Let me know if you have further questions, comments, or suggestions - happy to discuss!

[nickvergessen]

in combination with the IProvideUserSecretBackend class to encrypt user data in the DB using a user-specific secret

I get that, but the server might need to write data without the user password being available (e.g. in a background job, or when others trigger an update of "your" data. So the server is able to encrypt/decrypt and therefor also the sysadmin and any one that hacks themselves access to the server.
That's the reason why I wonder which of the attack vectors you are trying to prevent.

[summersab]

Well, I decided to use IProvideUserSecretBackend in my implementation (which comes with a number of limitations like you mentioned like background jobs - I have taken this into account with my setup). It is also possible to use something like the NC server secret for encrypting database columns, and all features work as expected.

Database encryption is my particular use-case for this PR, but I am sure there are other reasons why these events might be useful.

[summersab]

@nickvergessen, I took some metrics to address the subject of performance. This is not the most scientific calculation, but it provides a rough estimate. I created a contact in NC and edited it multiple times. Then, I enabled my QueryBuilder events and encryption logic, created a new contact, and edited it the same way. This was done on a pretty small VM with 1vCPU and 8GB RAM. Here are the results:

# of Queries  Avg (Default)    Avg (Encrypted)  Difference         Query
6             0.0009931723277  0.00150179863    0.0005086263021    DELETE FROM `*PREFIX*cards_properties` WHERE (`cardid` = :dcValue1) AND (`addressbookid` = :dcValue2)
36            0.0005187855826  0.00106041299    0.0005416274071    INSERT INTO `*PREFIX*cards_properties` (`addressbookid`, `cardid`, `name`, `value`, `preferred`) VALUES(:dcValue1, :dcValue2, :name, :value, :preferred)
6             0.0003132025401  0.000682870547   0.0003696680069    SELECT `id` FROM `*PREFIX*cards` WHERE (`uri` = :dcValue1) AND (`addressbookid` = :dcValue2)
18            0.0004307428996  0.0009056540097  0.0004749111101    SELECT `id`, `uri`, `lastmodified`, `etag`, `calendarid`, `size`, `calendardata`, `componenttype`, `classification`, `deleted_at` FROM `*PREFIX*calendarobjects` WHERE (`calendarid` = :dcValue1) AND (`uri` = :dcValue2) AND (`calendartype` = :dcValue3)
30            0.0004283030828  0.002046084404   0.001617781321     SELECT `id`, `uri`, `lastmodified`, `etag`, `size`, `carddata`, `uid` FROM `*PREFIX*cards` WHERE (`addressbookid` = :dcValue1) AND (`uri` = :dcValue2) LIMIT 1
5             0.001100826263   0.002524375916   0.001423549652     UPDATE `*PREFIX*cards` SET `carddata` = :dcValue1, `lastmodified` = :dcValue2, `size` = :dcValue3, `etag` = :dcValue4, `uid` = :dcValue5 WHERE (`uri` = :dcValue6) AND (`addressbookid` = :dcValue7)

Again, this is just my use-case for adding Event dispatchers to the QueryBuilder class. There are probably other uses. With that said, it is pretty clear that the performance hit is negligible (when logic is implemented carefully, of course).

[summersab]

What else is required for this PR? I'm not sure how to make CI happy...

[nickvergessen]

The autoloaders are not up to date
Please run: bash build/autoloaderchecker.sh
And commit the result

But the biggest problem is technical. This is a very fragile thing to be added. It comes with great risk (performance and listing to the event while triggering a SQL query will cause an infinite loop) and therefore we are currently unsure if this is the way to go, especially since the target usecase is soo limited. It does not even prevent admins from doing "bad" things, as they have access to the keys as well. So the only thing it would help against is when your database storage is stolen but not the nextcloud one?

[summersab]

I completely understand the concerns regarding the events being very fragile and risky. However, I would like to respectfully push back a little.

Many existing events in NC are fragile, risky, can lead to infinite loops, and allow admins to do bad things. There are events that allow developers to interact with the authentication process, user/group creation, and even password updates. Are there really that many use cases for OCP\Security\Events\GenerateSecurePasswordEvent? As far as I can see, it is only used by the password_policy app. However, it IS available for all developers to use if they wish. That would allow them to do some pretty bad things.

My proposed events really aren't much different. Yes, the use cases are limited, but so are many events. That doesn't mean that there aren't valid use cases.

Yes, directly accessing database queries is generally avoided because it is risky, but there are TONS of internal elements of the NC framework already exposed to developers. Many of them are just as risky.

Yes, this could lead to infinite loops if improperly used, but even the most benign APIs can lead to infinite loops if not used correctly. This has happened to every developer at some point.

I think this is the nature of open source and security. Systems should be as malleable as possible (even if it is risky) so developers don't have to "hack" system internals. In turn, developers should make their code available for public review and scrutiny. With that in mind, which option is more responsible?

1. I could fork NC, edit these classes, and implement my code. This would allow me to move forward, but I would have to edit core files, and it breaks the code signing in NC.
2. I could continue to politely petition for this PR to officially provide access to the QueryBuilder class even though there are risks and very few use cases (ahem - GenerateSecurePasswordEvent). This would make NC more malleable so I can write a proper integration for NC without hacking into core classes.

Either method will accomplish the same goal, but I think the second option is much more responsible. That is why I am pursuing it. I am certainly open to feedback and suggestions, however.

[summersab]

Any thoughts on how to better structure this and still provide the same sort of functionality, @nickvergessen? It's a touchy integration, but it's better than overriding classes in my NC instances. Plus, I could provide my app that enables per-user column encryption, and I think a lot of people would benefit from that (there are open tickets asking for that feature).

[nickvergessen]

You modified 3rdparty? That should not be needed.

On that note, what is needed is bash build/autoload....

[summersab]

@nickvergessen, the postgres11-php8.0 CI test is failing because Postgres fails to respond. I don't think this is an issue with my code - the test just failed to properly run, I believe.

[nickvergessen]

Yeah, seems unrelated

[summersab]

@nickvergessen, looks like I've finally gotten past CI - hurrah!

Sorry for being such a noob on this PR. It's my first PR to a project this complex, so I had a lot to learn.