release: twitch better profile 0.2.0 #9
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ith-auth feat: settings with new SettingsOption UDT
ci: set up lint pipeline
## Motivation One of the features which we want to provide to the extension is `watched time` under `channel` and `category` in the same way that Twitch delivers but faster. This PR brings the queries/models needed for this operation, where: - Each client will send a heartbeat request every minute to the consumer - The client will be able to consume this data anytime by REST operations ### Implementation Initially it would be only about a "how many minutes and messages" each person has, so the modeling started using mostly [COUNTERS TYPE](https://opensource.docs.scylladb.com/stable/using-scylla/counters.html): ```cql CREATE TABLE twitch.user_metrics_v1 ( user_id int, messages_count counter, watch_time_in_minutes counter, PRIMARY KEY (user_id) ) ``` Counters allow atomic operations, so it will never overwrite the same data. Also, there's a delay for each partition of **1 minute**, so it will not be a problem at all. To be fair, we still lacking authentication, but there's an workaround here: - Every time that a user authenticates at `tbp-platform`, they generate a new token - This token is sent to `tbp-consumer` under a "secret" request (i'm sorry about that) - For every user related request at `tbp-consumer`, there's a validation on the token. ```cql CREATE TABLE twitch.user_tokens_v1 ( access_token text, user_id int, PRIMARY KEY (access_token) ) ``` ```rust pub async fn is_authenticated( session: &Arc<CachingSession>, req: HttpRequest, ) -> Option<UserToken> { let header = req.headers().get("Authorization"); let header = header?.to_str(); if header.is_err() { return None; } let response = UserToken { access_token: header.unwrap().to_string(), ..Default::default() } .maybe_find_by_primary_key() .execute(session) .await .unwrap(); response.as_ref()?; Some(response.unwrap()) } ``` With the "AUTHENTICATION" done, for every `heartbeat` sent by the user, the app increments: - How many minutes the user watched a specific channel - How many minutes the user watched a specific category ```cql CREATE TABLE twitch.user_metrics_by_channel_v1 ( user_id int, channel_id text, messages_count counter, minutes_watched counter, PRIMARY KEY ((user_id, channel_id)) ) CREATE TABLE twitch.user_metrics_by_category_v1 ( user_id int, category_id text, messages_count counter, minutes_watched counter, PRIMARY KEY ((user_id, category_id)) ) ``` Since Counters can't be part of clustering key, create a "leaderboard" as asked wouldn't be a possibility, so at the same operation I added two new tables for both categories leaderboard on "most watched" category: ```cql CREATE TABLE twitch.user_most_watched_channels_leaderboard_v1 ( user_id int, minutes_watched int, channel_id text, PRIMARY KEY (user_id, minutes_watched, channel_id) ) WITH CLUSTERING ORDER BY (minutes_watched DESC, channel_id ASC); CREATE TABLE twitch.user_most_watched_category_leaderboard_v1 ( user_id int, minutes_watched int, category_id text, PRIMARY KEY (user_id, minutes_watched, category_id) ) WITH CLUSTERING ORDER BY (minutes_watched DESC, category_id ASC); ``` For each new heartbeat, the application erases the latest position under the clustering keys (minutes_watched, (category_id/channel_id)) and insert the new one! Since the clustering order is **DESC** we will always have a leaderboard being generated. All writing operations was set to `LocalOne` for a better latency. > [!WARNING] > All this authentication methods should be better implemented with ACTUAL Middlewares and a task will be assigned for it, but I'm not planning to make it LOL Please, review.
## First Approach This commit introduces a throttling mechanism to the heartbeat endpoint. A new `ThrottleState` struct is added to the `AppState` to keep track of the last request time for each user-channel pair. If a user tries to send a heartbeat for the same channel within a minute, a 429 Too Many Requests response is returned. The throttling state is shared across all requests using an `Arc<Mutex<ThrottleState>>`. Signed-off-by: Daniel Boll <[email protected]> ## Second Approach After realizing that would be quite similar to use ScyllaDB for this task, we did a new table for Throttling using TTL in the insertion of the query. The partition key is made of: - `request_uri`: /some/route - `user_id`: user related to the Bearer Token sent in the request - `content`: a random string that will be used to differentiate this request from the others. ```sql CREATE TABLE twitch.throttle_v1 ( uri text, user_id int, content text, updated_at timestamp, PRIMARY KEY ((uri, user_id, content), updated_at) ) WITH CLUSTERING ORDER BY (updated_at ASC) INSERT INTO throttle_v1 (uri, user_id, content, updated_at) VALUES (?, ?, ?, ?) USING TTL ? ``` Each request can have a dynamic TTL based on the user necessity: ```rust impl Throttle { pub async fn insert_throttle(&self, connection: &CachingSession, ttl: i32) -> anyhow::Result<()> { let query = Query::new(INSERT_THROTTLE_WITH_TTL); connection .execute(query, ( &self.uri, &self.user_id, &self.content, &self.updated_at, ttl, ), ) .await?; Ok(()) } } ``` --------- Signed-off-by: Daniel Boll <[email protected]> Co-authored-by: danielhe4rt <[email protected]>
## Motivation As discussed at "basementdevs/twitch-better-profile#60", all the related colors or effects would be synced in new UDT (User Defined Types). Either `settings_v1` than `settings_by_username_v1` got updated with the new fields. ```cql /** New types */ CREATE TYPE twitch.color_option ( name text, slug text, translation_key text, hex text, ); /** New types */ CREATE TYPE twitch.effect_option ( name text, slug text, translation_key text, class: text hex text, ); CREATE TABLE twitch.settings_v1 ( user_id int, is_developer boolean, locale text, color: frozen<color_option>, effect: frozen<effect_option>, occupation frozen<settingoptions>, pronouns frozen<settingoptions>, timezone text, updated_at timestamp, username text, PRIMARY KEY (user_id) ) ```
…workers (#6) ## Summary This PR introduces improvements aimed at enhancing the developer experience (DX): - **Makefile**: Improved for better usability with clearer and more understandable commands. - **Docker Compose Setup**: Added to streamline the development environment setup, though the database is not included directly in this setup. Instead, the database configuration is managed by a separate project [ws-scylla](https://github.com/gvieira18/ws-scylla). - **Max Workers**: Introduced the ability to specify the maximum number of workers. If `MAX_WORKERS` is not provided, it defaults to using all available cores. - **Additional Improvements**: - Added **EditorConfig** for consistent code formatting. - Simplified versioning by removing `current_schema`, reducing the annoyance of unnecessary updates every time `make migrate` is run. > [!NOTE] > A README about how to set up the system is pending and will be done later. This could be expanded to cover other services as well. - basementdevs/twitch-better-profile#54
…idation (#7) ## Summary This PR includes a series of updates and improvements aimed at refining both the development and production environments: - **Environment Handling**: - Improved environment variable loading and usage within various commands. - Removed the validation check for the `.env` file. This change was necessary because in production environments using Docker, the `.env` file is not utilized, which was causing unnecessary errors. - **Dockerfile Enhancements**: - Improved the Dockerfile to better accommodate both development and production environments, ensuring smoother transitions between these setups. - **App Configuration**: - Updated application settings specifically for the development environment, optimizing for local development workflows. - **CI/CD**: - Added Dependabot configuration for weekly dependency updates, enhancing project maintenance and security. relates to basementdevs/twitch-better-profile#56
## Motivation
After the latest changes of tbp-extension, now you can have different
profiles per each stream you visit.
By default, you will be always sending an
`/settings/{username}?channel_id=any-twitch-channel` together with the
username, if the user has an dedicated profile for that stream, it will
be prioritized, if not will fallback to the global profile.
Didn't wrote any test here tbh but it would be good to have in the
future.
---------
Co-authored-by: Daniel Boll <[email protected]>
gvieira18
approved these changes
Sep 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.