Fix Specter cache #136
Fix Specter cache #136
Conversation
| @@ -116,6 +116,7 @@ def _maybe_print_to_console_and_file(self, | |||
| paper_id = prediction_json['paper_id'] | |||
| cache_key = paper_id + "_" + str(self._metadata[paper_id]['mdate']) | |||
| self._redis_con.tensorset(key=cache_key, tensor=np.array(prediction_json['embedding'])) | |||
| self._redis_con.expire(cache_key, 2629746) ## Expire after 1 month | |||
There was a problem hiding this comment.
can we this in the config file?
There was a problem hiding this comment.
Hmm it can be set in the model_params of the config.json but I don't think the models will have access to the Flask config
| @@ -211,7 +212,6 @@ def set_archives_dataset(self, archives_dataset): | |||
| "authors": [profile_id], | |||
| "mdate": pub_mdate | |||
| } | |||
| self._remove_keys_from_cache(publication["id"]) | |||
There was a problem hiding this comment.
so this is the call that is taking time? tensorset is fast enough?
There was a problem hiding this comment.
Yes, tensorset is fast but the _remove_keys_from_cache function scans through all the keys to find matches which is quite slow
There was a problem hiding this comment.
How much is slow? I'm still confused as to why redis is slow here. Are there too many keys in the database?
I'm not sure what the self._remove_keys_from_cache method does, but its performance also depends on whether SCAN or KEYS is used.
This PR no longer attempts to find and remove all old instances of publications in the Redis cache, and instead sets an expiration date whenever inserting into Redis.