"Delayed Deduplication" background job

[jennydaman]

Problem: ChRIS Files are Highly Duplicated

Typically, a majority of file data in CUBE is duplicated. The actual amount depends on usage, though I sometimes run feeds where each plugin instance a copy operation.

ChRIS files are copied each time you run pl-dircopy. pl-topologicalcopy, ... meaning the same piece of data consumes many times more storage than its own size, depending on how many times it is used. Futhermore, the only way to rename a file is to create a copy of it (e.g. pl-bulk-rename) which creates more duplicated data.

Alternative Solutions

• (Synchronous) Content-based addressing: all files are hashed, store the data of each unique hash only once. This is very CPU intensive and will introduce large overhead between the time when a plugin instance is done and when its files are ready in the backend.
• Copy operation: create a special operation, "copy," which does a storage-efficient "copy" (akin to hard-link) of a file without duplicating its data. For instance, unextpath parameters and ts-plugins should do these efficient copy operations. However, this isn't easy to take advantage of with ChRIS plugins, we must assume ChRIS plugin developers are not CUBE experts and will not bother to use our special "copy" mechanism where appropriate.

How can we deduplicate data, (A) without the overhead of hashing, and (B) in a way that's transparent (no developmental burden for ChRIS plugin developers)?

Delayed Deduplication Service

Create a ChRIS plugin, pl-hash, which computes hashes. It writes a single file to its output directory. This output files will contain hashes of its input data.

Create the "Periodic Hashing Service" (phs*) which periodically** runs pl-hash on files in CUBE. phs reads from CUBE's database (db) directly, selecting the oldest files which have not been hashed. phs runs pl-hash on these files.

Create the "Periodic Deduplication Service" (pdds) which harvests completed pl-hash jobs. After each pl-hash, compare the new hashes with every existing file hash. For every duplicate found, delete the more recent file from storage, and create a pointer (akin to a hard link) in the db for the more recent file to the older file.

Architecture: phs and pdds connect directly to CUBE's database, storage, and a pfcon.

Footnotes

• *these names are subject to change. Maybe think of something fun, like "Periodic Hash Anything Daemon" phad?
• **Periodically, like a cron job. On Kubernetes, it could very well be a cron job. The periodicity should be configurable, and should be a balance between (idle) CPU v.s. storage efficiency. phs can also be configured to submit its jobs with low queueing priority, or to only run when the compute resource is idle.