Skip to content

Commit

Permalink
Version 592
Browse files Browse the repository at this point in the history
  • Loading branch information
@hydrusnetwork
hydrusnetwork committed Oct 2, 2024
1 parent 7e4b17a commit f7b4d42
Show file tree
Hide file tree
Showing 148 changed files with 7,113 additions and 6,064 deletions.
85 changes: 48 additions & 37 deletions docs/changelog.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,54 @@ title: Changelog
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).

## [Version 592](https://github.com/hydrusnetwork/hydrus/releases/tag/v592)

### misc

* the 'read' autocomplete dropdown has a new one-click 'clear search' button, just beside the favourites 'star' menu button. the 'empty page' favourite is removed from new users' defaults
* in an alteration to the recent Autocomplete key processing, Ctrl+c/Ctrl+Insert _will_ now propagate to the results list if you currently have none of the text input selected (i.e. if it would have been a no-op on the text input, we assume you wanted whatever is selected in the list)
* in the normal thumbnail/viewer menu and _review services_, the 'files' entry is renamed to 'locations'. this continues work in the left hand button of the autocomplete dropdown where you set the 'location', which can be all sorts of complicated things these days, rather than just 'file service key selector'. I don't think I'll rename 'my files' or anything, but I will try to emphasise this 'locations' idea more when I am talking about local file domains etc.. in other places going forward; what I often think of as 'oh yeah the files bit' isn't actually referring to the files themselves, but where they are located, so let's be precise
* last week's tag pair filtering in _tags->migrate tags_ now has 'if either the left or right of the pair have count', and when you hit 'Go' with any of the new count filter checkboxes hit, the preview summary on the yes/no confirmation dialog talks about it
* any time a watcher subject is parsed, if the text contains non-decoded html entities (like `>`), they are now auto-converted to normal chars. these strings are often ripped from odd places and are only used for user display, so this just makes that simpler
* if you are set to remove trashed files from view, this now works when the files are in multpile local file domains, and you choose 'delete from all local file services', and you are looking at 'all my files' or a subset of your local file domains
* we now log any time (when the client is non-idle) that a database job's work inside the transaction wrapper takes more than 15 seconds to complete
* fixed an issue caused by the sibling or parents system doing some regen work at an unlucky time

### default downloaders

* thanks to user help, the derpibooru post parser now additionally grabs the raw markdown of a description as a second note. this catches links and images better than the html string parse. if you strictly only want one of these notes, please feel free to dive into _network->downloaders->defailt import options_ for your derpi downloader and try to navigate the 'note import options' hell I designed and let me know how it could be more user friendly

### parsing system

* added a new NESTED formula type. this guy holds two formulae of any type internally, parsing the document with the first and passing those results on to the second. it is designed to solve the problem of 'how do I parse this JSON tucked inside HTML' and _vice versa_. various encoding stuff all seems to be handled, no extra work needed
* added Nested formula stuff to the 'how to make a downloader' help
* made all the screenshot in the parsing formula help clickable
* renamed the COMPOUND formula to ZIPPER formula
* all the 'String Processor' buttons across the program now have copy and paste buttons, so it is now easy to duplicate some rules you set up
* in the parsing system, sidecar importer, and clipboard watcher, all strings are now cleansed of errant 'surrogate' characters caused by the source incorrectly providing utf-16 garbage in a utf-8 stream. fingers crossed, the cleansing here will actually _fix_ problem characters by converting them to utf-8, but we'll see
* thanks to a user, the JSON parsing system has a new 'de-minify json' parsing rule, which decompresses a particular sort of minified JSON that expresses multiply-referenced values using list positions. as it happened that I added NESTED formulae this week, I wonder if we will migrate this capability to the string processing system, but let's give it time to breathe

### client api

* fixed the permission check on the new 'get file/thumbnail local path' commands--due to me copy/pasting stupidly, they were still just checking 'search files' perm
* added `/get_files/local_file_storage_locations`, which spits out the stuff in _database->move media files_ and lets you do local file access _en masse_
* added help and a unit test for this new command
* the client api version is now 72

### some security/library updates

* the 'old' OpenCV version in the `(a)dvanced` setup, which pointed to version 4.5.3.56, which had the webp vulnerability, is no longer an option. I believe this means that the program will no longer run on python 3.7. I understad Win 7 can run python 3.8 at the latest, so we are nearing the end of the line on that front
* the old/new Pillow choice in `(a)dvanced` setup, which offered support for python 3.7, is removed
* I have added a new question to the `(a)dvanced` venv setup to handle misc 'future' tests better, and I added a new future test for two security patches for `setuptools` and `requests`:
* A) `setuptools` is updated to 70.3.0 (from 69.1.1) to resolve a security issue related to downloading packages from bad places (don't think this would ever affect us, but we'll be good)
* B) `requests` is updated to 2.32.3 (from 2.31.0) to resolve a security issue with verify=False (the specific problem doesn't matter for us, but we'll be good)
* if you run from source and want to help me test, you might like to rebuild your venv this week and choose the new future choice. these version increments do not appear to be a big deal, so assuming no problems I will roll these new libraries into a 'future' test build next week, and then into the normal builds a week after

### boring code cleanup

* did a bunch more `super()` refactoring. I think all `__init__` is now converted across the program, and I cleared all the normal calls in the canvas and media results panel code too
* refactored `ClientGUIResults` into four files for the core class, the loading, the thumbnails, and some menu gubbins. also unified the mish-mash of `Results` and `MediaPanel` nomenclature to `MediaResultsPanel`

## [Version 591](https://github.com/hydrusnetwork/hydrus/releases/tag/v591)

### misc
Expand Down Expand Up @@ -337,40 +385,3 @@ title: Changelog
* the `psutil` library is now technically optional. it is still needed for a bunch of normal operations like 'is the client currently running?' and 'how much free space is there on this drive?', but if it is missing the client will now boot and try to muddle through anyway
* did more multi-column list 'select, sort, and scroll' tech for: the url class parameters list; the url class links list; the external launch paths list; the tag suggestion related tags namespace lists; import folder filename tagging options; manage custom network context headers; the string to string match widget list; the string match to string match widget list; the edit subscription queries list; and more of the edit subscriptions subscription list, including the merge and separate actions
* updated the QuickSync help a little, clarifying it is only useful for _new, empty_ clients

## [Version 582](https://github.com/hydrusnetwork/hydrus/releases/tag/v582)

### fixes

* fixed an issue where setting a file 'collect' was not automatically sorting the collected objects internally properly. normally when you collect, each collected object is supposed to be sorted internally by filesize or namespace or whatever--this is working again
* fixed a weird internal error state in the import folders manager where it could get confused about and throw an error regarding the import folders' next work times if an internal update notification occured during an import folder working
* fixed a typo error with the shortcut set 'special duplicate' button
* fixed pasting new query texts into the manage subscriptions dialog when one of the pasted texts resurrects a DEAD query. my new summary generation text was handling the DEAD report wrong!

### misc

* the advanced 'all deleted files' service, which is mostly just used for behind the scenes caching calculations, is renamed to 'deleted from anywhere'. the related 'regen->all deleted files' database command is also moved to 'check and repair->sync combined deleted files'
* the edit tag filter panel's 'load' button now shows all the current tag repositories' tag filters
* when you hit ctrl-enter on some tags (or otherwise trigger a linked remove+add action) in an active search list (e.g. top-left on a search page), which causes those tags to invert and thus sometimes sorted to a different position, the current selection now propagates through the inversion, with the keyboard focus moved to the post-topmost item. so, you can now basically hit ctrl+enter twice for a no-op
* fixed the paste button in the new 'purge tags' dialog
* thanks to a user, we have a new 'Purple' stylesheet
* I tweaked the some default stylesheet colours and think I fixed the display of the 'valid/invalid' controls you sometimes see (for instance in the new regex input, which goes green/red) for dark mode stylesheets that don't define colours for these. previously, the dark mode text, usually a light grey, was being washed out by the default green

### custom colours in QSS

* you can now set the _options->colours_ colours in a QSS stylesheet! if you are a stylesheet maker, check the default_hydrus.qss file to see how it works--it is the same deal as the animation scanbar previously
* the options in _options->colours_ remain, but they are now wrapped in a 'overwrite your stylesheet with these colours' checkbox _for now_. existing users are going to be set to 'yes overwrite', so nothing will suddenly change, but new users are going to default to using whatever the current QSS says. in future, I may collapse the light/darkmode distinction into one option set; I may morph it into a "colour highly rated files' thumbnail borders gold" dynamic options system; I may simply delete the whole thing and replace it with in-client QSS editing or something. not sure, so let's see how it goes and how Qt 7's darkmode stuff turns out.
* I have pasted the hydrus default darkmode colours into all the other stylesheets that come with the program, so new users selecting a darkmode style are going to get something reasonable out of the box rather than the previous ugly clash. the users who made the original stylesheets are welcome to figure out better colours and send them in
* if you are not set to override with the custom colours in _options->colours_, then hitting _help->darkmode_ now gives you a popup telling you what is going on

### sidecar UI

* the four 'edit sidecars' panels (under manual imports, import folders, manual exports, and export folders), which use paths and media as sources respectively, now have test panels to review the current sidecar route you have set up. they use up to 25 rows of example file paths/media from the actual thing you are working on. it provides a live update of what the sources you set up will load, so you know you have the json parse or .txt separator set up correct
* for the export folders case, there is a button in the panel 'edit export folders' panel to populate the text context, under your control, since this involves a potentially slow file search
* there is more to do here. I would like better test panels in the sub-dialogs and I'd like to collapse the related 'eight-nested-dialogs-deep' problem, and in the string processing and parsing UI more generally, but I'm happy with this step forward. let me know where it goes wrong!

### advanced autocomplete logic fixes

* when you enter a wildcard into a Read tag autocomplete, it no longer always delivers the 'always autocompleting' version. so, if you enter `sa*s`, it will suggest `sa*s (wildcard search)` and perhaps `sa*s (any namespace)`, but it will no longer suggest the `sa*s*` variants until you, obviously, actually type that trailing asterisk yourself. I intermittently had no idea what the hell I was doing when I originally developed this stuff
* the 'unnamespaced input gives `(any namespace)` wildcard results' tag display option is now correctly negatively enforced when entering unnamespaced wildcards. previously it was always adding them, and sometimes inserting them at the top of the list. the `(any namespace)` variant is now always below the unnamespaced when both are present
* fixed up a bunch of jank unit tests that were testing this badly
81 changes: 81 additions & 0 deletions docs/developer_api.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2247,6 +2247,87 @@ All thumbnails in hydrus have the .thumbnail file extension and in content are e

This will 400 if the given file type does not have a thumbnail in hydrus, and it will 404 if there should be a thumbnail but one does not exist and cannot be generated from the source file (which probably would mean that the source file was itself Not Found).

### **GET `/get_files/local_file_storage_locations`** { id="get_local_file_storage_locations" }

_Get the local file storage locations, as you see under **database->migrate files**._

Restricted access:
: YES. Search for Files permission and See Local Paths permission needed.

Required Headers: n/a

Arguments: n/a

Response:
: A list of the different file storage locations and what they store.

``` json title="Example response"
{
"locations" : [
{
"path" : "C:\my_thumbs",
"ideal_weight" : 1,
"max_num_bytes": None,
"prefixes" : [
"t00", "t01", "t02", "t03", "t04", "t05", "t06", "t07", "t08", "t09", "t0a", "t0b", "t0c", "t0d", "t0e", "t0f",
"t10", "t11", "t12", "t13", "t14", "t15", "t16", "t17", "t18", "t19", "t1a", "t1b", "t1c", "t1d", "t1e", "t1f",
"t20", "t21", "t22", "t23", "t24", "t25", "t26", "t27", "t28", "t29", "t2a", "t2b", "t2c", "t2d", "t2e", "t2f",
"t30", "t31", "t32", "t33", "t34", "t35", "t36", "t37", "t38", "t39", "t3a", "t3b", "t3c", "t3d", "t3e", "t3f",
"t40", "t41", "t42", "t43", "t44", "t45", "t46", "t47", "t48", "t49", "t4a", "t4b", "t4c", "t4d", "t4e", "t4f",
"t50", "t51", "t52", "t53", "t54", "t55", "t56", "t57", "t58", "t59", "t5a", "t5b", "t5c", "t5d", "t5e", "t5f",
"t60", "t61", "t62", "t63", "t64", "t65", "t66", "t67", "t68", "t69", "t6a", "t6b", "t6c", "t6d", "t6e", "t6f",
"t70", "t71", "t72", "t73", "t74", "t75", "t76", "t77", "t78", "t79", "t7a", "t7b", "t7c", "t7d", "t7e", "t7f",
"t80", "t81", "t82", "t83", "t84", "t85", "t86", "t87", "t88", "t89", "t8a", "t8b", "t8c", "t8d", "t8e", "t8f",
"t90", "t91", "t92", "t93", "t94", "t95", "t96", "t97", "t98", "t99", "t9a", "t9b", "t9c", "t9d", "t9e", "t9f",
"ta0", "ta1", "ta2", "ta3", "ta4", "ta5", "ta6", "ta7", "ta8", "ta9", "taa", "tab", "tac", "tad", "tae", "taf",
"tb0", "tb1", "tb2", "tb3", "tb4", "tb5", "tb6", "tb7", "tb8", "tb9", "tba", "tbb", "tbc", "tbd", "tbe", "tbf",
"tc0", "tc1", "tc2", "tc3", "tc4", "tc5", "tc6", "tc7", "tc8", "tc9", "tca", "tcb", "tcc", "tcd", "tce", "tcf",
"td0", "td1", "td2", "td3", "td4", "td5", "td6", "td7", "td8", "td9", "tda", "tdb", "tdc", "tdd", "tde", "tdf",
"te0", "te1", "te2", "te3", "te4", "te5", "te6", "te7", "te8", "te9", "tea", "teb", "tec", "ted", "tee", "tef",
"tf0", "tf1", "tf2", "tf3", "tf4", "tf5", "tf6", "tf7", "tf8", "tf9", "tfa", "tfb", "tfc", "tfd", "tfe", "tff"
]
},
{
"path" : "D:\hydrus_files_1",
"ideal_weight" : 5,
"max_num_bytes": None,
"prefixes" : [
"f00", "f02", "f04", "f05", "f08", "f0c", "f11", "f12", "f13", "f15", "f17", "f18", "f1a", "f1b", "f20", "f23",
"f25", "f26", "f27", "f2b", "f2e", "f2f", "f31", "f35", "f36", "f37", "f38", "f3a", "f40", "f42", "f43", "f44",
"f49", "f4b", "f4d", "f4e", "f50", "f51", "f55", "f59", "f60", "f63", "f64", "f65", "f66", "f68", "f69", "f6e",
"f71", "f73", "f78", "f79", "f7a", "f7d", "f7f", "f82", "f83", "f84", "f86", "f87", "f88", "f89", "f8f", "f90",
"f91", "f96", "f9e", "fa1", "fa4", "fa5", "fa7", "faa", "fad", "faf", "fb1", "fb9", "fba", "fbb", "fbf", "fc1",
"fc4", "fc7", "fc8", "fcf", "fd2", "fd6", "fd7", "fd8", "fd9", "fdf", "fe2", "fe8", "fe9", "fea", "feb", "fec",
"ff4", "ff7", "ffd", "ffe"
]
},
{
"path" : "E:\hydrus\hydrus_files_2",
"ideal_weight" : 2,
"max_num_bytes": 805306368000,
"prefixes" : [
"f01", "f03", "f06", "f07", "f09", "f0a", "f0b", "f0d", "f0e", "f0f", "f10", "f14", "f16", "f19", "f1c", "f1d",
"f1e", "f1f", "f21", "f22", "f24", "f28", "f29", "f2a", "f2c", "f2d", "f30", "f32", "f33", "f34", "f39", "f3b",
"f3c", "f3d", "f3e", "f3f", "f41", "f45", "f46", "f47", "f48", "f4a", "f4c", "f4f", "f52", "f53", "f54", "f56",
"f57", "f58", "f5a", "f5b", "f5c", "f5d", "f5e", "f5f", "f61", "f62", "f67", "f6a", "f6b", "f6c", "f6d", "f6f",
"f70", "f72", "f74", "f75", "f76", "f77", "f7b", "f7c", "f7e", "f80", "f81", "f85", "f8a", "f8b", "f8c", "f8d",
"f8e", "f92", "f93", "f94", "f95", "f97", "f98", "f99", "f9a", "f9b", "f9c", "f9d", "f9f", "fa0", "fa2", "fa3",
"fa6", "fa8", "fa9", "fab", "fac", "fae", "fb0", "fb2", "fb3", "fb4", "fb5", "fb6", "fb7", "fb8", "fbc", "fbd",
"fbe", "fc0", "fc2", "fc3", "fc5", "fc6", "fc9", "fca", "fcb", "fcc", "fcd", "fce", "fd0", "fd1", "fd3", "fd4",
"fd5", "fda", "fdb", "fdc", "fdd", "fde", "fe0", "fe1", "fe3", "fe4", "fe5", "fe6", "fe7", "fed", "fee", "fef",
"ff0", "ff1", "ff2", "ff3", "ff5", "ff6", "ff8", "ff9", "ffa", "ffb", "ffc", "fff"
]
}
]
}
```

Note that `ideal_weight` and `max_num_bytes` are provided for courtesy and mean nothing fixed. Each storage location might store anything, thumbnails or files or nothing, regardless of the ideal situation. Whenever a folder is non-ideal, the 'move media files' dialog shows "files need to be moved now", but it will still keep doing its thing.

For now, a prefix only occurs in one location, so there will always be 512 total prefixes in this response, all unique. **However, please note that this will not always be true!** In a future expansion, the client will be, on user command, slowly migrating files from one place to another in the background, and during that time there will be multiple valid locations for a file to actually be. When this happens, you will have to hit all the possible locations and test.

Also, it won't be long before the client supports moving to _some_ form of three- and four-character prefix. I am still thinking how this will happen other than it will be an atomic change--no slow migration where we try to support both at once--but it will certainly complicate something in here (e.g. while the prefix may be 'f012', maybe the subfolder will be '\f01\2'), so we'll see.

### **GET `/get_files/render`** { id="get_files_render" }

_Get an image file as rendered by Hydrus._
Expand Down
Loading

0 comments on commit f7b4d42

Please sign in to comment.