Skip to content

Latest commit

 

History

History
374 lines (298 loc) · 20.2 KB

README.md

File metadata and controls

374 lines (298 loc) · 20.2 KB
Raw

szurubooru-toolkit icon

szurubooru-toolkit

Python package and script collection to manage your szurubooru image board.

☑️ Requirements

In order to run the included scripts, a Python release >=3.8 and the configuratrion file config.toml is required.

The config.toml file needs to be always present in your current working directory from where you are executing the scripts. You can find a sample config file in the GitHub repository of this package.

🛠️ Installation

This package is available on PyPI and can be installed with pip: pip install szurubooru-toolkit

Alternatively, you can clone the package from GitHub and set everything up with Poetry. In the root directory of this repository, execute poetry install.

Please note this package requires Tensorflow for Deepbooru tagging and other required packages which do take up quite a lot of disk space (>400MB). A future release of this package will offer an option without the need to install Tensorflow if you don't want to use Deepbooru tagging.

Docker Instructions

If you would like to run the toolkit in a Docker container instead, follow the instructions below.

  1. Copy docker-compose.yml to the location where you want to run the toolkit.
  2. Copy config_sample.toml to the same location, renaming to config.toml and replacing with your configuration.
  3. Copy crontab_sample to the same location, renaming to crontab and adding the commands you would like to run regularly. An example command is provided in crontab_sample.
  4. Make sure to set the src_path option in config.toml to use /szurubooru-toolkit/upload_src. If you're using a different directory than upload_src, you may need to update the docker-compose.yml binding to be something like ./uploads:/szurubooru-toolkit/uploads, and set /szurubooru-toolkit/uploads as the src_path option instead.
  5. If you would like to use deepbooru or tag files, create misc/deepbooru and/or misc/tags in the same location and follow the instructions linked below
  6. Run touch szurubooru_toolkit.log in the same location to create a file for the log. You may need to set the log location to /szurubooru-toolkit/szurubooru_toolkit.log in config.toml
  7. Use docker-compose up or docker-compose up -d to start the container, or start the container in the background, respectively. You can use docker-compose logs or docker-compose logs -f to inspect the container output, which will include szuru toolkit's output if you append your cron jobs with >/proc/1/fd/1 2>&1 like in the example job.
  8. If you just want to run a one-time command, leave the crontab file blank and start the container with docker-compose up -d, taking note of the container_name option in docker-compose.yml. Then, you can run commands inside of the running container like this: docker exec -it container_name auto-tagger, replacing container_name with the container name.
  9. If you would like the container to run a one-time command and then quit with docker-compose.yml, add a command configuration like this.

📝 User configuration

Make your changes in the config_sample.toml file provided in the git repo and rename it to config.toml afterwards.

Note that path names have to be specified with forward slashes (/) if you're using Windows.

ℹ️ config.toml reference
Section Option Description Examples/Default
szurubooru url The URL of your szurubooru "https://szuru.example.com"
szurubooru username Username which connects to the szuruboori API "my_szuru_user"
szurubooru api_token API token of username. Generate one in szurubooru from Account > Login tokens > Create token "my_api_token"
szurubooru public If your szurubooru is reachable over the internet false
auto_tagger saucenao_api_token In case you want to increase your daily query limit "my_saucenao_api_token"
auto_tagger saucenao_enabled Set this to false and deepbooru_enabled to true if you only want to tag with Deepbooru true
auto_tagger deepbooru_enabled If enabled, tag the post with Deepbooru if no tags with SauceNAO were found false
auto_tagger deepbooru_model Path to the Deepbooru model "./misc/deepbooru/model-resnet_custom_v3.h5"
auto_tagger deepbooru_threshold Define how accurate the matched tag from Deepbooru has to be "0.7"
auto_tagger deepbooru_forced Always tag with SauceNAO and Deepbooru false
auto_tagger deepbooru_set_tag Tag Deepbooru post with tag deepbooru true
auto_tagger hide_progress Set this to true to hide the progress bar false
danbooru user Danbooru user "None"
danbooru api_key Danbooru api key "None"
gelbooru user Gelbooru user "None"
gelbooru api_key Gelbooru api key "None"
konachan user Konachan user "None"
konachan password Konachan password "None"
yandere user Yandere user "None"
yandere password Yandere password "None"
pixiv user Pixiv user. Currently not being used. "None"
pixiv password Pixiv password. Currently not being used. "None"
pixiv token Pixiv token. Currently not being used. "None"
twitter user_id The user id which should be queried. "None"
twitter consumer_key See https://developer.twitter.com/en/docs/authentication/oauth-1-0a "None"
twitter consumer_secret See https://developer.twitter.com/en/docs/authentication/oauth-1-0a "None"
twitter access_token See https://developer.twitter.com/en/docs/authentication/oauth-1-0a "None"
twitter access_token_secret See https://developer.twitter.com/en/docs/authentication/oauth-1-0a "None"
upload_media src_path Every valid media file under this dir (recursively) will get uploaded "/local/path/to/upload/dir"
upload_media hide_progress Set this to true to hide the progress bar false
upload_media cleanup Set this to true if images in the src_path should be deleted after upload false
upload_media tags These tags will get set for all uploaded posts. Separate them by a comma. ["tagme", "tag1", "tag2", "tagN"]
upload_media auto_tag Set this to true if you want your post to be automatically tagged after upload false
upload_media max_similarity Adjust this value to ignore posts if a similar post higher than the threshold has already been uploaded "0.99"
upload_media convert_to_jpg Convert images to JPG to save disk space. This won't overwrite the source files and only affects the uploaded image. Images might slip through identical post check. false
upload_media convert_quality Only images above this threshold will be converted to jpg if convert_to_jpg is True. "3MB|500KB"
upload_media shrink Set to true to shrink images to shrink_dimensions based on shrink_threshold below. Images might slip through identical post check. false
upload_media shrink_threshold Images which total pixel size exceeds this treshold will be resized to shrink_size. E.g. 2000x3000 results in 6000000. "6000000"
upload_media shrink_dimensions Set the max value for width/height. Keeps aspect ratio. E.g. 2000x4000 results in 700x1400, 4000x2000 in 1400x700 (with "1400x1400"). "2500x2500"
import_from_booru deepbooru_enabled Apply Deepbooru tagging additionally besides fetched tags from Booru false
import_from_booru hide_progress Set this to true to hide the progress bar false
import_from_twitter saucenao_enabled Tag posts with SauceNAO false
import_from_twitter deepbooru_enabled Tag posts with Deepbooru false
import_from_twitter hide_progress Set this to true to hide the progress bar false
create_tags hide_progress Set this to true to hide the progress bar false
delete_posts hide_progress Set this to true to hide the progress bar false
reset_posts hide_progress Set this to true to hide the progress bar false
tag_posts hide_progress Set this to true to hide the progress bar false
logging log_enabled If logging to a log file should be enabled false
logging log_file Specify the path of the log file "C:/Users/Foo/Desktop/szurubooru_toolkit.log"
logging log_level Specify the log level. DEBUG logs the most information "DEBUG"|"INFO"|"WARNING"|"ERROR"|"CRITICAL"
logging log_colorized If the log file should be colorized. Requires compatible viewer (e.g. less -r <log_file>). true

Creating a SauceNAO account and an API key is recommended. Please consider supporting the SauceNAO team as well by upgrading your plan. With a free plan, you can request up to 200 posts in 24h.

For Deepbooru support, download the current release here (v3-20211112-sgd-e28) and extract the contents of the zip file. Specify the path of the folder with the extracted files in deepbooru_model. Please note that you have to set deepbooru_enabled if you want to use it.

📃 Scripts

Following scripts are currently available:

  • auto-tagger: Batch tagging of posts with SauceNAO and Deepbooru
  • create-tags: Batch creation of tags with their categories
  • delete-posts: Batch delete of posts
  • import-from-booru: Batch importing of posts with their tags from various Boorus
  • import-from-twitter: Batch importing of Twitter favorites
  • reset-posts: Batch resetting of posts (remove tags and sources)
  • upload-media: Batch upload of media files from local source folder
  • tag-posts: Manual batch tagging

See the descriptions below on how to use them.

If you installed this package with pip, you can generally just call the scripts from your shell (if your $PATH is set correctly).

If you cloned the repo from GitHub, prefix the above scripts with poetry run, e.g. poetry run auto-tagger "date:today". Note that your current working directory has to be the the root of the GitHub project.

⚙️ auto-tagger

This script accepts a szurubooru query as a user input, fetches all posts returned by it and attempts to tag it using SauceNAO/Deepbooru.

If no matches from SauceNAO were found, the script keeps the previously set tags of the post and additionally appends the tag tagme.

You can set deepbooru_enabled to true in your config.toml file. In that case, the script falls back to tag posts with the supplied Deepbooru model. If you only want to use Deepbooru, set deepbooru_enabled to true and saucenao_enabled to false. If you want to use SauceNAO and Deepbooru, set following options to true: saucenao_enabled, deepbooru_enabled and deepbooru_forced.

Usage

usage: auto-tagger [-h] [--sankaku_url SANKAKU_URL] [--add-tags ADD_TAGS] [--remove-tags REMOVE_TAGS] query

This script will automagically tag your szurubooru posts based on your input query.

positional arguments:
  query                 Specify a single post id to tag or a szuru query. E.g. "date:today tag-count:0"

optional arguments:
  -h, --help            show this help message and exit
  --sankaku_url SANKAKU_URL
                        Fetch tags from specified Sankaku URL instead of searching SauceNAO.
  --add-tags ADD_TAGS   Specify tags, separated by a comma, which will be added to all posts matching your query
  --remove-tags REMOVE_TAGS
                        Specify tags, separated by a comma, which will be removed from all posts matching your query

Examples

  • auto-tagger "date:today tag-count:0"
  • auto-tagger "date:2021-04-07"
  • auto-tagger "tagme"
  • auto-tagger "id:100..111"
  • auto-tagger "id:100,101"
  • auto-tagger --add-tags "foo,bar" --remove-tags "baz" "tagme"

If we want to tag a single post, we can omit the keyword id in our query:

  • auto-tagger 100

Alternatively, we can tag a single post and specify --sankaku_url to fetch the tags from the supplied URL (untested with current release):

auto-tagger --sankaku_url https://chan.sankakucomplex.com/post/show/<id> 100

This is especially useful since Sankaku has changed their API and aggregator sites like SauceNAO don't have the latest results there.

↘️ import-from-booru

This scripts imports posts and their tags from various Boorus that matched your input query.

In the config.toml file, you can set if the post should be additionally tagged with Deepbooru and if the progress bar should be shown. Since this script is using the upload-media script to upload the post, following settings apply from the upload-media section:

  • max_similarity
  • convert_to_jpg
  • convert_threshold
  • shrink
  • shrink_threshold
  • shrink_dimensions

Usage

usage: import-from-booru [-h] [--limit LIMIT] {danbooru,gelbooru,konachan,yandere,all} query

This script downloads and tags posts from various Boorus based on your input query.

positional arguments:
  {danbooru,gelbooru,konachan,yandere,all}
                        Specify the Booru which you want to query. Use all to query all Boorus.
  query                 The search query for the posts you want to download and tag

optional arguments:
  -h, --help            show this help message and exit
  --limit LIMIT         Limit the search results to be returned (default: 100)

Examples

  • import-from-booru danbooru "tag1 tagN"
  • import-from-booru yandere "tag1 tag2 -tagN"
  • import-from-booru all "tag1 -tagN"

Note that if you specify all to download from all Boorus, you are limited to two tags because free Danbooru accounts are limited to two tags per query. If you have a Gold/Platinum account, set your credentials in config.toml. Note that it's currently untested if the script will work with upgraded accounts.

🕊️ import-from-twitter

This script fetches media files from your Twitter likes, uploads and optionally tags them.

⚠️ OAuth 1.0a credentials are required to read the likes from a user. See https://developer.twitter.com/en/docs/authentication/oauth-1-0a on how to generate them.

The user_id can be converted on sites like https://tweeterid.com/. If you configured above credentials, you can also get your own ID from the access_token, which is in following format: <user_id>-<random_string>

In the config.toml file, you can set if the post should be additionally tagged with SauceNAO or Deepbooru and if the progress bar should be shown. Since this script is using the upload-media script to upload the post, following settings apply from the upload-media section:

  • max_similarity
  • convert_to_jpg
  • convert_threshold
  • shrink
  • shrink_threshold
  • shrink_dimensions

Usage

usage: import-from-twitter [-h] [--limit LIMIT] [--user-id USER_ID]

This script fetches media files from your Twitter likes, uploads and optionally tags them.

optional arguments:
  -h, --help         show this help message and exit
  --limit LIMIT      Limit the amount of Twitter posts returned (default: 25)
  --user-id USER_ID  Fetch likes from the specified user id

Examples

  • import-from-twitter --limit 50
  • import-from-twitter --limit 50 --user_id 123

↗️ upload-media

This script searches through your specified upload folder in the config.toml file for any image/video files and uploads them to your szurubooru.

Usage

  • upload-media

🏷️ tag-posts

Usage

usage: tag-posts [-h] [--add-tags ADD_TAGS] [--remove-tags REMOVE_TAGS] [--mode {append,overwrite}] query

This script will tag your szurubooru posts based on your input arguments and mode.

positional arguments:
  query                 The search query for the posts you want to tag

optional arguments:
  -h, --help            show this help message and exit
  --add-tags ADD_TAGS   Specify tags, separated by a comma, which will be added to all posts matching your query.
  --remove-tags REMOVE_TAGS
                        Specify tags, separated by a comma, which will be removed from all posts matching your query.
  --mode {append,overwrite}
                        Set mode to overwrite to remove already set tags, set append to keep them (default: append).
  --update-implications
                        Fetches all tags from the posts matching the query and updates them if tag implications are missing (default: False)

Examples

  • tag-posts --add-tags "foo,bar" "date:2021-04-07"
  • tag-posts --add-tags "foo,bar" --mode "overwrite" "foo bar"
  • tag-posts --add-tags "foo,bar" --remove-tags "baz" "foo"

🔄 reset-posts

Usage

usage: reset-posts [-h] [--except-ids EXCEPT_IDS] [--add-tags ADD_TAGS] query

This script will remove tags and sources from your szurubooru posts based on your input search query.

positional arguments:
  query                 The search query for the posts you want to reset.

optional arguments:
  -h, --help            show this help message and exit
  --except-ids EXCEPT_IDS
                        Specify the post ids, separated by a comma, which should not be reset. Example: --except-ids "3,4,5"
  --add-tags ADD_TAGS   Specify tags, separated by a comma, which will be added to all posts matching your query after resetting.

Examples

  • reset-posts "foobar"
  • reset-posts --add-tags "tagme" "foobar"
  • reset-posts --add-tags "tagme,foo" "foobar"
  • reset-posts --except-ids "2,4" --add-tags "tagme,foo" "foobar"

🗑️ delete-posts

Usage

usage: delete-posts [-h] [--except-ids EXCEPT_IDS] query

This script will delete your szurubooru posts based on your input search query.

positional arguments:
  query                 The search query for the posts you want to delete.

optional arguments:
  -h, --help            show this help message and exit
  --except-ids EXCEPT_IDS
                        Specify the post ids, separated by a comma, which should not be deleted. Example: --except-ids "3,4,5

Examples

  • delete-posts "id:10,11,100,23"
  • delete-posts --except-ids "12,23,44" "id:10..50"

🏷️ create-tags

Usage

usage: create-tags [-h] [--tag-file TAG_FILE] [--query QUERY] [--min-post-count MIN_POST_COUNT] [--limit LIMIT] [--overwrite]

This script will read the tags from specified file and create them in your szurubooru.

optional arguments:
  -h, --help            show this help message and exit
  --tag-file TAG_FILE   Specify the local path to the file containing the tags and categories. If specified, ignores other arguments.
  --query QUERY         Search for specific tags (default: "*").
  --min-post-count MIN_POST_COUNT
                        The minimum amount of posts the tag should have been used in (default: 10).
  --limit LIMIT         The amount of tags that should be downloaded. Start from the most recent ones (default: 100).
  --overwrite           Overwrite tag category if the tag already exists.

If no tag_file is specified, the script will download the most recent 100 tags from Danbooru which have been used at least ten times.

You can use tools like Grabber to download a tag list from common boorus.

The tag_file has to be in following format:

<tag_a>,<category_name>
<tag_b>,<category_name>
<tag_..n>,<category_name>

The category has to be created beforehand manually (e.g. default, artist, series, character and meta).

Examples

  • create-tags
  • create-tags --query genshin* --overwrite
  • create-tags --tag-file tags.txt

ℹ️ Image credit

GitHub repo icon: Code icons created by Smashicons - Flaticon