Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updates for animation support #116

Draft
wants to merge 238 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
238 commits
Select commit Hold shift + click to select a range
f9193c1
smooshed porting work from local dev
dmarx Oct 30, 2022
b066edd
moved api-interfaces
dmarx Oct 31, 2022
c322e9d
Animation test notebook
pharmapsychotic Oct 31, 2022
a9176b4
improve api-interfaces integration (#114)
dmarx Oct 31, 2022
f2ea7fc
checkout to animation branch of submodule
dmarx Oct 31, 2022
a27313e
fix imports and logger
dmarx Oct 31, 2022
1886bac
Animation notebook updates
pharmapsychotic Oct 31, 2022
415353d
Merge branch 'anima' of https://github.com/Stability-AI/stability-sdk…
pharmapsychotic Oct 31, 2022
32622fd
simplified installation
dmarx Oct 31, 2022
5e3c8ae
migrated more utility functions
dmarx Oct 31, 2022
5b039d4
cleanup
dmarx Oct 31, 2022
23b80ff
migrated open_channel
dmarx Oct 31, 2022
146bb8a
fix uninstall step
dmarx Oct 31, 2022
b4db4f9
cleanup
dmarx Oct 31, 2022
537b4c0
added xrform stuff now that interfaces is updated
dmarx Oct 31, 2022
e51ffc7
cleanup
dmarx Oct 31, 2022
8287529
cleanup
dmarx Oct 31, 2022
f4e17f5
added imports for tx migration
dmarx Oct 31, 2022
5bc56db
migrate color match
dmarx Oct 31, 2022
4e9aa5f
add engine_id to tx call signature
dmarx Oct 31, 2022
550274b
removed no-longer-relevant test case
dmarx Nov 1, 2022
3b7631a
cleanup
dmarx Nov 1, 2022
01174fb
migrated util
dmarx Nov 1, 2022
ab1b9fe
added test_img_mix
dmarx Nov 1, 2022
b3e3d01
added tobytes testcase
dmarx Nov 1, 2022
16c458e
added test asset
dmarx Nov 1, 2022
eab2d46
see if we can grab the test asset via glob
dmarx Nov 1, 2022
337b196
syntax error
dmarx Nov 1, 2022
1a51dfc
fixed missing import
dmarx Nov 1, 2022
b0b1ca7
test for image_to_png_bytes
dmarx Nov 1, 2022
c17604a
test image_to_prompt
dmarx Nov 1, 2022
3d11f88
test image_to_prompt_mask
dmarx Nov 1, 2022
c4b34a0
test color_match_from_string
dmarx Nov 1, 2022
d07df99
added border_mode_from_str
dmarx Nov 1, 2022
adb6a00
test border mode
dmarx Nov 1, 2022
4115df5
fixed variable name
dmarx Nov 1, 2022
3fe4df6
differentiate 2d and 3d border mode utils
dmarx Nov 1, 2022
b3b458f
removed duplicate imports
dmarx Nov 1, 2022
ffb95e0
case insensitivity
dmarx Nov 1, 2022
e14fd2f
added engine_id parameters
dmarx Nov 1, 2022
26a16b1
fixed tests
dmarx Nov 1, 2022
b79f881
added some input validation
dmarx Nov 1, 2022
8899c4b
migrate image_gen and image_inpaint
dmarx Nov 1, 2022
c6c8540
straightening out image2prompt stuff
dmarx Nov 1, 2022
c6ca434
fix tests
dmarx Nov 1, 2022
d677575
fixed variable name
dmarx Nov 1, 2022
25d55bc
cleanup
dmarx Nov 1, 2022
3395ff6
deprecate duplicate function
dmarx Nov 1, 2022
eb73ca7
clarified duplicated work
dmarx Nov 1, 2022
ad9e731
assertions code golf
dmarx Nov 1, 2022
3137325
added note, rearranged some stuff
dmarx Nov 1, 2022
8f97aa0
fixed func name
dmarx Nov 1, 2022
b4bcb72
refactored rendering functions
dmarx Nov 1, 2022
e8fe3cd
small fixes to refactor
dmarx Nov 1, 2022
7f56a96
modularize prior frame ops construction
dmarx Nov 1, 2022
a72a387
refactor fixes for prior transforms
dmarx Nov 1, 2022
b509533
added a note
dmarx Nov 1, 2022
206e80e
fixes for video input
dmarx Nov 1, 2022
640e624
added note
dmarx Nov 1, 2022
3f5ebb1
fleshed out test suite, continued refactoring (#118)
dmarx Nov 2, 2022
533f75b
added missing arguments
dmarx Nov 3, 2022
27677fa
added colormatch_op() helper func
dmarx Nov 3, 2022
2ea8fb8
warp op helpers call signature consistency
dmarx Nov 3, 2022
ad8115f
depthcalc_op helper
dmarx Nov 3, 2022
36e7570
fix copypasta bug, added notes and hints
dmarx Nov 3, 2022
27ce459
hints and notes
dmarx Nov 3, 2022
f7b217c
removed unused variable (previously pushed down)
dmarx Nov 3, 2022
20af508
removed unused imports
dmarx Nov 3, 2022
95e951a
more unused imports
dmarx Nov 3, 2022
382dca2
added warplfow_op helper
dmarx Nov 3, 2022
1cec213
notes
dmarx Nov 3, 2022
e186e38
cleanup
dmarx Nov 3, 2022
327df2e
test_warp2d_op
dmarx Nov 3, 2022
4b67468
fixed warp2d tests, added warp3d test
dmarx Nov 3, 2022
75ccf81
removed unused imports, init logger
dmarx Nov 3, 2022
5eab3d7
fixed invalid mode tests cases
dmarx Nov 3, 2022
775bf6f
test_colormatch_op
dmarx Nov 3, 2022
c7b3179
added missing test cases
dmarx Nov 3, 2022
fb19501
fixed tests
dmarx Nov 3, 2022
3245574
fixed argument name
dmarx Nov 3, 2022
087337e
more helpers + parameterized arguments + notebook UI (#121)
dmarx Nov 3, 2022
6bdc9ff
Updates to animation notebook
pharmapsychotic Nov 11, 2022
a1ec60b
Update animation tests
pharmapsychotic Nov 11, 2022
1d59033
Remove local SDK use from notebook,
pharmapsychotic Nov 11, 2022
211c269
Merge pull request #126 from Stability-AI/pharma.anim
dmarx Nov 11, 2022
e618e7a
add init image support
dmarx Nov 12, 2022
874889a
add new arg to test fixture
dmarx Nov 12, 2022
e0691e1
simplify test defaults
dmarx Nov 12, 2022
e7405e7
refactor tests
dmarx Nov 12, 2022
9c771a6
test init_image, elim unneeded fixture
dmarx Nov 12, 2022
4b37b8e
Image.open -> cv2.imread
dmarx Nov 15, 2022
75a8310
fixed syntax error, cleanup
dmarx Nov 15, 2022
f5c62ba
init_image overrides prior frames
dmarx Nov 15, 2022
1938388
Merge pull request #128 from Stability-AI/dmarx.anima.init_image
dmarx Nov 15, 2022
9cf9b79
Big update to animation render loop, diffusion cadence, and cadence i…
pharmapsychotic Nov 17, 2022
a7ff715
Merge branch 'anima_stable' into pharma.anims
pharmapsychotic Nov 17, 2022
f1cc56f
Get tests running again
pharmapsychotic Nov 17, 2022
be0f0d9
Add .vscode to .gitignore
pharmapsychotic Nov 17, 2022
026811c
Fix for tensorizer after updating to latest api-interfaces.
pharmapsychotic Nov 18, 2022
74329ff
Misc updates:
pharmapsychotic Nov 19, 2022
23f4f42
Notebook support for override_settings_path
pharmapsychotic Nov 19, 2022
7b9f7f0
Update so all interpolation modes use same endpoint.
pharmapsychotic Nov 19, 2022
932a056
Update api-interfaces submodule
pharmapsychotic Nov 19, 2022
464524b
Remove some debug code that snuck into a prior commit
pharmapsychotic Nov 19, 2022
ad5705e
Fix for unit tests
pharmapsychotic Nov 19, 2022
ef93233
Merge branch 'main' into pharma.anims
pharmapsychotic Nov 21, 2022
4eb7d3a
Remove duplicate import
pharmapsychotic Nov 21, 2022
630e993
Fix animation resuming
pharmapsychotic Nov 21, 2022
91523c3
Implement init image sizing options
pharmapsychotic Nov 21, 2022
3c652da
Masking!
pharmapsychotic Nov 23, 2022
aa48464
Grab latest api-interfaces
pharmapsychotic Nov 23, 2022
630c7b2
Fix video test
pharmapsychotic Nov 24, 2022
d6d0896
Docstrings, tests, and clean up
pharmapsychotic Nov 25, 2022
04ddbdf
.
pharmapsychotic Nov 25, 2022
5cece34
Basic tests for Api methods
pharmapsychotic Nov 28, 2022
df120f1
Merge pull request #143 from Stability-AI/pharma.anims
dmarx Nov 28, 2022
5e2f801
SD 2.0 support and transform accumulation for 3D mode!
pharmapsychotic Dec 6, 2022
62a9ed4
Missed animation.py in last commit :)
pharmapsychotic Dec 6, 2022
924e837
Merge branch 'main' into pharma.anims
pharmapsychotic Dec 7, 2022
e127d8b
Chain requests for init prep and depth warp
pharmapsychotic Dec 13, 2022
e4241dd
New color and depth animated settings!
pharmapsychotic Dec 15, 2022
a97bc7c
Debug option to not use chain requests (helpful for development)
pharmapsychotic Dec 15, 2022
92d0166
Update to latest api-interfaces animation branch
pharmapsychotic Dec 15, 2022
2972b7c
Support for depth conditioning!
pharmapsychotic Dec 20, 2022
7548059
Video Input updates:
pharmapsychotic Dec 21, 2022
65ebacc
Change name of interpolate endpoint
pharmapsychotic Dec 21, 2022
83c5f33
.
pharmapsychotic Dec 22, 2022
b47f4ca
Have image_to_jpg_bytes and image_to_png_bytes support both PIL Image…
pharmapsychotic Jan 3, 2023
34739cf
Project API's!
pharmapsychotic Jan 3, 2023
b25f208
Update tests
pharmapsychotic Jan 3, 2023
5a82d2b
.
pharmapsychotic Jan 3, 2023
a74fa69
Animation notebook using Gradio for UI!
pharmapsychotic Jan 5, 2023
68a96e9
Merge branch 'anima.nb.20221221' of https://github.com/Stability-AI/s…
pharmapsychotic Jan 5, 2023
2101cc2
Clean up:
pharmapsychotic Jan 5, 2023
789b81e
PR feedback: use @property's with Project API, document reason for Ma…
pharmapsychotic Jan 6, 2023
0372658
Grab values from Gradio in render call instead of keeping in sync via…
pharmapsychotic Jan 9, 2023
76339b0
Fixes for Colab:
pharmapsychotic Jan 9, 2023
b7cf0f5
Merge pull request #166 from Stability-AI/pharma.anims
dmarx Jan 10, 2023
86972e7
Fix for project setting saving
pharmapsychotic Jan 12, 2023
aff4173
Merge pull request #167 from Stability-AI/pharma.project-fix
aletts Jan 12, 2023
ff8cb82
Remove TODO and use the new `reverse` depth option
pharmapsychotic Jan 12, 2023
1f2f905
Merge pull request #168 from Stability-AI/pharma.depth-reverse
pharmapsychotic Jan 18, 2023
bf2015f
Inpainting test notebook and API update
pharmapsychotic Jan 23, 2023
623de4b
Cleanup of inpaint API
pharmapsychotic Jan 24, 2023
bbca430
Clean up typing
pharmapsychotic Jan 24, 2023
bd92100
Update inpainting test notebook
pharmapsychotic Jan 25, 2023
26f2368
Inpaint notebook fixes and tweaks for running in Colab
pharmapsychotic Jan 25, 2023
32e99c9
Addition of `mask_max` setting
pharmapsychotic Jan 25, 2023
817da8d
Retry GRPC request support
pharmapsychotic Jan 26, 2023
8ef7dbc
print -> logger.warning
pharmapsychotic Jan 26, 2023
6c0b14a
New 3D animation algorithm that uses camera pose, occlusion map and i…
kostarion Jan 26, 2023
4947296
Additional test model option
pharmapsychotic Jan 26, 2023
5ce63a9
Update branch name in inpaint test notebook
pharmapsychotic Jan 26, 2023
b22437b
Better support for testing custom models
pharmapsychotic Jan 26, 2023
1df23f1
Inpainting API updates, test notebook, auto request retries
pharmapsychotic Jan 27, 2023
f08d43c
Retry support for classifier obfuscations (#173)
pharmapsychotic Jan 31, 2023
0b2e5c4
Merge remote-tracking branch 'origin/anima_stable' into cut_pow/anima…
kostarion Feb 7, 2023
9f452cc
Updates to animation: (#177)
pharmapsychotic Feb 8, 2023
39d5f70
animation_gradio.ipynb set share to True to work correctly in Colab
pharmapsychotic Feb 8, 2023
43841bc
Animation updates (#180)
pharmapsychotic Feb 13, 2023
fd103dc
Merge remote-tracking branch 'origin/anima_stable' into cut_pow/anima…
kostarion Feb 15, 2023
81865aa
Merge remote-tracking branch 'origin/anima_stable' into cut_pow/anima…
kostarion Feb 15, 2023
1ef34a9
Refactoring Parameters -> Settings
kostarion Feb 20, 2023
dba6b99
Keep track of only the last previous frame data when cadence is off.
kostarion Feb 20, 2023
35a691f
Chains and optimizations (#181)
pharmapsychotic Feb 22, 2023
57cde79
Optimize number of inpainting calls
kostarion Feb 22, 2023
f9c61ff
Merge branch 'animation' of github.com:Stability-AI/stability_sdk int…
kostarion Feb 24, 2023
a9145be
Update notebooks to include new 3D mode
kostarion Feb 24, 2023
d6691ea
Merge remote-tracking branch 'origin/anima_stable' into cut_pow/anima…
kostarion Feb 27, 2023
40e229b
Fix tests
kostarion Feb 27, 2023
cc7e938
Address feedback. 'Pseudo 3D' -> '3D warp', '3D' -> '3D render'
kostarion Feb 27, 2023
4f4bd63
Fix tests
kostarion Feb 27, 2023
364d96b
Merge pull request #184 from Stability-AI/cut_pow/anima_camera_pose_3d
kostarion Feb 27, 2023
37139e2
Misc animation updates: (#187)
pharmapsychotic Feb 27, 2023
42ec0a7
Flow inpaint mask and _debug_no_chains tensors (#189)
pharmapsychotic Mar 2, 2023
0a505f3
Remove alternate interpolate engine id.
pharmapsychotic Mar 3, 2023
e366092
Additional fixes from merges and server update
pharmapsychotic Mar 3, 2023
c7fe459
Update default output FPS from 24 to 12
pharmapsychotic Mar 3, 2023
976d9c9
Animation updates (#190)
pharmapsychotic Mar 7, 2023
0cd4223
Move Api out of client.py into its own file
pharmapsychotic Mar 8, 2023
8d658c8
Merge branch 'main' into pharma/organize
pharmapsychotic Mar 8, 2023
53e3096
Add display for user credits balance and picture
pharmapsychotic Mar 9, 2023
c4c34a0
Move bulk of code from animation_gradio.ipynb to animation_ui.py
pharmapsychotic Mar 9, 2023
1f09e22
Add new models to droplist, validate which models and samplers suppor…
pharmapsychotic Mar 9, 2023
93880bb
Update last video button under Output tab. Helpful to quickly rebuild…
pharmapsychotic Mar 9, 2023
d740fb5
Error handling for running out of credits
pharmapsychotic Mar 10, 2023
6db9daa
api.Api -> api.Context
pharmapsychotic Mar 10, 2023
004025b
Address a batch of sonarcloud warnings
pharmapsychotic Mar 10, 2023
cea8956
Fix test
pharmapsychotic Mar 10, 2023
9698448
Minor update to README grammar
pharmapsychotic Mar 10, 2023
d549c9e
Reorganize API plus anim notebook improvements #192
pharmapsychotic Mar 10, 2023
f9aaeba
Handle Gradio difference in Colab for credits display
pharmapsychotic Mar 10, 2023
593fab3
Post-process tab for frame interpolation (#196)
pharmapsychotic Mar 22, 2023
43f40d3
[DA-250] Batch of SDK updates for animation (#200)
pharmapsychotic Mar 28, 2023
6417b42
Merge branch 'main' into anima_stable
pharmapsychotic Mar 28, 2023
e461852
Updated merged code from main to work with anim branch
pharmapsychotic Mar 28, 2023
80888dd
Simlification of 3D render mode parameters
kostarion Mar 30, 2023
d4eadee
Remove grayscale mask postprocessing for 3D render mode. Update presets
kostarion Apr 4, 2023
a36861e
Merge pull request #201 from Stability-AI/cut_pow/3d_render_animation
kostarion Apr 5, 2023
5c729b0
Update submodule api-interfaces link
kostarion Apr 5, 2023
c5fe5c3
Merge pull request #204 from Stability-AI/cut_pow/3d_render_animation
kostarion Apr 5, 2023
264b5a5
Support for style presets (#205)
pharmapsychotic Apr 5, 2023
a1defc8
Misc animation updates: (#213)
pharmapsychotic Apr 19, 2023
41616d5
Release prep! (#217)
pharmapsychotic Apr 25, 2023
22bae34
Update anim notebooks to pip install with [anim] and [anim_ui]
pharmapsychotic Apr 25, 2023
9a83d6a
Merge branch 'main' into 'anima_stable'
pharmapsychotic Apr 25, 2023
4f7dbe9
Streamline animation_gradio.ipynb connect cell
pharmapsychotic Apr 25, 2023
37e9ad8
Inpaint with updated seed
kostarion Apr 24, 2023
c791710
Same render mode for both images and masks
kostarion Apr 24, 2023
813d7fd
Remove deprecated mask_fixup option
kostarion Apr 24, 2023
5690431
Merge branch 'main' into anima_stable
pharmapsychotic Apr 28, 2023
6e66e39
inpainting and mask postprocessing fixes
kostarion May 1, 2023
be1e826
Inpainting changes
kostarion May 1, 2023
52f96fb
Merge remote-tracking branch 'origin/anima_stable' into cut_pow/anima…
kostarion May 1, 2023
26c540b
Mask blur adjustments for cadence mode
kostarion May 2, 2023
4deb7ad
Cadence span inpainting change
kostarion May 3, 2023
1675a71
Preset updates
kostarion May 3, 2023
11dded1
Merge pull request #219 from Stability-AI/cut_pow/anima_inpaint_fixes
kostarion May 3, 2023
abaf540
SDK progress (#221)
pharmapsychotic May 3, 2023
5ea0828
Merge branch 'main' of https://github.com/Stability-AI/stability-sdk …
pharmapsychotic May 3, 2023
a51b4ac
Fix for gRPC error retry and out of credits detection.
pharmapsychotic May 8, 2023
089ca0d
resume in gradio
kostarion May 8, 2023
8eb5672
postprocess videofile
kostarion May 9, 2023
9d6fbc5
upscale frames in order
kostarion May 9, 2023
9fec58d
Merge branch 'anima_stable' of https://github.com/Stability-AI/stabil…
kostarion May 9, 2023
b613759
Misc animation changes (#228)
pharmapsychotic May 10, 2023
69ac724
Remove frame_idx from calls to _postprocess_inpainting_mask
pharmapsychotic May 10, 2023
2d710c8
fix mask binarization for 3d render
kostarion May 10, 2023
d4e83c3
create_video_from_frames to utils
kostarion May 10, 2023
02fda68
remove_frames_from_path typing
kostarion May 10, 2023
ea0d19c
Merge remote-tracking branch 'origin/anima_stable' into cut_pow/resum…
kostarion May 10, 2023
ea0b2bc
mask_binarization_thr default to 0.5, change doc
kostarion May 10, 2023
b2224d3
Merge pull request #225 from Stability-AI/cut_pow/resume_and_postproc…
kostarion May 10, 2023
1e193ba
Animated color match + sync 2D/3D camera directions (#229)
pharmapsychotic May 10, 2023
e353c64
avoid 'prefill' border when 'inpaint_border' is disabled
kostarion May 12, 2023
763bbce
not inpaint_border
kostarion May 12, 2023
3a06ade
Merge pull request #231 from Stability-AI/cut_pow/prefill_fix
kostarion May 12, 2023
8c50e65
Animation follow up (#232)
pharmapsychotic May 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/unit_testing.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,17 @@ jobs:
cache: 'pip'
cache-dependency-path: |
**/setup.py

- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Install OpenCV
run: |
pip install opencv-python-headless
- name: Install package
run: |
pip install .[dev]
pip install .[dev,anim]
- name: Test with pytest
run: |
pytest -s --ignore=src/stability_sdk/interfaces
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
*.pyc
*.egg-info
*.png
#*.png
.vscode/
build/
dist/
pyenv/
*venv/
.env
generation-*.pb.json
*settings.txt

Pipfile*
10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ It will generate and put PNGs in your current directory.
To upscale:
`python3 -m stability_sdk upscale -i "/path/to/image.png"`

## Animation UI

Install with
`pip install stability-sdk[anim_ui]`

Then run with
`python3 -m stability_sdk animate --gui`

## SDK Usage

Be sure to check out [Platform](https://platform.stability.ai) for comprehensive documentation on how to interact with our API.
Expand All @@ -57,7 +65,7 @@ options:
--width WIDTH, -W WIDTH
[512] width of image
--start_schedule START_SCHEDULE
[0.5] start schedule for init image (must be greater than 0, 1 is full strength
[0.5] start schedule for init image (must be greater than 0; 1 is full strength
text prompt, no trace of image)
--end_schedule END_SCHEDULE
[0.01] end schedule for init image
Expand Down
320 changes: 320 additions & 0 deletions nbs/animation.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,320 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rWXrHouW3pq_"
},
"source": [
"# Animation SDK example"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"cellView": "form",
"id": "eeA1mYLdxr2j"
},
"outputs": [],
"source": [
"#@title Install the Stability SDK\n",
"%%capture captured --no-stderr\n",
"import os\n",
"import shutil\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"path = Path('stability-sdk')\n",
"if os.path.exists(\"../src/stability_sdk\"):\n",
" sys.path.append(\"../src\") # use local SDK src\n",
"else:\n",
" if path.exists():\n",
" shutil.rmtree(path)\n",
" !pip uninstall -y stability-sdk\n",
" !git clone -b anima_stable --recurse-submodules https://github.com/Stability-AI/stability-sdk\n",
" !pip install ./stability-sdk[anim]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "jCZ-IphH3prD"
},
"outputs": [],
"source": [
"#@title Mount Google Drive\n",
"try:\n",
" from google.colab import drive\n",
" drive.mount('/content/gdrive')\n",
" outputs_path = \"/content/gdrive/MyDrive/AI/StableAnimation\"\n",
" os.makedirs(outputs_path, exist_ok=True)\n",
"except:\n",
" outputs_path = \".\"\n",
"print(f\"Animations will be saved to {outputs_path}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "zj56t6tc3prF"
},
"outputs": [],
"source": [
"#@title Connect to the Stability API\n",
"import datetime\n",
"import getpass\n",
"import json\n",
"import os\n",
"import panel as pn\n",
"import param\n",
"\n",
"from base64 import b64encode\n",
"from IPython import display\n",
"from tqdm import tqdm\n",
"from types import SimpleNamespace\n",
"\n",
"from stability_sdk.api import Context\n",
"from stability_sdk.animation import AnimationArgs, Animator\n",
"from stability_sdk.utils import create_video_from_frames\n",
"\n",
"# @markdown To get your API key visit https://next.dreamstudio.pages.dev/account\n",
"STABILITY_HOST = \"grpc-staging.stability.ai:443\" #@param {type:\"string\"}\n",
"STABILITY_KEY = getpass.getpass('Enter your API Key')\n",
"\n",
"# Connect to Stability API\n",
"context = Context(STABILITY_HOST, STABILITY_KEY)\n",
"\n",
"# Test the connection\n",
"context.get_user_info()\n",
"print(\"Connection successful!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "ldUAFmur3prH"
},
"outputs": [],
"source": [
"# @title Settings\n",
"\n",
"# @markdown Run this cell to reveal the settings UI grouped across several tabs. After entering values, move on to the next step.\n",
"\n",
"# @markdown To reset values to default, simply re-run this cell.\n",
"\n",
"show_documentation = True # @param {type:'boolean'}\n",
"\n",
"# #@markdown ####**Resume:**\n",
"resume_timestring = \"\" #@param {type:\"string\"}\n",
"\n",
"#@markdown ####**Override Settings:**\n",
"override_settings_path = \"\" #@param {type:\"string\"}\n",
"\n",
"###################\n",
"\n",
"from stability_sdk.animation import (\n",
" AnimationArgs,\n",
" Animator,\n",
" AnimationSettings,\n",
" BasicSettings,\n",
" CoherenceSettings,\n",
" ColorSettings,\n",
" DepthSettings,\n",
" InpaintingSettings,\n",
" Rendering3dSettings,\n",
" CameraSettings,\n",
" VideoInputSettings,\n",
" VideoOutputSettings,\n",
")\n",
"\n",
"args_generation = BasicSettings()\n",
"args_animation = AnimationSettings()\n",
"args_camera = CameraSettings()\n",
"args_coherence = CoherenceSettings()\n",
"args_color = ColorSettings()\n",
"args_depth = DepthSettings()\n",
"args_render_3d = Rendering3dSettings()\n",
"args_inpaint = InpaintingSettings()\n",
"args_vid_in = VideoInputSettings()\n",
"args_vid_out = VideoOutputSettings()\n",
"arg_objs = (\n",
" args_generation,\n",
" args_animation,\n",
" args_camera,\n",
" args_coherence,\n",
" args_color,\n",
" args_depth,\n",
" args_render_3d,\n",
" args_inpaint,\n",
" args_vid_in,\n",
" args_vid_out,\n",
")\n",
"\n",
"def _show_docs(component):\n",
" cols = []\n",
" for k, v in component.param.objects().items():\n",
" if k == 'name':\n",
" continue\n",
" col = pn.Column(v, v.doc)\n",
" cols.append(col)\n",
" return pn.Column(*cols)\n",
"\n",
"def build(component):\n",
" if show_documentation:\n",
" component = _show_docs(component)\n",
" return pn.Row(component, width=1000)\n",
"\n",
"pn.extension()\n",
"\n",
"pn.Tabs(*[\n",
" (a.name[:-5], build(a)) for a in arg_objs\n",
"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "_SudvbZG3prI"
},
"source": [
"### Prompts"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "FT9slDSw3prJ"
},
"outputs": [],
"source": [
"animation_prompts = {\n",
" 0: \"a painting of a delicious cheeseburger\",\n",
" 24: \"a painting of the the answer to life the universe and everything\",\n",
"}\n",
"\n",
"negative_prompt = \"\"\n",
"negative_prompt_weight = -1.0\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Rpqv6t303prJ"
},
"outputs": [],
"source": [
"#@title Render the animation\n",
"\n",
"args_d = {}\n",
"[args_d.update(a.param.values()) for a in arg_objs]\n",
"args=AnimationArgs(**args_d)\n",
"\n",
"\n",
"# load override settings if provided\n",
"if override_settings_path:\n",
" if not os.path.exists(override_settings_path):\n",
" raise ValueError(f\"Override settings file not found: {override_settings_path}\")\n",
" with open(override_settings_path, 'r') as f:\n",
" overrides = json.load(f)\n",
" args = vars(args)\n",
" for k in args.keys():\n",
" if k in overrides:\n",
" args[k] = overrides[k]\n",
" args = SimpleNamespace(**args)\n",
" animation_prompts = overrides.get('animation_prompts', animation_prompts)\n",
" animation_prompts = {int(k): v for k, v in animation_prompts.items()}\n",
" negative_prompt = overrides.get('negative_prompt', negative_prompt)\n",
" negative_prompt_weight = overrides.get('negative_prompt_weight', negative_prompt_weight)\n",
"\n",
"# create folder for frames output\n",
"if resume_timestring:\n",
" out_dir = os.path.join(outputs_path, resume_timestring)\n",
" if not os.path.exists(out_dir):\n",
" raise Exception(\"Can't resume {resume_timestring} because path {out_dir} doesn't exist. Please make sure the timestring is correct.\")\n",
" timestring = resume_timestring\n",
"else:\n",
" timestring = datetime.datetime.now().strftime('%Y%m%d%H%M%S')\n",
" out_dir = os.path.join(outputs_path, timestring)\n",
" os.makedirs(out_dir, exist_ok=True)\n",
"print(f\"Saving animation frames to {out_dir}...\")\n",
"\n",
"animator = Animator(\n",
" api_context=context,\n",
" animation_prompts=animation_prompts,\n",
" args=args,\n",
" out_dir=out_dir, \n",
" negative_prompt=negative_prompt,\n",
" negative_prompt_weight=negative_prompt_weight,\n",
" resume=len(resume_timestring) != 0,\n",
")\n",
"animator.save_settings(f\"{timestring}_settings.txt\")\n",
"\n",
"for frame in tqdm(animator.render(), initial=animator.start_frame_idx, total=args.max_frames):\n",
" display.clear_output(wait=True)\n",
" display.display(frame)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "aWhJnLNX3prL"
},
"outputs": [],
"source": [
"#@title Create video from frames\n",
"skip_video_for_run_all = False #@param {type: 'boolean'}\n",
"fps = 12 #@param {type:\"number\"}\n",
"\n",
"if skip_video_for_run_all == True:\n",
" print('Skipping video creation, uncheck skip_video_for_run_all if you want to run it')\n",
"else:\n",
" mp4_path = os.path.join(out_dir, f\"{timestring}.mp4\")\n",
" print(f\"Compiling animation frames to {mp4_path}...\")\n",
" create_video_from_frames(out_dir, mp4_path, fps)\n",
"\n",
" mp4 = open(mp4_path,'rb').read()\n",
" data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
" display.display( display.HTML(f'<video controls loop><source src=\"{data_url}\" type=\"video/mp4\"></video>') )"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.5"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 0
}
Loading