Skip to content

Commit

Permalink
tmp
Browse files Browse the repository at this point in the history
  • Loading branch information
hinthornw committed Oct 28, 2024
1 parent c6ffe24 commit 7c473f6
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 17 deletions.
32 changes: 23 additions & 9 deletions .github/scripts/run_langgraph_cli_test.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import asyncio

Check notice on line 1 in .github/scripts/run_langgraph_cli_test.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results

......................................... fanout_to_subgraph_10x: Mean +- std dev: 48.2 ms +- 0.6 ms ......................................... fanout_to_subgraph_10x_sync: Mean +- std dev: 46.2 ms +- 3.2 ms ......................................... fanout_to_subgraph_10x_checkpoint: Mean +- std dev: 76.2 ms +- 1.6 ms ......................................... fanout_to_subgraph_10x_checkpoint_sync: Mean +- std dev: 84.5 ms +- 0.8 ms ......................................... fanout_to_subgraph_100x: Mean +- std dev: 472 ms +- 10 ms ......................................... fanout_to_subgraph_100x_sync: Mean +- std dev: 426 ms +- 5 ms ......................................... fanout_to_subgraph_100x_checkpoint: Mean +- std dev: 787 ms +- 29 ms ......................................... fanout_to_subgraph_100x_checkpoint_sync: Mean +- std dev: 830 ms +- 16 ms ......................................... react_agent_10x: Mean +- std dev: 28.9 ms +- 0.6 ms ......................................... react_agent_10x_sync: Mean +- std dev: 22.3 ms +- 1.6 ms ......................................... react_agent_10x_checkpoint: Mean +- std dev: 47.7 ms +- 3.3 ms ......................................... react_agent_10x_checkpoint_sync: Mean +- std dev: 37.4 ms +- 3.2 ms ......................................... react_agent_100x: Mean +- std dev: 329 ms +- 12 ms ......................................... react_agent_100x_sync: Mean +- std dev: 261 ms +- 13 ms ......................................... react_agent_100x_checkpoint: Mean +- std dev: 911 ms +- 7 ms ......................................... react_agent_100x_checkpoint_sync: Mean +- std dev: 815 ms +- 8 ms ......................................... wide_state_25x300: Mean +- std dev: 18.2 ms +- 0.3 ms ......................................... wide_state_25x300_sync: Mean +- std dev: 10.8 ms +- 0.1 ms ......................................... wide_state_25x300_checkpoint: Mean +- std dev: 273 ms +- 6 ms ......................................... wide_state_25x300_checkpoint_sync: Mean +- std dev: 262 ms +- 4 ms ......................................... wide_state_15x600: Mean +- std dev: 21.6 ms +- 0.6 ms ......................................... wide_state_15x600_sync: Mean +- std dev: 12.9 ms +- 0.3 ms ......................................... wide_state_15x600_checkpoint: Mean +- std dev: 476 ms +- 6 ms ......................................... wide_state_15x600_checkpoint_sync: Mean +- std dev: 475 ms +- 19 ms ......................................... wide_state_9x1200: Mean +- std dev: 21.2 ms +- 0.4 ms ......................................... wide_state_9x1200_sync: Mean +- std dev: 12.6 ms +- 0.2 ms ......................................... wide_state_9x1200_checkpoint: Mean +- std dev: 307 ms +- 4 ms ......................................... wide_state_9x1200_checkpoint_sync: Mean +- std dev: 304 ms +- 14 ms

Check notice on line 1 in .github/scripts/run_langgraph_cli_test.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main

+----------------------------------------+---------+-----------------------+ | Benchmark | main | changes | +========================================+=========+=======================+ | fanout_to_subgraph_10x | 54.6 ms | 48.2 ms: 1.13x faster | +----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x_checkpoint_sync | 84.7 ms | 84.5 ms: 1.00x faster | +----------------------------------------+---------+-----------------------+ | wide_state_25x300 | 18.1 ms | 18.2 ms: 1.01x slower | +----------------------------------------+---------+-----------------------+ | wide_state_9x1200 | 21.1 ms | 21.2 ms: 1.01x slower | +----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_100x | 466 ms | 472 ms: 1.01x slower | +----------------------------------------+---------+-----------------------+ | wide_state_15x600_checkpoint_sync | 469 ms | 475 ms: 1.01x slower | +----------------------------------------+---------+-----------------------+ | wide_state_15x600 | 21.1 ms | 21.6 ms: 1.02x slower | +----------------------------------------+---------+-----------------------+ | wide_state_15x600_sync | 12.5 ms | 12.9 ms: 1.03x slower | +----------------------------------------+---------+-----------------------+ | Geometric mean | (ref) | 1.00x faster | +----------------------------------------+---------+-----------------------+ Benchmark hidden because not significant (20): fanout_to_subgraph_100x_checkpoint, wide_state_9x1200_sync, fanout_to_subgraph_100x_sync, wide_state_25x300_sync, react_agent_10x, fanout_to_subgraph_100x_checkpoint_sync, react_agent_10x_checkpoint_sync, fanout_to_subgraph_10x_checkpoint, wide_state_9x1200_checkpoint, wide_state_25x300_checkpoint_sync, react_agent_100x_checkpoint_sync, wide_state_25x300_checkpoint, react_agent_100x, react_agent_10x_sync, react_agent_100x_checkpoint, wide_state_15x600_checkpoint, react_agent_100x_sync, fanout_to_subgraph_10x_sync, react_agent_10x_checkpoint, wide_state_9x1200_checkpoint_sync
import json
import os
import pathlib
import sys
import langgraph_cli
Expand All @@ -7,6 +9,7 @@

from langgraph_cli.exec import Runner, subp_exec
from langgraph_cli.progress import Progress
from langgraph_cli.docker import DEFAULT_POSTGRES_URI
from langgraph_cli.constants import DEFAULT_PORT


Expand Down Expand Up @@ -70,26 +73,37 @@ def test(
]
)

completed_startup = False

def on_stdout(line: str):
if "GET /ok" in line:
nonlocal completed_startup
if "GET /ok" in line or "Uvicorn running on" in line:
set("")
sys.stdout.write(
f"""Ready!
- API: http://localhost:{port}
"""
)
sys.stdout.flush()
completed_startup = True
runner.close()
return True
return False

runner.run(
subp_exec(
"docker",
*args,
tag,
verbose=verbose,
on_stdout=on_stdout,
try:
runner.run(
subp_exec(
"docker",
*args,
tag,
verbose=verbose,
on_stdout=on_stdout,
)
)
)
except asyncio.CancelledError:
if not completed_startup:
raise ValueError("Failed to complete startup.")
pass


if __name__ == "__main__":
Expand Down
17 changes: 9 additions & 8 deletions .github/workflows/_integration_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,23 +44,24 @@ jobs:
working-directory: libs/cli/examples
run: |
# The build-arg isn't used; just testing that we accept other args
langgraph build -t langgraph-test-a
python ../../../.github/scripts/run_langgraph_cli_test.py -c langgraph.json -t langgraph-test-a
langgraph build -t langgraph-test-a --base-image "langchain/langgraph-trial"
cp .env.example .envg
timeout 60 python ../../../.github/scripts/run_langgraph_cli_test.py -c langgraph.json -t langgraph-test-a
- name: Build and test service B
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs
run: |
langgraph build -t langgraph-test-b
python ../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-b
langgraph build -t langgraph-test-b --base-image "langchain/langgraph-trial"
timeout 60 python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-b
- name: Build and test service C
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs_reqs_a
run: |
langgraph build -t langgraph-test-c
python ../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-c
langgraph build -t langgraph-test-c --base-image "langchain/langgraph-trial"
timeout 60 python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-c
- name: Build and test service D
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs_reqs_b
run: |
langgraph build -t langgraph-test-d
python ../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-d
langgraph build -t langgraph-test-d --base-image "langchain/langgraph-trial"
timeout 60 python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-d

0 comments on commit 7c473f6

Please sign in to comment.