Skip to content

Commit

Permalink
tmp
Browse files Browse the repository at this point in the history
  • Loading branch information
@hinthornw
hinthornw committed Oct 28, 2024
1 parent c6ffe24 commit b647ec2
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 7 deletions.
2 changes: 2 additions & 0 deletions .github/scripts/run_langgraph_cli_test.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import json

Check notice on line 1 in .github/scripts/run_langgraph_cli_test.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results 

.........................................
fanout_to_subgraph_10x: Mean +- std dev: 48.1 ms +- 0.6 ms
.........................................
fanout_to_subgraph_10x_sync: Mean +- std dev: 46.0 ms +- 3.1 ms
.........................................
fanout_to_subgraph_10x_checkpoint: Mean +- std dev: 76.5 ms +- 1.6 ms
.........................................
fanout_to_subgraph_10x_checkpoint_sync: Mean +- std dev: 85.0 ms +- 1.0 ms
.........................................
fanout_to_subgraph_100x: Mean +- std dev: 467 ms +- 10 ms
.........................................
fanout_to_subgraph_100x_sync: Mean +- std dev: 424 ms +- 5 ms
.........................................
fanout_to_subgraph_100x_checkpoint: Mean +- std dev: 790 ms +- 48 ms
.........................................
fanout_to_subgraph_100x_checkpoint_sync: Mean +- std dev: 833 ms +- 17 ms
.........................................
react_agent_10x: Mean +- std dev: 29.0 ms +- 0.6 ms
.........................................
react_agent_10x_sync: Mean +- std dev: 22.5 ms +- 1.7 ms
.........................................
react_agent_10x_checkpoint: Mean +- std dev: 48.2 ms +- 3.8 ms
.........................................
react_agent_10x_checkpoint_sync: Mean +- std dev: 37.6 ms +- 3.6 ms
.........................................
react_agent_100x: Mean +- std dev: 328 ms +- 13 ms
.........................................
react_agent_100x_sync: Mean +- std dev: 262 ms +- 12 ms
.........................................
react_agent_100x_checkpoint: Mean +- std dev: 903 ms +- 8 ms
.........................................
react_agent_100x_checkpoint_sync: Mean +- std dev: 815 ms +- 10 ms
.........................................
wide_state_25x300: Mean +- std dev: 18.1 ms +- 0.3 ms
.........................................
wide_state_25x300_sync: Mean +- std dev: 10.8 ms +- 0.1 ms
.........................................
wide_state_25x300_checkpoint: Mean +- std dev: 271 ms +- 3 ms
.........................................
wide_state_25x300_checkpoint_sync: Mean +- std dev: 262 ms +- 4 ms
.........................................
wide_state_15x600: Mean +- std dev: 21.0 ms +- 0.4 ms
.........................................
wide_state_15x600_sync: Mean +- std dev: 12.5 ms +- 0.1 ms
.........................................
wide_state_15x600_checkpoint: Mean +- std dev: 470 ms +- 9 ms
.........................................
wide_state_15x600_checkpoint_sync: Mean +- std dev: 467 ms +- 14 ms
.........................................
wide_state_9x1200: Mean +- std dev: 21.1 ms +- 0.4 ms
.........................................
wide_state_9x1200_sync: Mean +- std dev: 12.5 ms +- 0.1 ms
.........................................
wide_state_9x1200_checkpoint: Mean +- std dev: 304 ms +- 3 ms
.........................................
wide_state_9x1200_checkpoint_sync: Mean +- std dev: 300 ms +- 13 ms

Check notice on line 1 in .github/scripts/run_langgraph_cli_test.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main 

+----------------------------------------+---------+-----------------------+
| Benchmark                              | main    | changes               |
+========================================+=========+=======================+
| fanout_to_subgraph_10x                 | 54.6 ms | 48.1 ms: 1.14x faster |
+----------------------------------------+---------+-----------------------+
| wide_state_15x600_checkpoint           | 474 ms  | 470 ms: 1.01x faster  |
+----------------------------------------+---------+-----------------------+
| wide_state_9x1200_sync                 | 12.6 ms | 12.5 ms: 1.01x faster |
+----------------------------------------+---------+-----------------------+
| wide_state_9x1200_checkpoint           | 307 ms  | 304 ms: 1.01x faster  |
+----------------------------------------+---------+-----------------------+
| react_agent_100x_checkpoint            | 909 ms  | 903 ms: 1.01x faster  |
+----------------------------------------+---------+-----------------------+
| fanout_to_subgraph_100x_sync           | 427 ms  | 424 ms: 1.01x faster  |
+----------------------------------------+---------+-----------------------+
| wide_state_25x300_checkpoint           | 272 ms  | 271 ms: 1.00x faster  |
+----------------------------------------+---------+-----------------------+
| wide_state_15x600                      | 21.1 ms | 21.0 ms: 1.00x faster |
+----------------------------------------+---------+-----------------------+
| wide_state_15x600_sync                 | 12.5 ms | 12.5 ms: 1.00x faster |
+----------------------------------------+---------+-----------------------+
| fanout_to_subgraph_10x_checkpoint_sync | 84.7 ms | 85.0 ms: 1.00x slower |
+----------------------------------------+---------+-----------------------+
| Geometric mean                         | (ref)   | 1.01x faster          |
+----------------------------------------+---------+-----------------------+

Benchmark hidden because not significant (18): wide_state_9x1200_checkpoint_sync, fanout_to_subgraph_100x_checkpoint, wide_state_15x600_checkpoint_sync, wide_state_25x300_sync, wide_state_9x1200, react_agent_100x, wide_state_25x300_checkpoint_sync, wide_state_25x300, react_agent_100x_checkpoint_sync, fanout_to_subgraph_100x, fanout_to_subgraph_100x_checkpoint_sync, fanout_to_subgraph_10x_sync, react_agent_10x, fanout_to_subgraph_10x_checkpoint, react_agent_100x_sync, react_agent_10x_checkpoint_sync, react_agent_10x_sync, react_agent_10x_checkpoint
import os
import pathlib
import sys
import langgraph_cli
Expand All @@ -7,6 +8,7 @@

from langgraph_cli.exec import Runner, subp_exec
from langgraph_cli.progress import Progress
from langgraph_cli.docker import DEFAULT_POSTGRES_URI
from langgraph_cli.constants import DEFAULT_PORT


Expand Down
15 changes: 8 additions & 7 deletions .github/workflows/_integration_test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,23 +44,24 @@ jobs:
working-directory: libs/cli/examples
run: |
# The build-arg isn't used; just testing that we accept other args
langgraph build -t langgraph-test-a
langgraph build -t langgraph-test-a --base-image "langchain/langgraph-trial"
cp .env.example .envg
python ../../../.github/scripts/run_langgraph_cli_test.py -c langgraph.json -t langgraph-test-a
- name: Build and test service B
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs
run: |
langgraph build -t langgraph-test-b
python ../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-b
langgraph build -t langgraph-test-b --base-image "langchain/langgraph-trial"
python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-b
- name: Build and test service C
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs_reqs_a
run: |
langgraph build -t langgraph-test-c
python ../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-c
langgraph build -t langgraph-test-c --base-image "langchain/langgraph-trial"
python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-c
- name: Build and test service D
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs_reqs_b
run: |
langgraph build -t langgraph-test-d
python ../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-d
langgraph build -t langgraph-test-d --base-image "langchain/langgraph-trial"
python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-d

0 comments on commit b647ec2

Please sign in to comment.