Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add DatabaseEmitter implementation class #33

Merged
merged 113 commits into from
Dec 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
4d1deb9
feat: added mongo emitter
AlexPatrie Nov 25, 2023
d7ba0cf
feat: added fallback serializer function
AlexPatrie Nov 25, 2023
207aadc
updated type annotations
AlexPatrie Nov 25, 2023
ec2722f
feat: updated emitter file with copied logic from vivarium
AlexPatrie Nov 25, 2023
cd21f9a
feat: updated db emitter implementation with update signature
AlexPatrie Nov 25, 2023
a2f567c
feat: updated process registration for db emitter on init
AlexPatrie Nov 25, 2023
f2a6932
feat: implemented query method
AlexPatrie Nov 28, 2023
0583b53
added config schema to database emitter
AlexPatrie Nov 28, 2023
1759dfe
feat: updated instance docstring for db emitter
AlexPatrie Dec 1, 2023
15d19d4
added docstrings to emitter
AlexPatrie Dec 1, 2023
8f89d76
updated docstrings
AlexPatrie Dec 1, 2023
9d6efe6
chore: removed instance
AlexPatrie Dec 1, 2023
ae4aadd
feat: added to experiments library
AlexPatrie Dec 1, 2023
743aafb
chore: updated docstring in db emitter
AlexPatrie Dec 1, 2023
9a68305
added abc for database emitter
AlexPatrie Dec 1, 2023
89bbc94
added sqlite implementation
AlexPatrie Dec 1, 2023
d5c8539
added format emit data method
AlexPatrie Dec 1, 2023
69c7458
adjusted config_schema to include emission types
AlexPatrie Dec 1, 2023
7edec48
updated mongo emitter
AlexPatrie Dec 1, 2023
8a35e6a
updated docs
AlexPatrie Dec 1, 2023
e1d60cc
updated formatter
AlexPatrie Dec 1, 2023
501a808
docs: updated database emitter emit docstring
AlexPatrie Dec 1, 2023
a251fe1
updated emitter emit method
AlexPatrie Dec 1, 2023
fe9520b
docs: updated docstrings and comments
AlexPatrie Dec 1, 2023
63be099
minor update
AlexPatrie Dec 1, 2023
f4823ca
added test file and updated emitter
AlexPatrie Dec 1, 2023
d9a8918
added example model
AlexPatrie Dec 1, 2023
647117f
updated db
AlexPatrie Dec 2, 2023
918120d
added update docs
AlexPatrie Dec 5, 2023
f831b33
updated emitter
AlexPatrie Dec 5, 2023
29fed4d
updated all
AlexPatrie Dec 5, 2023
89a9dfc
updated update method in db emitter
AlexPatrie Dec 5, 2023
5895e83
removed unused code
AlexPatrie Dec 5, 2023
081e1c6
added to emitter test
AlexPatrie Dec 6, 2023
cc012e7
added to experiment
AlexPatrie Dec 6, 2023
247c63f
updated emitter structure
AlexPatrie Dec 6, 2023
5e9d67d
fix: updated input and output port handling in determine steps and ou…
AlexPatrie Dec 7, 2023
3dc9020
feat: added tellurium process to experiments dir
AlexPatrie Dec 7, 2023
3d20fdf
minor adjustments
AlexPatrie Dec 7, 2023
eac402d
implemented default type to embed path for database emitter
AlexPatrie Dec 7, 2023
1f04b44
updated emitter function docstrings
AlexPatrie Dec 7, 2023
398b769
chore: removed unused print statements in composite
AlexPatrie Dec 7, 2023
e949a5a
fix: updated emitter instance declaration
AlexPatrie Dec 7, 2023
8ff9ada
chore: added new database emitter test func for tellurium process
AlexPatrie Dec 8, 2023
e9ebf03
feat: updated emitter:
AlexPatrie Dec 8, 2023
36ed4b1
added signal example to examples
AlexPatrie Dec 8, 2023
ed75d8a
updated API contract
AlexPatrie Dec 8, 2023
519b99d
updated process with implementation copied
AlexPatrie Dec 8, 2023
a69addf
updated implementation
AlexPatrie Dec 8, 2023
d51bb1c
updated func
AlexPatrie Dec 8, 2023
f9078c3
updated output
AlexPatrie Dec 8, 2023
5a33062
refactored to include emitter library
AlexPatrie Dec 8, 2023
44632b5
refactored to include type system library
AlexPatrie Dec 8, 2023
e2d4959
removed type_system library
AlexPatrie Dec 8, 2023
e8f0fe9
removed emitter library
AlexPatrie Dec 8, 2023
861fcbc
adjusted for refactor
AlexPatrie Dec 8, 2023
fe087e4
added processes to init
AlexPatrie Dec 8, 2023
7be3db2
updated process test
AlexPatrie Dec 8, 2023
e001b07
updated init to include all main
AlexPatrie Dec 8, 2023
c6ddb6b
chore: refactored experiments library to include model files dir
AlexPatrie Dec 8, 2023
44db3ab
updated init with pf and pp and added copasi process
AlexPatrie Dec 8, 2023
33df610
chore: added relevant copasi model sbml and updated filepath in demo …
AlexPatrie Dec 8, 2023
15b0edc
added visualize processes notebook and smoldyn process example
AlexPatrie Dec 8, 2023
584ea21
updated minE example
AlexPatrie Dec 8, 2023
901f4f3
updated test file
AlexPatrie Dec 8, 2023
108590f
updated visualize notebook and minimal_gillespie
AlexPatrie Dec 8, 2023
5e2e027
minor updates
AlexPatrie Dec 8, 2023
df25287
fix: updated model filepath
AlexPatrie Dec 8, 2023
074a314
updated signal modulatation file
AlexPatrie Dec 8, 2023
2c5fe79
added ring mod
AlexPatrie Dec 8, 2023
3434afa
updated signal_modulation
AlexPatrie Dec 8, 2023
1fff8c1
fix updated process example
AlexPatrie Dec 10, 2023
9031fe3
minor refactoring
AlexPatrie Dec 11, 2023
ad23a43
updated signal modulation instances
AlexPatrie Dec 11, 2023
5f2f804
refactored plot methods
AlexPatrie Dec 11, 2023
6b4d444
minor code formatting
AlexPatrie Dec 11, 2023
9558f5c
feat: implemented plotting function on each update
AlexPatrie Dec 11, 2023
f801142
implemented plot function
AlexPatrie Dec 11, 2023
0b69624
updated instance func
AlexPatrie Dec 11, 2023
e811ba6
minor modulation update
AlexPatrie Dec 11, 2023
5f83c55
added phaser process
AlexPatrie Dec 11, 2023
f3464d2
feat: added delay example
AlexPatrie Dec 11, 2023
21786c0
added test delay to runtime
AlexPatrie Dec 11, 2023
a3119e6
updated delay example
AlexPatrie Dec 11, 2023
96333fe
fix: updated apply mod
AlexPatrie Dec 11, 2023
4467cf6
chore: removed unused and added to gitignore
AlexPatrie Dec 11, 2023
b237f73
updated tests
AlexPatrie Dec 11, 2023
0c4fc6e
added test pedalboard function
AlexPatrie Dec 12, 2023
5bd7c75
implemented pedalboard process
AlexPatrie Dec 12, 2023
81c07a8
updated pedalboard test
AlexPatrie Dec 12, 2023
25f9fa5
removed ported processes model files and refactored testing dir
AlexPatrie Dec 14, 2023
5b0d708
updated process registry lines in init for implementations
AlexPatrie Dec 14, 2023
588c861
updated signal modulation for input signal
AlexPatrie Dec 14, 2023
2e89cbe
updated readme to include mongo instructions from vivarium
AlexPatrie Dec 14, 2023
0b3d68b
updated signal modulation to include input in schema
AlexPatrie Dec 14, 2023
840e703
adjusted pedalboard composite in signal modulation experiment
AlexPatrie Dec 14, 2023
7f36f8f
added print and run statement to pedalboard composite instance
AlexPatrie Dec 14, 2023
76b3fd7
chore: new version for content
AlexPatrie Dec 14, 2023
f784728
adjusted signal mod
AlexPatrie Dec 14, 2023
3121b0e
updated process test
AlexPatrie Dec 15, 2023
4111596
feat: added processes dir for core_processes and added pprint to setup
AlexPatrie Dec 15, 2023
f38a08b
updated setup
AlexPatrie Dec 15, 2023
6d06491
fix: removed pprint
AlexPatrie Dec 15, 2023
32307dc
added empty to gitignore
AlexPatrie Dec 16, 2023
87ba199
updated processes with correct imports
AlexPatrie Dec 16, 2023
e20f287
removed core process library and adjusted init imports
AlexPatrie Dec 16, 2023
f3a9e9f
chore: removed unused extras require
AlexPatrie Dec 16, 2023
bc489a0
chore: removed unused imports in init from core processes
AlexPatrie Dec 16, 2023
6c58e7e
chore: removed scipy and matplotlib lines
AlexPatrie Dec 16, 2023
4f1c12e
chore: commented out matplotlib and scipy
AlexPatrie Dec 16, 2023
e2666f6
used ram-emitter
AlexPatrie Dec 16, 2023
e299631
fix: updated signal modulation instances for pytest
AlexPatrie Dec 16, 2023
fdb9bba
Merge branch 'main' into emitter-adding
AlexPatrie Dec 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
Binary file not shown.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ __pycache__/
.idea/
*.ipynb_checkpoints/
dist/
.DS_Store
.empty
build/

process_bigraph.egg-info/
71 changes: 71 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,77 @@ You can install `process-bigraph` using pip:
pip install process-bigraph
```

MongoDB
-------

We use a MongoDB database to store the data collected from running
simulations via the `DatabaseEmitter` class. The use of this class is specified in the
`'emitter'` section of the instance declaration for a given process as such:

...,
'emitter': {
'_type': 'step',
'address': 'local:database-emitter',
'config': {
'ports': {
'inputs': {
'table': 'string',

. This can be a remote server, but for this guide we will
run a MongoDB server locally.

**Note**: MongoDB is only required if you want to store data in MongoDB
or want to run experiments that do so via the declaration shown above. You don't need MongoDB to work
through this guide.

*Check Installation*

$ mongod --version
db version v4.2.3
...

Make sure you see a version at least 3.2.

*Install*

If you are on macOS, you can install MongoDB using [Homebrew](https://brew.sh). You will need to add the MongoDB tap following the
instructions [here](https://github.com/mongodb/homebrew-brew).

If you are on Linux, see the MongoDB documentation's [instructions](https://docs.mongodb.com/manual/administration/install-on-linux/).

*Setup*

You can get a MongoDB server up and running locally any number of ways.
Here is one:

1. Create a folder ``process_bigraph_work/mongodb``. This is where MongoDB will
store the database We store the database here instead of at the
default location in ``/usr/local/var/mongodb`` to avoid permissions
issues if you are not running as an administrator.
2. Make a copy of the ``mongod`` configuration file so we can make
changes:

$ cp /usr/local/etc/mongod.conf process_bigraph_work/mongod.conf

Note that your configuration file may be somewhere slightly
different. Check the MongoDB documentation for your system.
3. In ``process_bigraph_work/mongod.conf`` change the path after ``dbPath:`` to
point to ``process_bigraph_work/mongodb``.
4. Create a shell script ``process_bigraph_work/mongo.sh`` with the following
content:

#!/bin/bash

mongod --config mongod.conf

5. Make the script executable:

$ chmod 700 process_bigraph_work/mongo.sh

6. Now you can launch MongoDB by running this script:

$ process_bigraph_work/mongo.sh

## Tutorial

To get started with Bigraph-viz, explore our resources:
Expand Down
237 changes: 237 additions & 0 deletions notebooks/visualize_processes.ipynb

Large diffs are not rendered by default.

37 changes: 31 additions & 6 deletions process_bigraph/__init__.py
Original file line number Diff line number Diff line change
@@ -1,15 +1,40 @@
from process_bigraph.composite import Process, Step, Composite
from process_bigraph.type_system import types
import pprint
from process_bigraph.protocols import local_lookup
from process_bigraph.registry import protocol_registry, process_registry
from process_bigraph.emitter import ConsoleEmitter, RAMEmitter
from process_bigraph.emitter import ConsoleEmitter, RAMEmitter, DatabaseEmitter
from process_bigraph.composite import Process, Step, Composite
from process_bigraph.type_system import types


__all__ = [
'Process',
'Step',
'Composite',
'types',
'pp',
'pf',
'protocol_registry',
'process_registry'
]


pretty = pprint.PrettyPrinter(indent=2)


def pp(x):
"""Print ``x`` in a pretty format."""
pretty.pprint(x)


def pf(x):
"""Format ``x`` for display."""
return pretty.pformat(x)


# register protocols
protocol_registry.register('local', local_lookup)

# register processes
# TODO
# register emitters
process_registry.register('console-emitter', ConsoleEmitter)
process_registry.register('ram-emitter', RAMEmitter)

process_registry.register('database-emitter', DatabaseEmitter)
17 changes: 11 additions & 6 deletions process_bigraph/composite.py
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,6 @@ def find_instances(state, instance_type='process_bigraph.composite.Process'):
for key, inner in state.items():
if isinstance(inner, dict) and isinstance(inner.get('instance'), process_class):
found[key] = inner

return found


Expand Down Expand Up @@ -383,20 +382,24 @@ def find_downstream(steps, nodes, upstream):
down = set([])
for step_path in downstream:
if step_path not in visited:
for output in steps[step_path]['output_paths']:
step_outputs = steps[step_path]['output_paths']
if step_outputs is None:
step_outputs = [] # Ensure step_outputs is always an iterable
for output in step_outputs:
for dependent in nodes[output]['after']:
down.add(dependent)
visited.add(step_path)
downstream ^= down
downstream |= down

return downstream


def determine_steps(steps, remaining, fulfilled):
to_run = []

for step_path in remaining:
step_inputs = steps[step_path]['input_paths']
if step_inputs is None:
step_inputs = []
all_fulfilled = True
for input in step_inputs:
if len(fulfilled[input]) > 0:
Expand All @@ -407,6 +410,8 @@ def determine_steps(steps, remaining, fulfilled):
for step_path in to_run:
remaining.remove(step_path)
step_outputs = steps[step_path]['output_paths']
if step_outputs is None:
step_outputs = []
for output in step_outputs:
fulfilled[output].remove(step_path)

Expand Down Expand Up @@ -459,7 +464,7 @@ def __init__(self, config=None, local_types=None):
composition_schema = types.access(composition)
self.composition = copy.deepcopy(composition_schema)

# find all processes, steps, and emitters in the state
# find all processes, steps, and emitter in the state
self.process_paths = find_instance_paths(
state,
'process_bigraph.composite.Process')
Expand Down Expand Up @@ -833,7 +838,7 @@ def trigger_steps(self, update_paths):

def gather_results(self, queries=None):
'''
a map of paths to emitters --> queries for the emitter at that path
a map of paths to emitter --> queries for the emitter at that path
'''

if queries is None:
Expand Down
Loading