Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

event.app can be None on relation-broken, putting the charm in error state #158

Open
sed-i opened this issue Sep 13, 2024 · 1 comment
Open

event.app can be None on relation-broken, putting the charm in error state #158

sed-i opened this issue Sep 13, 2024 · 1 comment

Comments

@sed-i
Copy link
Contributor

sed-i commented Sep 13, 2024 •
edited
Loading

Via @pedrofragola.
When the prometheus relation is removed,

graph LR
cos-proxy --- prometheus
Loading

the charm goes into error state because event.app is None.

cos-proxy-operator/src/charm.py

Lines 622 to 624 in 49eb7f0

def _prometheus_target_relation_broken(self, event: RelationBrokenEvent):
self._stored.have_targets = False
self._handle_prometheus_alert_rule_files(RULES_DIR, event.app.name) 

App               Version  Status       Scale  Charm             Channel        Rev  Exposed  Message
cos-proxy         n/a      error            1  cos-proxy         latest/stable   73  no       hook failed: "dashboards-relation-broken"

Grafana

2024-09-02 14:34:50 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-02 14:34:53 WARNING unit.cos-proxy/7.juju-log server.go:325 dashboards:699: 'app' expected but not received.
2024-09-02 14:34:53 WARNING unit.cos-proxy/7.juju-log server.go:325 dashboards:699: 'app_name' expected in snapshot but not found.
2024-09-02 14:34:53 ERROR unit.cos-proxy/7.juju-log server.go:325 dashboards:699: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 643, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1685, in remove_dashboards
    app_ids = _type_convert_stored(self._stored.id_mappings.get(event.app.name, ""))  # type: ignore
AttributeError: 'NoneType' object has no attribute 'name'

Prometheus

2024-09-12 09:56:50 ERROR unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 643, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 560, in _prometheus_target_relation_broken
    self._handle_prometheus_alert_rule_files(RULES_DIR, event.app.name)
AttributeError: 'NoneType' object has no attribute 'name'
2024-09-12 09:56:50 ERROR juju.worker.uniter.operation runhook.go:180 hook "prometheus-target-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1
2024-09-12 09:56:50 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-12 10:00:46 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-12 10:01:50 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-12 10:01:53 WARNING unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: 'app' expected but not received.
2024-09-12 10:01:53 WARNING unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: 'app_name' expected in snapshot but not found.
2024-09-12 10:01:53 WARNING unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: 'app_name' expected in snapshot but not found.
2024-09-12 10:01:53 ERROR unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 643, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 560, in _prometheus_target_relation_broken
    self._handle_prometheus_alert_rule_files(RULES_DIR, event.app.name)
AttributeError: 'NoneType' object has no attribute 'name'
2024-09-12 10:01:54 ERROR juju.worker.uniter.operation runhook.go:180 hook "prometheus-target-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1
2024-09-12 10:01:54 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-12 10:05:44 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-12 10:06:54 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
2024-09-12 10:06:57 WARNING unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: 'app' expected but not received.
2024-09-12 10:06:57 WARNING unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: 'app_name' expected in snapshot but not found.
2024-09-12 10:06:57 WARNING unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: 'app_name' expected in snapshot but not found.
2024-09-12 10:06:57 ERROR unit.cos-proxy/7.juju-log server.go:325 prometheus-target:700: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 643, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-7/charm/./src/charm.py", line 560, in _prometheus_target_relation_broken
    self._handle_prometheus_alert_rule_files(RULES_DIR, event.app.name)
AttributeError: 'NoneType' object has no attribute 'name'
2024-09-12 10:06:57 ERROR juju.worker.uniter.operation runhook.go:180 hook "prometheus-target-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1
2024-09-12 10:06:57 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-broken" hook
@sed-i
Copy link
Contributor Author

sed-i commented Sep 13, 2024 •
edited
Loading

It seems like ops's fix should be included in cos-proxy rev73.
Recent ops feature that could be useful here: canonical/operator#1091

We should have an integration test for relation removal. For example something like:

graph LR
ubuntu --- nrpe --- cos-proxy --- scrape-config --- prometheus
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sed-i