Add retry logic for database and broker connections

[jmborr]

Description of the changes

When testfixture01 is deployed in the test environment of webmonchow, there's no guarantee that neither the AMQ broker or the Webmon database servers have deployed. Because of this reason, webmonchow will retry (every 5 seconds by default) to connect to these services, in the hope that they will eventually run.

Check all that apply:

• added release notes (if not, provide an explanation in the work description)
• updated documentation
• Source added/refactored
• Added unit tests
• Added integration tests

References:

• Links to IBM EWM items: 6972, 6968

Manual test for the reviewer

Deploy WebMon locally in your machine, but edit Dockerfile.webmonchow so that instead of using the webmonchow conda package, it will use the webmonchow feature branch relentless_attempt_db_connection. The edited Dockerfile:

# Miniconda with native Python 3.10
FROM continuumio/miniconda3:23.3.1-0
RUN conda install --yes -n base conda-libmamba-solver

# Install webmonchow
#RUN conda install --yes --solver=libmamba -n base -c conda-forge -c neutrons postgresql=14 webmonchow=1.0.0

# Install webmonchow from a feature branch (only to test the latest changes in webmonchow)
RUN conda install --yes --solver=libmamba -n base -c conda-forge postgresql=14
RUN python -m pip install git+https://github.com/neutrons/webmonchow.git@relentless_attempt_db_connection#egg=webmonchow

# Start the AMQ and PV generators when starting the container
CMD ["sh", "-c", "broadcast_pv & broadcast_amq --broker \"activemq:61613\" & wait"]

Check list for the reviewer

• release notes updated, or an explanation is provided as to why release notes are unnecessary
• best software practices
  • clearly named variables (better to be verbose in variable names)
  • code comments explaining the intent of code blocks
• All the tests are passing
• The documentation is up to date
• code comments added when explaining intent

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.95%. Comparing base (f6940da) to head (2a357c9).

Additional details and impacted files

@@            Coverage Diff             @@
##             next      #12      +/-   ##
==========================================
+ Coverage   89.51%   92.95%   +3.44%     
==========================================
  Files           3        3              
  Lines         124      142      +18     
==========================================
+ Hits          111      132      +21     
+ Misses         13       10       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[backmari]

Local deployment of WebMon pointing to webmonchow in this branch works as expected. The instruments PV:s are being updated and DASMON status is green.

[backmari]

Good addition with retry logic 🙌

[jmborr]

@backmari thanks for the review!