Assume non-LSF host error is flaky

The LSF driver experiences crashes stemming from bsub returning with the
error message 'Request from non-LSF host rejected'. There are reasons to
believe this is not a permanent error, but some flakyness in the IP
infrastructure, and thus should should be categorized as a retriable
failure.

The reason for believing this is flakyness is mostly from the fact that
the same error is also seen on 'bjobs'-calls. If it was a permanent
failure scenario, there would be an enourmous amount of error from these
bjobs calls, but there is not.

src/ert/scheduler/lsf_driver.py

@@ -94,7 +94,14 @@ class RunningJob:
 LSF_INFO_JSON_FILENAME = "lsf_info.json"
 FLAKY_SSH_RETURNCODE = 255
 JOB_ALREADY_FINISHED_BKILL_MSG = "Job has already finished"
-BSUB_FAILURE_MESSAGES = ("Job not submitted",)
+BSUB_FAILURE_MESSAGES = (
+    "Error in rusage section",
+    "Expeced number, string",
+    "No such queue",
+    "Too many processors requested",
+    "cannot be used in the resource requirement section",
+    "duplicate section",
+)
 
 
 def _parse_jobs_dict(jobs: Mapping[str, JobState]) -> dict[str, AnyJob]:

tests/unit_tests/scheduler/test_lsf_driver.py

@@ -578,7 +578,6 @@ async def test_that_bsub_will_retry_and_fail(
             " '&' cannot be used in the resource requirement section. Job not submitted.",
         ),
         (255, "Error in rusage section. Job not submitted."),
-        (255, "Job not submitted."),
     ],
 )
 async def test_that_bsub_will_fail_without_retries(
@@ -604,6 +603,8 @@ async def test_that_bsub_will_fail_without_retries(
     [
         (0, "void"),
         (FLAKY_SSH_RETURNCODE, ""),
+        (0, "Request from non-LSF host rejected"),
+        (FLAKY_SSH_RETURNCODE, "Request from non-LSF host rejected"),
     ],
 )
 async def test_that_bsub_will_retry_and_succeed(