Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apparent hello storm after scheduler reloaded #6294

Open
grondo opened this issue Sep 19, 2024 · 2 comments
Open

apparent hello storm after scheduler reloaded #6294

grondo opened this issue Sep 19, 2024 · 2 comments

Comments

@grondo
Copy link
Contributor

grondo commented Sep 19, 2024 •
edited
Loading

In the logs captured below, the scheduler was apparently loadedat 10:58, then we see the messages:

[ +45.189807] job-manager[0]: alloc: stop due to disconnect: Success
[ +47.946744] sched-fluxion-resource[0]: disconnect_request_cb: a notify request aborted

Followed by what appears to be a hello storm, with at least 35 hello requests sent to the job manager if the logs can be believed. Full logs here:

[Sep19 10:58] job-manager[0]: scheduler: hello
[Sep19 10:59] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[  +3.152109] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[  +4.710717] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[  +6.724351] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[  +7.308041] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[  +7.921133] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +7.955663] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +7.983040] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.007288] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +8.030388] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.070941] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.073458] job-manager[0]: scheduler: ready unlimited
[Sep19 11:00] job-exec[0]: job-exception: id=f7fZxjpetmd: resource allocation expired
[  +5.002067] job-exec[0]: Sending SIGKILL to job f7fZxjpetmd
[  +5.972579] job-manager[0]: housekeeping: f7fZxjpetmd started
[ +45.189807] job-manager[0]: alloc: stop due to disconnect: Success
[ +47.947512] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947585] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947646] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947707] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947767] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947827] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.946744] sched-fluxion-resource[0]: disconnect_request_cb: a notify request aborted
[ +47.947885] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947947] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948005] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948066] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948124] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948185] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948244] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948301] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948358] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948428] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948643] job-manager[0]: scheduler: hello
[ +47.951330] job-manager[0]: scheduler: hello
[ +47.951430] job-manager[0]: scheduler: hello
[ +47.954927] job-manager[0]: scheduler: hello
[ +47.955021] job-manager[0]: scheduler: hello
[ +47.955696] job-manager[0]: scheduler: hello
[ +47.955788] job-manager[0]: scheduler: hello
[ +47.955884] job-manager[0]: scheduler: hello
[ +47.957521] job-manager[0]: scheduler: hello
[ +47.957697] job-manager[0]: scheduler: hello
[ +47.957812] job-manager[0]: scheduler: hello
[ +47.957880] job-manager[0]: scheduler: hello
[ +47.957952] job-manager[0]: scheduler: hello
[ +47.959617] job-manager[0]: scheduler: hello
[ +47.959925] job-manager[0]: scheduler: hello
[ +47.959999] job-manager[0]: scheduler: hello
[ +47.960738] job-manager[0]: scheduler: hello
[ +47.960803] job-manager[0]: scheduler: hello
[ +47.961847] job-manager[0]: scheduler: hello
[ +47.963804] job-manager[0]: scheduler: hello
[ +47.966084] job-manager[0]: scheduler: hello
[ +47.966188] job-manager[0]: scheduler: hello
[ +47.966276] job-manager[0]: scheduler: hello
[ +47.966767] job-manager[0]: scheduler: hello
[ +47.966838] job-manager[0]: scheduler: hello
[ +47.966899] job-manager[0]: scheduler: hello
[ +47.966968] job-manager[0]: scheduler: hello
[ +47.967031] job-manager[0]: scheduler: hello
[ +47.967099] job-manager[0]: scheduler: hello
[ +47.967167] job-manager[0]: scheduler: hello
[ +47.967233] job-manager[0]: scheduler: hello
[ +47.967301] job-manager[0]: scheduler: hello
[ +49.118986] job-manager[0]: housekeeping: f7fZxjpetmd complete
[ +50.025627] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025716] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025807] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025895] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025974] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026049] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026120] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026193] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026266] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026342] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026429] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026503] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026577] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026645] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026712] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026788] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026855] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026925] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027000] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027072] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027143] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027210] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027277] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027345] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027424] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027492] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027580] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027650] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027717] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027795] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027871] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027949] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +53.302036] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.302466] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.302844] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.303211] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.303579] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.303934] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.304288] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.304655] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.305011] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.305362] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.305725] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.306081] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.306445] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.306798] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.307152] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.307525] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.308063] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.308641] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.309197] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.309775] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.310351] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.310929] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.311496] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.312057] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.312585] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.313087] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.313588] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.314077] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.314579] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.315070] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.315597] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.316095] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.316235] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316296] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316340] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316386] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316436] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316477] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316521] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316577] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316631] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316676] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316719] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316761] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316804] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316845] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316884] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316931] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316977] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.317008] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.318064] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.319966] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.320115] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.320217] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.320318] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.320426] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.321449] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.321961] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.322055] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.322718] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.322821] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.323029] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.323104] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.323179] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.323989] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.324272] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324352] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324428] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324505] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324563] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324618] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324672] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324729] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324849] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324915] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.325133] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325241] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325478] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325661] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325724] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.326122] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.326534] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.326623] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.327162] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.327469] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.327532] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.328196] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.328262] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.328430] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.329002] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329222] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.329287] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329426] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329559] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329767] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.330115] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.330183] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331042] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.331280] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331348] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331478] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331548] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331620] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331686] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331746] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332003] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332069] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.332169] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332260] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332398] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332468] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332530] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[ +53.332586] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332644] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.333515] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.334151] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.334217] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.334273] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.334524] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.335621] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.335686] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.335804] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.336159] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[ +53.336246] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.336311] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.336398] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.337001] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.337070] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.337345] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.338276] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.338789] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.339201] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.339449] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[ +53.339586] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.339749] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.339935] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340046] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340158] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340266] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340365] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340472] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340653] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340756] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340879] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340986] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.341142] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.341515] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.341694] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.341950] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.342157] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.342504] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[ +53.342817] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.342928] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.343284] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.343390] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.343495] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.343946] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.344426] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.344533] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.344797] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345288] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345397] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345504] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345598] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345695] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345792] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345889] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345982] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346080] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.346175] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346267] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346360] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346471] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346640] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346971] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.347073] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.347180] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.347396] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.347519] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.348542] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[ +53.348841] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.348943] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.349180] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.349286] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[Sep19 11:01] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.000405] job-manager[0]: scheduler: ready unlimited
[  +0.000519] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.000628] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.000897] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.001004] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.001327] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[  +0.001453] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.001555] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.001895] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[  +0.002004] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.002106] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.002205] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.003214] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.003340] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.003643] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.003799] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.004314] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.004666] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.004798] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.004902] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005005] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005105] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005204] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005310] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005692] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.006067] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.006316] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.006819] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.007014] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.007552] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.007636] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.007826] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.007911] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.008208] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.008371] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.009506] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.009700] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.009844] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.010418] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.010652] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.010765] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.010939] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.011246] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.011308] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.011630] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.011898] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821248] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821317] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.821400] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821448] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821503] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821548] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821594] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821645] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.821689] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821733] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821777] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.821822] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821865] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821907] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821948] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822005] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.822059] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822102] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822146] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822203] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.822257] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822299] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822342] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822393] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822439] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822626] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822670] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822711] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822752] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822796] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822838] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.298808] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305264] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305345] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305414] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305469] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305520] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305584] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305641] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305692] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.305770] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305816] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305864] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.305910] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.305956] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306003] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.306046] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306094] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306191] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306252] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306436] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306488] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.306539] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306584] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306665] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306710] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306756] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.306812] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306895] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306947] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306993] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.307036] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.517900] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.518481] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.518645] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.519121] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.519254] job-manager[0]: scheduler: ready unlimited
[  +8.519692] job-manager[0]: scheduler: ready unlimited
[  +8.519710] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.520158] job-manager[0]: scheduler: ready unlimited
[  +8.520277] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.520435] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520497] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520552] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520621] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520678] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520715] job-manager[0]: scheduler: ready unlimited
[  +8.520734] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520788] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520847] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.520903] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520953] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.521019] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521075] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521130] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521186] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521246] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521324] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521416] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521511] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521840] job-manager[0]: scheduler: ready unlimited
[  +8.522091] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.522286] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.522370] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.522463] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.522533] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.522608] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.522673] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.523446] job-manager[0]: scheduler: ready unlimited
[  +9.146982] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.147564] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.148109] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.148350] job-manager[0]: scheduler: ready unlimited
[  +9.148679] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.148692] job-manager[0]: scheduler: ready unlimited
[  +9.149229] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.149329] job-manager[0]: scheduler: ready unlimited
[  +9.149525] job-manager[0]: scheduler: ready unlimited
[  +9.149776] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.150314] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.150688] job-manager[0]: scheduler: ready unlimited
[  +9.150871] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.150927] job-manager[0]: scheduler: ready unlimited
[  +9.151269] job-manager[0]: scheduler: ready unlimited
[  +9.151416] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.151951] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.152182] job-manager[0]: scheduler: ready unlimited
[  +9.152493] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.152632] job-manager[0]: scheduler: ready unlimited
[  +9.152942] job-manager[0]: scheduler: ready unlimited
[  +9.153028] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.153570] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.154103] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.154109] job-manager[0]: scheduler: ready unlimited
[  +9.154149] job-manager[0]: scheduler: ready unlimited
[  +9.154607] job-manager[0]: scheduler: ready unlimited
[  +9.154646] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.155067] job-manager[0]: scheduler: ready unlimited
[  +9.155179] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.155720] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.155751] job-manager[0]: scheduler: ready unlimited
[  +9.156252] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.156793] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.156824] job-manager[0]: scheduler: ready unlimited
[  +9.156936] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.157030] job-manager[0]: scheduler: ready unlimited
[  +9.157427] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.157520] job-manager[0]: scheduler: ready unlimited
[  +9.157578] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.157942] job-manager[0]: scheduler: ready unlimited
[  +9.158059] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.158200] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.158268] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.158586] job-manager[0]: scheduler: ready unlimited
[  +9.159212] job-manager[0]: scheduler: ready unlimited
[  +9.736240] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.736825] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.737290] job-manager[0]: scheduler: ready unlimited
[  +9.737377] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.737772] job-manager[0]: scheduler: ready unlimited
[  +9.737934] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.738474] job-manager[0]: scheduler: ready unlimited
[  +9.739052] job-manager[0]: scheduler: ready unlimited
@garlick
Copy link
Member

garlick commented Sep 19, 2024

@trws - is there any way the exception wrapper fluxion puts around mod_main() could be causing it to be called repeatedly or recursively? That woud be

https://github.com/flux-framework/flux-sched/blob/master/src/common/c%2B%2Bwrappers/eh_wrapper.hpp

which is way over my head.

@trws
Copy link
Member

trws commented Sep 22, 2024

It certainly shouldn't be. The passed function object is called only once. If there's a way the call from the event loop can be retried, like if it's not reset or something, then it's possible an exception could be causing it to exit before hitting that point and the event loop is re-calling it but it would have to be something like that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@garlick @trws @grondo