Only delete workflows after a configurable timeout

[jameshcorbett]

@bdevcich asked,

would be possible to teardown workflows but leave them around for a configurable period of time (e.g. 24h) before deleting the resources? I'm thinking about nnfdatamovement resources and how they are "lost" to a normal user since they cannot query the flux-coral-dws2 journal. Then flux could query the nnfdatamovement resources for a workflow after the fact without having to store it.

The nnfdatamovement resources exist in kubernetes. If Flux were to somehow query the nnfdatamovement resources, it would have to send requests to kubernetes at users' demands, while also restricting requests for a user to only the resources associated with their jobs. Otherwise it would have to store them someplace, but since the resources are too large to go in the KVS, some new solution would have to be worked out.

Just leaving workflows around for admins to inspect the nnfdatamovement resource though could be done by just adding the timeout.