Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Execution workflows loop infinitely on fatal errors with CM #11

Open
vholer opened this issue Jan 4, 2018 · 0 comments
Open

Execution workflows loop infinitely on fatal errors with CM #11

vholer opened this issue Jan 4, 2018 · 0 comments

Comments

@vholer
Copy link
Contributor

vholer commented Jan 4, 2018

If scale out/in workflows fail on a fatal error (e.g., VM fails to boot and is manually cleaned), the Cloudify Manager stuck in the infinite loop of retries.

  Timestamp Event Type Log Level Operation Node Name Node ID Message
  2018-01-04 20:45:41 Task failed - cloudify.interfaces.lifecycle.start workerNode workerNode_jmlkc1 Task failed 'cloudify_occi_plugin.tasks.start' -> Failed to run occi: F, [2018-01-04T20:45:41.693050 #21070] FATAL -- : [rOCCI-cli] An error occurred! Message: Net::HTTP::Get with ID["85dc1002-3764-45f9-950f-f4c22a57ad3e"] failed! HTTP Response status: [404] Not Found : "Instance with ID 114237 does not exist!" [retry 2463]
  2018-01-04 20:45:41 - INFO cloudify.interfaces.lifecycle.start workerNode workerNode_jmlkc1 Exited with code=1
  2018-01-04 20:45:39 - INFO cloudify.interfaces.lifecycle.start workerNode workerNode_jmlkc1 Executing ['/usr/local/bin/occi', '--output-format', 'json', '--endpoint', u'https://carach5.ics.muni.cz:11443', '--auth', u'x509', '--user-cred', u'/tmp/x509up_u1000', '--voms', '--action', 'describe', '--resource', u'https://carach5.ics.muni.cz:11443/compute/114237']
  2018-01-04 20:45:39 - INFO cloudify.interfaces.lifecycle.start workerNode workerNode_jmlkc1 Starting node
  2018-01-04 20:45:38 Task started - cloudify.interfaces.lifecycle.start workerNode workerNode_jmlkc1 Task started 'cloudify_occi_plugin.tasks.start' [retry 2463]
  2018-01-04 20:45:38 Task sent - cloudify.interfaces.lifecycle.start workerNode workerNode_jmlkc1 Sending task 'cloudify_occi_plugin.tasks.start' [retry 2463]

See 2463 retries of just checking something which doesn't exist anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant