Monitoring code location status

[stevenmurphy12]

I'm running a Kubernetes deployment with 3 code locations.

Over the weekend, a scheduled job failed because the code location containing that schedule failed to load (due to a transient issue). I have Slack notifications enabled for when a job fails, though it did not work in this case because the job was never triggered (as the daemon would have no awareness of it).

Is there a recommended means by which I could monitor the status of a code location? My current thought is I'd need to have an external process to periodically ping the Deployment tab of the UI and look for code locations that are showing as failed.

[alangenfeld]

periodically ping the Deployment tab of the UI and look for code locations that are showing as failed

You can query the webserver directly using GraphQL to get to the underlying information in the same way the UI does. You can reference the query that the UI does by observing it in the network tab of your browsers developer tools or locating the query in the dagster code base.

https://docs.dagster.io/concepts/webserver/graphql#graphql-api

[garethbrickman]

Here's an example GraphQL query to check on the state / health of code locations:

query LocationStatusesQuery {
  workspaceOrError {
	  ... on Workspace{
      locationEntries {
        ... on WorkspaceLocationEntry {
          id
          loadStatus
          locationOrLoadError {
            __typename
          }
        }
      }
    }
  }
}

loadStatus will be LOADING or LOADED. You can use that additional locationOrLoadError field to distinguish between error and not. If the typename is PythonError, it loaded with an error state. If it's a RepositoryLocation it loaded in a working state.

Example response:

{'data': {'workspaceOrError': {'locationEntries': [{'id': 'code-server-1',
                                                    'loadStatus': 'LOADED',
                                                    'locationOrLoadError': {'__typename': 'RepositoryLocation'}},
                                                   {'id': 'code-server-2',
                                                    'loadStatus': 'LOADED',
                                                    'locationOrLoadError': {'__typename': 'RepositoryLocation'}},
                                                   {'id': 'code-server-3',
                                                    'loadStatus': 'LOADED',
                                                    'locationOrLoadError': {'__typename': 'PythonError'}}]}}}

[alangenfeld]

Dagster Cloud users can set up code location alerts policies
https://docs.dagster.io/dagster-cloud/managing-deployments/setting-up-alerts