Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batched heartbeat got canceled due to caller context cancellation #1574

Open
codemonkeycxy opened this issue Jul 31, 2024 · 1 comment
Open

Comments

@codemonkeycxy
Copy link

Expected Behavior

Say I have the following activity code

func (a *Activities) MyActivity(ctx) (out, error)(
  g, gctx := errgroup.WithContext(ctx)
  g.SetLimit(4)
  for _, batch := range batches {
    batch := batch
    g.Go(func() error {
      activity.RecordHeartbeat(gctx)
      // Goroutine logic
    })
  }
  if err := g.Wait(); err != nil {
    return nil, err
  }

  for _, item := range array {
    activity.RecordHeartbeat(ctx)
    // Additional logic
  }
}

I expect heartbeat to get sent out to the server and my activity to NOT time out

Actual Behavior

Activity timed out with activity Heartbeat timeout

This is because g.Wait() cancels gctx but gctx is still referenced in SDK Heartbeat function's goroutine https://github.com/temporalio/sdk-go/blob/master/internal/internal_task_handlers.go#L2031. Due to the batch heartbeat logic, gctx was used after g.Wait() had finished in my code and caused a context canceled error when making the call to the server

I could change activity.RecordHeartbeat(gctx) to activity.RecordHeartbeat(ctx) but wonder if SDK can avoid this pitfall by using context.Background()

Steps to Reproduce the Problem

Specifications

  • Version: go.temporal.io/sdk v1.27.0
  • Platform: Linux
@Quinn-With-Two-Ns
Copy link
Contributor

Talked offline discussed we should:

  • Clarify the documentation around how RecordHeartbeat will behave if the calling context is cancelled
  • If RecordHeartbeat is called with multiple contexts we should use the last non cancelled context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants