inability to create multiple vcd_nsxt_edgegateway resources concurrently

[brainplot]

Hello,

I am unable to create multiple vcd_nsxt_edgegateway resources because of what appears to be some sort of global lock against concurrent writes.

Terraform Version

Terraform v1.9.5
on linux_amd64
+ provider registry.terraform.io/vmware/vcd v3.14.0

Affected Resource(s)

Please list the resources as a list, for example:

• vcd_nsxt_edgegateway

Terraform Configuration Files

This is a minimal reproducible example that shows the issue. The subnet ranges may need to be adjusted according to the environment this code is tested on.

terraform {
  required_providers {
    vcd = {
      source  = "vmware/vcd"
      version = "~> 3.0"
    }
  }
}

variable "vcd_user" {
  type = string
}

variable "vcd_password" {
  type      = string
  sensitive = true
}

variable "vcd_url" {
  type = string
}

variable "vcd_vdc" {
  type = string
}

variable "vcd_org" {
  type = string
}

variable "vcd_external_network_name" {
  type = string
}

variable "edge_gateway_subnets" {
  type = set(string)
  default = [
    "10.20.251.0/24",
    "10.20.252.0/24",
    "10.20.253.0/24",
    "10.20.254.0/24",
    "10.20.255.0/24",
  ]
}

provider "vcd" {
  user     = var.vcd_user
  password = var.vcd_password
  org      = "System"
  url      = var.vcd_url
  vdc      = var.vcd_vdc

  allow_unverified_ssl = true
}

data "vcd_org_vdc" "this" {
  name = var.vcd_vdc
  org  = var.vcd_org
}

data "vcd_external_network_v2" "this" {
  name = var.vcd_external_network_name
}

locals {
  first_ip_scope = tolist(data.vcd_external_network_v2.this.ip_scope)[0]
}

resource "vcd_nsxt_edgegateway" "this" {
  for_each = var.edge_gateway_subnets

  org      = var.vcd_org
  owner_id = data.vcd_org_vdc.this.id
  name     = format("issue-egw-%s", replace(each.value, "/[./]/", "-"))

  external_network_id = data.vcd_external_network_v2.this.id

  subnet {
    gateway       = local.first_ip_scope.gateway
    prefix_length = local.first_ip_scope.prefix_length
    primary_ip    = cidrhost(each.value, 0)
    allocated_ips {
      start_address = cidrhost(each.value, 0)
      end_address   = cidrhost(each.value, 255)
    }
  }
}

Debug Output

Here's the Gist with both files: https://gist.github.com/brainplot/21d22e847a60eace8ce2ef74a1795677

Expected Behavior

Ability to create multiple vcd_nsxt_edgegateway resources when either count or for_each are present.

Actual Behavior

By default, Terraform tries to create up to 10 resources in parallel so that means it will try to create all 5 (in my example) vcd_nsxt_edgegateway resources at the same time, failing with the following error:

Error: error creating NSX-T Edge Gateway: error creating Edge Gateway: error waiting completion of task (https://<REDACTED>/api/task/599a293e-e13c-4d2e-9413-5515e20a3a04): task did not complete successfully:  [500:INTERNAL_SERVER_ERROR] - [ 22-2024-09-19-12-33-45-963--cfae3c5b-e02b-494c-9cc0-e1e669ed6401 ] Internal Server Error
 - Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1
 - Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1

  with vcd_nsxt_edgegateway.this["10.20.252.0/24"],
  on main.tf line 69, in resource "vcd_nsxt_edgegateway" "this":
  69: resource "vcd_nsxt_edgegateway" "this" {

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

I can work around this by setting -parallelism=2 in terraform apply but that slows down the entire deployment just because of this one resource

References

Unfortunately, Terraform does not provide a way to control per-resource parallelism, as noted in hashicorp/terraform-plugin-sdk#67. According to that same issue, the provider should handle such cases, as noted by @apparentlymart.