vcd_nsxt_ipsec_vpn_tunnel tunnel1 is ok tunnel2 is failing

[andrzej-dobrzynski-pushgaming]

Hello @adambarreiro et al

vcd_nsxt_ipsec_vpn_tunnel tunnel1 is ok tunnel2 is failing

Terraform Version

% tfenv version-name
1.5.7

Affected Resource(s)

vcd_nsxt_ipsec_vpn_tunnel

Terraform Configuration Files

resource "vcd_nsxt_ipsec_vpn_tunnel" "tunnel1_primary" {
  provider          = vcd.primary
  edge_gateway_id   = data.vcd_nsxt_edgegateway.primary.id
  name              = "${var.platform_instance}-tunnel1"
  pre_shared_key    = aws_vpn_connection.primary.tunnel1_preshared_key
  local_ip_address  = data.vcd_nsxt_edgegateway.primary.primary_ip
  local_networks    = [var.vcd_vdc_primary_cidr_block]
  remote_ip_address = aws_vpn_connection.primary.tunnel1_address
  remote_networks   = [var.aws_vpc_main_cidr_block]
  logging           = true
  # tags = {
  #   policy_rule_priority = 1
  # }
}

resource "vcd_nsxt_ipsec_vpn_tunnel" "tunnel2_primary" {
  provider          = vcd.primary
  edge_gateway_id   = data.vcd_nsxt_edgegateway.primary.id
  name              = "${var.platform_instance}-tunnel2"
  pre_shared_key    = aws_vpn_connection.primary.tunnel2_preshared_key
  local_ip_address  = data.vcd_nsxt_edgegateway.primary.primary_ip
  local_networks    = [var.vcd_vdc_primary_cidr_block]
  remote_ip_address = aws_vpn_connection.primary.tunnel2_address
  remote_networks   = [var.aws_vpc_main_cidr_block]
  logging           = true
  # tags = {
  #   policy_rule_priority = 2
  # }
}

Expected Behavior

tunnel1 is ok 
tunnel2 is ok

Actual Behavior

tunnel1 is ok 
tunnel2 is failing

vcd_nsxt_ipsec_vpn_tunnel.tunnel1_primary: Creating...
vcd_nsxt_ipsec_vpn_tunnel.tunnel1_primary: Creation complete after 8s [id=859eaaef-6a88-4fbd-a187-7752c04280cb]

vcd_nsxt_ipsec_vpn_tunnel.tunnel2_primary: Creating...
╷
│ Error: [nsx-t ipsec vpn tunnel create] error creating NSX-T IPsec VPN Tunnel configuration: task failed while creating NSX-T IPsec VPN Tunnel configuration: task did not complete successfully:  [400:INTERNAL_SERVER_ERROR] - [ 10-2024-09-14-16-01-50-490--cc277ce8-1653-45a6-9313-6462864eeb8e ] Bad Request: Found errors in the request. Please refer to the related Errors for details. Related errors: Rule=[/infra/tier-1s/b2a515a7-a193-47c0-ac25-a747b09f0529/ipsec-vpn-services/b2a515a7-a193-47c0-ac25-a747b09f0529/sessions/f0ddfe5a-87a1-4e34-9616-beca8c68029a/rules/f0ddfe5a-87a1-4e34-9616-beca8c68029a] has source and destination networks overlapping with existing rule=[/infra/tier-1s/b2a515a7-a193-47c0-ac25-a747b09f0529/ipsec-vpn-services/b2a515a7-a193-47c0-ac25-a747b09f0529/sessions/859eaaef-6a88-4fbd-a187-7752c04280cb/rules/859eaaef-6a88-4fbd-a187-7752c04280cb]., error code 500060
│ 
│   with vcd_nsxt_ipsec_vpn_tunnel.tunnel2_primary,
│   on c8-primary.tf line 117, in resource "vcd_nsxt_ipsec_vpn_tunnel" "tunnel2_primary":
│  117: resource "vcd_nsxt_ipsec_vpn_tunnel" "tunnel2_primary" {
│ 
╵

Steps to Reproduce

as above

References

https://knowledge.broadcom.com/external/article/325098/overlapping-subnets-in-ipsec-vpn-policy.html

following up link above can tags be added to terraform resource ?

resource "vcd_nsxt_ipsec_vpn_tunnel" "tunnel1_primary" {
  tags = {
    policy_rule_priority = 1
  }

resource "vcd_nsxt_ipsec_vpn_tunnel" "tunnel2_primary" {
  tags = {
    policy_rule_priority = 2
  }

[Didainius]

Hello @andrzej-dobrzynski-pushgaming ,
Would the error hint anything?

Error: [nsx-t ipsec vpn tunnel create] error creating NSX-T IPsec VPN Tunnel configuration: task failed while creating NSX-T IPsec VPN Tunnel configuration: task did not complete successfully:  [400:INTERNAL_SERVER_ERROR] - [ 10-2024-09-14-16-01-50-490--cc277ce8-1653-45a6-9313-6462864eeb8e ] Bad Request: Found errors in the request. Please refer to the related Errors for details. Related errors: Rule=[/infra/tier-1s/b2a515a7-a193-47c0-ac25-a747b09f0529/ipsec-vpn-services/b2a515a7-a193-47c0-ac25-a747b09f0529/sessions/f0ddfe5a-87a1-4e34-9616-beca8c68029a/rules/f0ddfe5a-87a1-4e34-9616-beca8c68029a] has source and destination networks overlapping with existing rule=[/infra/tier-1s/b2a515a7-a193-47c0-ac25-a747b09f0529/ipsec-vpn-services/b2a515a7-a193-47c0-ac25-a747b09f0529/sessions/859eaaef-6a88-4fbd-a187-7752c04280cb/rules/859eaaef-6a88-4fbd-a187-7752c04280cb]., error code 500060

In particular the Found errors in the request. Please refer to the related Errors for details. Related errors: Rule=[/infra/tier-1s/b2a515a7-a193-47c0-ac25-a747b09f0529/ipsec-vpn-services/b2a515a7-a193-47c0-ac25-a747b09f0529/sessions/f0ddfe5a-87a1-4e34-9616-beca8c68029a/rules/f0ddfe5a-87a1-4e34-9616-beca8c68029a] has source and destination networks overlapping with existing rule=[/infra/tier-1s/b2a515a7-a193-47c0-ac25-a747b09f0529/ipsec-vpn-services/b2a515a7-a193-47c0-ac25-a747b09f0529/sessions/859eaaef-6a88-4fbd-a187-7752c04280cb/rules/859eaaef-6a88-4fbd-a187-7752c04280cb]., error code 500060

In this case the errors comes directly from VCD API and it looks you have coinciding subnets. Could you try to create just the second tunnel without the first?

[andrzej-dobrzynski-pushgaming]

Hello @Didainius

i destroyed tunnel1 and i created tunnel2 is ok now but aws site to site vpn needs both tunnel1 and tunnel2 be ok for redundancy

https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNTunnels.html

thus i assume tunnel1 and tunnel2 should have overlapping subnets but different priorities whereby traffic goes via tunnel1 in normal running or via tunnel2 failing tunnel1 ?

[Didainius]

Hello @Didainius

i destroyed tunnel1 and i created tunnel2 is ok now but aws site to site vpn needs both tunnel1 and tunnel2 be ok for redundancy

https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNTunnels.html

thus i assume tunnel1 and tunnel2 should have overlapping subnets but different priorities whereby traffic goes via tunnel1 in normal running or via tunnel2 failing tunnel1 ?

I am afraid I can't comment much on infrastructure requirements and configuration. Are you able to achieve such configuration using VCD UI?

[andrzej-dobrzynski-pushgaming]

no since VCD UI does not expose NSXT tags but NSXT UI does therefore the ask if tags can be added to terraform resource

[Didainius]

Ah, so the point is that if VCD UI doesn't have tags - the API won't have it either. And this provider only uses VCD APIs (not NSX-T)