Subnets evaluation during terraform plan phase.

[VioletCranberry]

It looks like parsing of locals variables

terraform-oci-oke/modules/network/subnets.tf

Line 53 in 02aeaeb

subnet_info = {

and subsequent

terraform-oci-oke/modules/network/subnets.tf

Line 82 in 02aeaeb

subnets_to_create = try(merge(

terraform-oci-oke/modules/network/subnets.tf

Line 102 in 02aeaeb

resource "null_resource" "validate_subnets" {

terraform-oci-oke/modules/network/subnets.tf

Line 123 in 02aeaeb

resource "oci_core_subnet" "oke" {

is not working as expected.

I am planning to run the cluster on my own subnets and VPC, expecting the resources to be provisioned within these subnets.
Terraform cannot fully evaluate mentioned variables during the plan phase, leading to errors.

Module settings:

create_operator        = false
create_bastion         = false
load_balancers         = "both" (does not really matter in this scenario). 

Terraform Version and Provider Version

❯ terraform --version
Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.4
+ provider registry.terraform.io/hashicorp/helm v2.14.0
+ provider registry.terraform.io/hashicorp/http v3.4.3
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.2
+ provider registry.terraform.io/hashicorp/time v0.11.2
+ provider registry.terraform.io/oracle/oci v5.46.0

Terraform Configuration Files

Subnets are to be created as a separate cloud resources during terraform apply run.

1st scenario - following the example here https://oracle-terraform-modules.github.io/terraform-oci-oke/guide/network_subnets.html (Use existing subnets)
subnets = {
    cp       = { id = oci_core_subnet.cp.id }
    int_lb   = { id = oci_core_subnet.int_lb.id }
    pub_lb   = { id = oci_core_subnet.pub_lb.id }
    workers  = { id = oci_core_subnet.workers.id }
    pods     = { id = oci_core_subnet.pods.id }
 }

2nd scenario - explicitely denying subnets' creation:
subnets = {
    cp       = { create = "never", id = oci_core_subnet.cp.id }
    int_lb   = { create = "never", id = oci_core_subnet.int_lb.id }
    pub_lb   = { create = "never", id = oci_core_subnet.pub_lb.id }
    workers  = { create = "never", id = oci_core_subnet.workers.id }
    pods     = { create = "never", id = oci_core_subnet.pods.id }
  }

Expected Behaviour

Second scenario:

  Error: Invalid for_each argument
│ 
│   on .terraform/modules/oke/modules/network/subnets.tf line 148, in resource "oci_core_security_list" "oke":
│  148:   for_each = {
│  149:     for k, v in local.subnets_to_create : k => v
│  150:     if tobool(lookup(v, "create_seclist", false))
│  151:   }
│     ├────────────────
│     │ local.subnets_to_create will be known only after apply
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will
│ identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully
│ converge.

First scenario:

│ Error: Invalid count argument
│ 
│   on .terraform/modules/oke/modules/network/subnets.tf line 103, in resource "null_resource" "validate_subnets":
│  103:   count = anytrue([for k, v in local.subnet_cidrs_new : contains(["netnum", "newbits", "cidr"], v.type)
│  104:     if lookup(v, "create", "auto") != "never"
│  105:   ]) ? 1 : 0
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this,
│ use the -target argument to first apply only the resources that the count depends on.

+ the error from the second scenario.

Actual Behaviour

terraform plan errors, creating cluster is blocked. VCN and subnets are about to be created successfully.

Steps to Reproduce

Start from scratch, define own VCN and subnets resources, consume IDs with the module.

[hyder]

Hi,
Thanks for logging this issue. Are you running apply by module?

[VioletCranberry]

Hi @hyder what do you mean?

[hyder]

I mean are you using terraform targets? e.g. terraform apply --target=module.network ..

Or just a straightforward terraform apply?

[VioletCranberry]

Ah got it.

This issue involves a straightforward single Terraform run where the subnet resources have yet to be created. The module then uses the ID attributes of these resources. This results in errors during the terraform plan/apply phases since Terraform cannot fully evaluate the mentioned variables until terraform apply is run for the subnets only.

Using --target in Terraform should generally be avoided, as it circumvents Terraform’s dependency graph. This can lead to incomplete or inconsistent infrastructure states. It’s preferable not to rely on targeted applies merely to use a module.

[hyder]

I've tested both of your scenarios and they work. When you are providing the values:

subnets = {
    cp       = { id = oci_core_subnet.cp.id }
    int_lb   = { id = oci_core_subnet.int_lb.id }
    pub_lb   = { id = oci_core_subnet.pub_lb.id }
    workers  = { id = oci_core_subnet.workers.id }
    pods     = { id = oci_core_subnet.pods.id }
 }

are you providing actual ocids, or just like the above? The ids have to be actual ocids strings within ""

[alcampag]

I am having the same issue too, specifying the ids generates:

╷
│ Error: Invalid for_each argument
│ 
│   on .terraform/modules/oke/modules/network/subnets.tf line 148, in resource "oci_core_security_list" "oke":
│  148:   for_each = {
│  149:     for k, v in local.subnets_to_create : k => v
│  150:     if tobool(lookup(v, "create_seclist", false))
│  151:   }
│     ├────────────────
│     │ local.subnets_to_create will be known only after apply
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

Note that I am creating the subnet through a standard terraform module, then I am creating the OKE cluster.
Also, the first apply actually generates the infrastructure, then subsequent plan/apply fail with this error.

[max-tremblay]

I'm in the same situation.

For resource "null_resource" "validate_subnets" in oke/modules/network/subnets.tf, TF doesn't like to rely on v.type when evaluating the count clause:

resource "null_resource" "validate_subnets" {
  count = anytrue([for k, v in local.subnet_cidrs_new : contains(["netnum", "newbits", "cidr"], v.type)
    if lookup(v, "create", "auto") != "never"

So solution is to avoid using v.type like such:

resource "null_resource" "validate_subnets" {
  count = anytrue([for k, v in local.subnet_cidrs_new : (lookup(v, "netnum", null) == null && lookup(v, "newbits", null) != null) || # netbits
          (lookup(v, "netnum", null) != null && lookup(v, "newbits", null) != null) || # netnum
          lookup(v, "cidr", null) != null # cidr
  ]) ? 1 : 0

# The rest [...]
}

I also had the same error from scenario #2 mentioned by @VioletCranberry. It stopped whining in my case when I defined subnets by using create = never on all objects:

  subnets = {
    bastion  = { create = "never" }
    operator = { create = "never" }
    fss      = { create = "never" }
    cp       = { create = "never", id = oci_core_subnet.oke["cp"].id }
    int_lb   = { create = "never", id = oci_core_subnet.oke["int_lb"].id }
    pub_lb   = { create = "never", id = oci_core_subnet.oke["pub_lb"].id }
    workers  = { create = "never", id = oci_core_subnet.oke["workers"].id }
    pods     = { create = "never", id = oci_core_subnet.oke["pods"].id }
  }