Terraform not honouring OS IPv4 settings, using IPv6 dst to call *.googleapis.com

[mhanline]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

tf version
Terraform v0.12.28
+ provider.google v3.29.0
+ provider.google-beta v3.29.0

Affected Resource(s)

All resources, not specific to any one.

Terraform Configuration Files

While this happens intermittently and it's not specific to this config, it seems to happen with longer Terraform runs. You may need to apply / destroy 1-2 times before seeing this issue.

gist link to config

Debug Output

I see this output sporadically, and not on the same API call. Note the DST IP is an IPv6 address, but Cloud Shell does not enable IPv6 in the OS:
Link to gist

Console output when issue occurs (Note the IPv6 address is being used):

Error: Error when reading or editing Project Service [project-id]/trafficdirector.googleapis.com: Get "https://cloudresourcemanager.googleapis.com/v1/projects/[project-id]?alt=json&prettyPrint=false": dial tcp [2404:6800:4003:c00::5f]:443: connect: cannot assign requested address

Error: Error retrieving available container cluster versions: Get "https://container.googleapis.com/v1beta1/projects/[project-id]/locations/asia-east1-c/serverConfig?alt=json&prettyPrint=false": dial tcp [2404:6800:4003:c04::5f]:443: connect: cannot assign requested address

Error: Error when reading or editing Project Service [project-id]/trafficdirector.googleapis.com: Get "https://cloudresourcemanager.googleapis.com/v1/projects/[project-id]?alt=json&prettyPrint=false": dial tcp [2404:6800:4003:c03::5f]:443: connect: cannot assign requested address

Expected Behavior

Terraform / Google provider should respect the OS network settings and use IPv4 addresses to call out to *.googleapis.com.

Actual Behavior

tf apply / tf destroy does not always successfully complete, and will return the errors above.

Steps to Reproduce

1. Open Google Cloud Shell (no IPv6 stack)
2. Run tf apply or tf destroy on the linked config
3. Most times it will succeed, but about every second attempt it report the above errors

Note, if I statically configure /etc/hosts to resolve to a specific IPv4 address - say 199.36.153.8, the above errors never occur.

Important Factoids

Authenticating using application default credentials, built into Cloud Shell.

Confirm IPv6 is not enabled on the OS:

myusername@cloudshell:~$ sudo sysctl -n net.ipv6.conf.all.disable_ipv6 && sysctl -n net.ipv6.conf.default.disable_ipv6
1
1

References

Similar issue 1 (with Go)
Similar issue 2
Workaround solution

• b/160321706

[danawillow]

Here's what I know so far:

Based on golang/go#25321 and vmware/terraform-provider-vsphere#636, something that could fix it would be to compile with CGO enabled.
The build script that I assume our release pipeline uses explicitly disables CGO. This was introduced in hashicorp/terraform#7107 because it ensures the compiled binaries are statically linked (hashicorp/terraform#6714).
If I'm reading https://blog.madewithdrew.com/post/statically-linking-c-to-go/ right, then there should be a way to resolve this without having to explicitly disable CGO. It's also possible that things are different now than they were 4 years ago when the previous issues were brought up.

@megan07, is that indeed the build script that's used for the providers? If you don't mind, could you ask around to see if anyone at HashiCorp has any ideas on this? In the meantime, marking it upstream since I think it'll be good to have open as a reference for people that run into this, but I don't expect there being much we can do on the provider end.

[shermanyin]

I'm running into similar issue intermittently as well in GCP cloud shell.

$ ~/bin/terraform --version
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/google v3.49.0
+ provider registry.terraform.io/hashicorp/google-beta v3.49.0
+ provider registry.terraform.io/hashicorp/http v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/time v0.6.0

In my particular case, my script uploads a file to a Windows Server. I first get this error:

module.dc.null_resource.upload-scripts: Still creating... [4m50s elapsed]
Error: timeout - last error: unknown error Post "https://35.236.28.181:5986/wsman": dial tcp 35.236.28.181:5986: i/o timeout

I checked that the firewalls are opened to the IP of the cloudshell instance. I try to do terraform apply again, and I would run into these "cannot assign requested address" errors while refreshing state. e.g. my first run I get:

module.cac.module.cac-regional[0].random_shuffle.zone: Refreshing state... [id=-]

Error: Error when reading or editing ComputeNetwork "projects/[project-id]/global/networks/vpc-cas": Get "https://compute.googleapis.com/compute/v1/projects/[project-id]/global/networks/vpc-cas?alt=json": dial tcp [2607:f8b0:400e:c09::5f]:443: connect: cannot assign requested address

Then immediately I run terraform apply again, and it would fail in a different place.

google_compute_router_nat.nat: Refreshing state... [id=[project-id]/us-west2/router/nat]

Error: Error when reading or editing Storage Bucket "pcoip-scripts-7d731c": Get "https://storage.googleapis.com/storage/v1/b/pcoip-scripts-7d731c?alt=json&prettyPrint=false": dial tcp [2607:f8b0:400e:c07::80]:443: connect: cannot assign requested address

Finally, 3rd time it would let me type "yes" to apply the changes, but it will fail again timing out trying to upload the files. We run this same script a few times a week but most of the time there are no issues.

$ sysctl  net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
$ sysctl  net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.default.disable_ipv6 = 1

[bharathkkb]

@c2thorn @rileykarson some of the team members have been running into this lately. Any possibilities for a fix?

/cc @daniel-cit

[ocervell]

same here, could you give steps to resolve ?

[ferrarimarco]

Hi there :)

We experienced this as well in a relatively long terraform apply (5-6 mins), running from Cloud Shell. Thanks for your support!

Error: Error creating service account: Post "https://iam.googleapis.com/v1/projects/[REDACTED_PROJECT_ID]/serviceAccounts?alt=json&prettyPrint=false": dial tcp [REDACTED_IP_V6_ADDRESS]:443: connect: cannot assign requested address

/cc @jbrook

[isimluk]

Quick and dirt plug:

# Workaround https://github.com/hashicorp/terraform-provider-google/issues/6782
    sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1 > /dev/null
    export APIS="googleapis.com www.googleapis.com storage.googleapis.com iam.googleapis.com container.googleapis.com cloudresourcemanager.googleapis.com"
    for name in $APIS
    do
      ipv4=$(getent ahostsv4 "$name" | head -n 1 | awk '{ print $1 }')
      grep -q "$name" /etc/hosts || ([ -n "$ipv4" ] && sudo sh -c "echo '$ipv4 $name' >> /etc/hosts")
    done
# Workaround end