Plannable Import: Don't fail, but create resource, if not exist

[DJAlPee]

Terraform Version

Terraform v1.5.2

Use Cases

In our CI/CD system, I want to import resources, that eventually already had been created. If they do not exist, terraform could safely create them using the existing configuration.

My concrete example:
I have some (AWS) Lambda functions, that had been created by terraform. When being executed for the first time, the functions will create a LogGroup with the function name in CloudWatch. Unfortunately, the default config for these LogGroups doesn't fit our needs (e.g. no log retention being set). When I add the LogGroup to the terraform configuration, applying will fail in most (but not all!) cases, because it tries to create the LogGroup with the existing name.

Attempted Solutions

In similar situations, we added some commands before doing the "apply" and imported the resources using the CLI import command or just deleted the resource. The new import block would be a game changer for us...

Thanks to CDKTF, as a workaround we can make the "import" block optional and check the existence of the resource using AWS API.

Proposal

I see two possible ways to tackle this:

1. Flag in the CLI, which allows to ignore "Cannot import non-existent remote object" errors
2. Optional property in the "import" block, which tells terraform how to proceed, when resource does not exist.

Option 2 feels best for me, because the behavior can be configured individually for each resource/import. The config could look like:

import {
   id              = "/aws/lambda/function-name"
   to              = aws_cloudwatch_log_group.lambda_log_group
   fail_on_missing = false # optional, default: "true"
}

More "positiv" sounding proposal by @acdha:

import {
   id                  = "/aws/lambda/function-name"
   to                  = aws_cloudwatch_log_group.lambda_log_group
   create_when_missing = true # optional, default: "false"
}

References

No response

[or-shachar]

This is definitely the more common use case for us. We reuse a lot of our terraform code.

• In a new instance mode we would like the resources to be created
• In recovery mode we probably need to import the existing resources into the state

Having something like fail_on_missing is great for us.

Also possibly related to #33624 where we want to possibly switch off the import blocks.

[antoinedeschenes]

This would definitely help by allowing to predefine imports in modules we provide to our devs. We're mostly migrating existing infrastructure, and handling terraform imports outside the CI pipeline is a time waster right now.

[antoinedeschenes]

I thought I'd be able to workaround this by using terraform import to determine which import blocks I could generate, but there's no dry-run feature available, per this closed issue: #25713

[antoinedeschenes]

This patch 350f9b8 works, though it assumes any import error should be ignored:

diff --git a/internal/terraform/node_resource_plan_instance.go b/internal/terraform/node_resource_plan_instance.go
index a4bbaa5359..561faa7371 100644
--- a/internal/terraform/node_resource_plan_instance.go
+++ b/internal/terraform/node_resource_plan_instance.go
@@ -165,7 +165,11 @@ func (n *NodePlannableResourceInstance) managedResourceExecute(ctx EvalContext)
        // If the resource is to be imported, we now ask the provider for an Import
        // and a Refresh, and save the resulting state to instanceRefreshState.
        if importing {
-               instanceRefreshState, diags = n.importState(ctx, addr, importId, provider, providerSchema)
+               var importDiags tfdiags.Diagnostics
+               instanceRefreshState, importDiags = n.importState(ctx, addr, importId, provider, providerSchema)
+               if importDiags.HasErrors() {
+                       importing = false
+               }
        } else {
                var readDiags tfdiags.Diagnostics
                instanceRefreshState, readDiags = n.readResourceInstanceState(ctx, addr)

Looks like ignoring Cannot import non-existent remote object errors might not be sufficient, as some providers just error out on some resources when attempting an import (ie. GCP project IAM bindings):

│ Error: Cannot find binding for "project \"gcp-project\"" with role "roles/...", member "serviceAccount:...@gcp-project.iam.gserviceaccount.com", and condition title ""

[acdha]

We have a very similar use-case: our AWS account configuration is managed using Terraform. import blocks provide a good way to handle our existing accounts but that won't work for a new account unless you run the configuration as of an old config version first and then let it upgrade, or do something kludgy like deleting the imports first. I think that fail_on_missing proposal would be great but I'm wondering whether the name should be something more along the lines of create_when_missing to focus on the desired outcome.

[garkenxian]

We have a very similar use-case: our AWS account configuration is managed using Terraform. import blocks provide a good way to handle our existing accounts but that won't work for a new account unless you run the configuration as of an old config version first and then let it upgrade, or do something kludgy like deleting the imports first. I think that fail_on_missing proposal would be great but I'm wondering whether the name should be something more along the lines of create_when_missing to focus on the desired outcome.

I disagree the create_when_missing is needed. Remember that you're attaching the import block to a resource. the import block needs to just be ignored if the fail_on_missing flag is false, allowing the terraform script to gracefully continue and plan to Create a new resource instead of failing the entire script.

[AMEvers]

Something like this would be hugely helpful. We are forced to use local backends for our terraform and have had cases where our state file was lost. Being able to have something equivalent to "import if exists else create" would be amazing. Given @antoinedeschenes comment about only catching missing resources error being insufficient, I would suggest something along the lines of allow_failure or ignore_error. Either that or ignore the fact that not all providers return proper errors and encourage them to align with standards.

[DJAlPee]

I disagree the create_when_missing is needed. Remember that you're attaching the import block to a resource. the import block needs to just be ignored if the fail_on_missing flag is false, allowing the terraform script to gracefully continue and plan to Create a new resource instead of failing the entire script.

The property name doesn't have to decribe the behavior in the code.
Something like create_when_missing feels more "natural" regarding the "Developer Experience". In my opinion (!), it might make sense, to have this as default behavior after some time... But that would be a breaking change 😄

[...] I would suggest something along the lines of allow_failure or ignore_error. Either that or ignore the fact that not all providers return proper errors and encourage them to align with standards.

This sounds too generic... What about errors e.g. regarding permissions?

Would be great, if there is already something in place, that is able to search the given ID using the providers implementation. Catching an import error (and deciding to throw it or not) feels not right to me and could lead into unexpected behaviors for some providers.
The workflow should look like:

• Resource is already in state -> Import directive ignored (current behavior)
• Resource is NOT in state
  • create_when_missing = true -> Search, if given ID exist (new behavior)
    • If YES: Import
    • If NO: Create new resource
  • create_when_missing = false -> Import (current behavior)

This would cause one additional API call during the import, but only once!

[kevinkuszyk]

@jbardin are you able to share a timeframe when this might be included?

And in the meantime, are there any recommended patterns for workarounds?

I'm also hitting the issue with cloud watch log groups that are automatically created by other AWS services. I need to:

1. Reference them in the same terraform apply that creates a related lambda function for example. In this case terraform throws an error because the log group hasn't been created yet
2. Update existing log groups that have already been created by another AWS service

[jbardin]

@kevinkuszyk, No there is no timeline yet, but it does not sound like it would fit your use case. The import action and planning decisions all need to happen during the plan, so if there are log groups created implicitly by other resources, you're still going to end up creating conflicting log groups during apply. What you're describing is a much more complex feature which requires support from providers and a new protocol to handle. It's something we're aware of, but not directly related to import.