The Terraform helm_template data source

I have kind of a love/hate relationship with helm. For a vendor, providing templated yaml that interpolates variables has got to be the best way to provide kubernetes manifests. However, as an enterprise type infra person, they require a lot of work. If the provider doesn't make hooks for everything we wind up having to either fork the chart or do stuff with kustomize. It can get pretty ugly either way, and the kinds of modifications we need are often not the ones open source projects think about (how will this app work without internet access being the most common). Its still better than the other solutions though.

I still prefer to have all of my infra managed gitops style though, and do not much like the helm cli outside of `helm template`. This has kept me from using Terraform to manage helm charts because the `helm_release` resource is a wrapper around the CLI. You can't really get a view into diffs/changes. They offer a `helm_template` data source, but I hit problems when I tried to use it and just defaulted back to letting argo manage them.

Today I spent some time working on it. I learned a few things that may be helpful-

data "local_file" "argo-values" {
  filename = "${path.module}/argo_values.yaml"
}

resource "kubernetes_namespace" "argo" {
    metadata {
        name = "argo"
    }
}

data "helm_template" "argo" {
    name       = "argocd"
    repository = "https://argoproj.github.io/argo-helm"
    chart      = "argo-cd"
    version    = "5.36.0"
    namespace  = kubernetes_namespace.argo.metadata[0].name

    kube_version = data.aws_eks_cluster.this.version
    include_crds = true
    values = [data.local_file.argo-values.content]
}

This was my basic first attempt. I provide a values.yaml file as a data source, setup my k8s namespace, and now have the template resource. However, my first attempt at applying things hit a snag-

resource "kubernetes_manifest" "argo" {
    for_each = data.helm_template.argo.manifests
    manifest = yamldecode(each.value)
}
│ Error: Invalid for_each argument
│ 
│   on argo.tf line 24, in resource "kubernetes_manifest" "argo":
│   24:     for_each = data.helm_template.argo.manifests
│     ├────────────────
│     │ data.helm_template.argo.manifests is a map of string, known only after apply
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.

This is the error I never really looked into before (just swapping to helm_release and argo). Playing around, I discovered that if I did a partial apply (with the kubernetes_manifest block removed), it would apply, and I could then put the manifest back in and get a valid plan. This made the problem clear- because the `data` block is depending on another resource, it couldn't be rendered until that resource existed!

This leaves us with two straightforward paths (assuming that "building up our module with multiple apply commands" isn't the preferred approach).

First, we can move dependencies.

data "helm_template" "argo" {
    name       = "argocd"
    repository = "https://argoproj.github.io/argo-helm"
    chart      = "argo-cd"
    version    = "5.36.0"
    namespace  = var.namespace

    kube_version = data.aws_eks_cluster.this.version
    include_crds = true
    values = [data.local_file.argo-values.content]
}

resource "kubernetes_manifest" "argo" {
    for_each = data.helm_template.argo.manifests
    manifest = yamldecode(each.value)
    depends_on = [kubernetes_namespace.argo]
}

If your data source depends on the string instead of the resource, we can now plan correctly. Adding a depends_on to the manifest block will still keep the dag ordered directly. This gets us a nice simple one step deploy. One tricky problem is passing values in. Imagine we wanted to create an SQS queue and pass the ARN into the chart...only option now is to have the chart depend on the resource, since the arn can't be derived ahead of time.

If the chart is set up properly though, we can work around it. The solution will be something like this-

resource "kubernetes_config_map" "this" {
    metadata {
        name = local.config
    }
    data = {
        sqs_arn = aws_sqs_queue.this.arn
    }
}

data "helm_template" "this" {
    ...
    
    set {
        name = "custom_config"
        value = local.config
    }
}

Basically, instead of letting the chart build the configmap, create it yourself as a resource and pass it in. This is especially useful for secrets! Do not let helm generate random passwords- have terraform do it! If the chart does not have a hook for this though, we're down to the final technique- patching.

resource "kubernetes_manifest" "argo" {
    for_each = {for k, v in data.helm_template.argo.manifests: k => v if k != "templates/argocd-repo-server/clusterrole.yaml"}
    manifest = yamldecode(each.value)
    depends_on = [kubernetes_namespace.argo]
}

resource "kubernetes_manifest" "cluster_role" {
    manifest = merge(
        yamldecode(data.helm_template.argo.manifests["templates/argocd-repo-server/clusterrole.yaml"]),
        {metadata = merge(yamldecode(data.helm_template.argo.manifests["templates/argocd-repo-server/clusterrole.yaml"])["metadata"], {name = "changed"})}
    )
}

Basically, we pull individual manifests out and use the merge function to overwrite specific fields. This can be tricky (notice that we have merge in there twice- we don't want to completely replace "metadata", just change that one field!) but gives us the functionality of a kustomize patch on top of helm template.

There is one more problem. If the chart depends on CRDs that are created by the chart, you will not be able to do a one step apply at all. The only option is to do something like this

resource "kubernetes_manifest" "argo_crd" {
    for_each = toset(data.helm_template.argo.crds)
    manifest = yamldecode(each.value)
    depends_on = [kubernetes_namespace.argo]
}

resource "kubernetes_manifest" "argo" {
    for_each = data.helm_template.argo.manifests
    manifest = yamldecode(each.value)
    depends_on = [kubernetes_namespace.argo, kubernetes_manifest.argo_crd]
}

and apply with `tf apply -target kubernetes_manifest.argo_crd` first (or split the CRD specifically into a separate terraform setup- these will never depend on config values, so can be anywhere really)