How to deploy with ArgoCD when Helm values and Chart are in different repositories

riddle

Published in

MIXI DEVELOPERS

10 min readFeb 21, 2022

Hi, I’m riddle from the SRE Group of mixi Group’s Development Operations.

I have a brief self-introduction here, if you would like to read it. (Sorry, Japanese only…)

TOC

Introduction
Objective
Configuration for our new GitOps workflow
GitOps Workflow with multiple repositories
4.1 Development Phase (in Dev)
4.2 Deployment Phase (in Staging, Production)
Implementation for new workflow
5.1 About Helm Chart in the application repository
5.2 About CI to generate the Helm package
5.3 Setting up ArgoCD
5.4 Synchronizing Manifest in the infrastructure repository
Conclusion
Our tech ebook is available!

Introduction

In our Kubernetes environment, we have two repositories, one for application source code and one for infrastructure source code.

The uses of each repository are:

For Applications: Manage the application code and Kubernetes manifests in the development environment.
For Infrastructure: Manage Terraform and Kubernetes manifests for staging and production environments.

We create the development environment on GKE using the manifest from the app repository. For staging and production, we build in the infrastructure repository using the manifest in almost the same way.

Why do we have multiple repositories? Because the app repository uses Helm to change parameters flexibly, while the infrastructure repository, kustomize is used because the configuration is fixed and easy to use.

However, this workflow led to the problem of “Forgetting to reflect the manifest diffs updated in the development environment to the infrastructure repository”.

Objective

Make it easy to manage manifests in each repository with ArgoCD which we already use in the current GitOps.
Avoid changing the workflow as much as possible and do not increase the cost for developers.
Keep the design as simple and easy to understand as possible to make it less difficult to manage and operate.

Configuration for our new GitOps workflow

The final configuration looks like this:

Manifest management for multiple repositories using ArgoCD

* We are using Google Cloud, which has a CI tool called Cloud Build and a service to store packages called Artifact Registry, but other cloud services can be used instead.

GitOps Workflow with multiple repositories

Development Phase (in Dev)

Submit a Pull Request (PR) to update the Helm Chart.
Merge the PR.
When the PR is merged, CI runs, and Helm Chart is packaged and stored in the Artifact Registry.

* At this time, the package is saved with the tag “1.0.0-Git_Commit_Hash”.
* The development environment will be updated when it is merged into the main branch.

Deployment Phase (in Staging, Production)

Create PR that specifies the version of the Helm package for production.
Merge the PR.
ArgoCD deploys it when the merged PR is detected.

As a result of this change, we now only have to make one modification, whereas before we had to modify each repository’s manifest.

Implementation for new workflow

The specific architecture will be described in the following order:

About Helm Chart in the application repository
About CI to generate the Helm package
Setting up ArgoCD
Synchronizing Manifest in the infrastructure repository

1. About Helm Chart in the application repository

The application repository contains the Helm Chart of the application in the following directory structure.

manifests
├── Charts
│ ├── app1
│ │ ├── .helmignore
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── _helpers.tpl
│ │ │ ├── backendconfig.yaml
…omit…
│ │ │ ├── managedcertificate.yaml
│ │ │ ├── service.yaml
│ │ │ └── serviceaccount.yaml
│ │ └── values.yaml
│ │
│ └── app2
│   ├── .helmignore
│   ├── Chart.yaml
│   ├── templates
│   │ ├── _helpers.tpl
│   │ ├── backendconfig.yaml
…omit…
│   │ ├── managedcertificate.yaml
│   │ ├── service.yaml
│   │ └── serviceaccount.yaml
│   └── values.yaml
│ 
├── app1
│ ├── Chart.yaml
│ └── values.yaml
│
├── app2
│ ├── Chart.yaml
│ └── values.yam

The manifest for each application is managed under the manifests/Charts directory, and the variables for the development environment are set in values.yaml under the manifests/app1 directory.

The Helm Chart to be used in app 1 will be specified by configuring dependencies in manifests/app1/Chart.yaml.

apiVersion: v2
name: app1
type: application
version: 1.0.0dependencies:
- name: app1
  version: 1.0.0
  repository: "file://../Charts/app1/"

The purpose of this configuration is to make Helm Chart independent and easy to use in other environments.

2. About CI to generate the Helm package

From version 3.8.0, Helm supports OCI as an official feature.

https://github.com/opencontainers/artwork#oci-icons

OCI (Open Container Initiative) is an organization that aims to create an open industry standard for container formats and runtimes, and in this context, it refers to the OCI Image Spec and OCI Distribution Spec specified by OCI.

The OCI Image Spec is a standard for container images, and the OCI Distribution Spec is a standard for container registry API protocols.

The following figure shows a container image as defined by the OCI Image Spec.

Source: https://github.com/opencontainers/image-spec/blob/main/spec.md

Originally, Docker was the main force behind container technology. Container technology has been standardized as it grew to be used in various fields.

OCI can be used to store a variety of files in a container registry that was originally only able to store container images.

For example, there is a library called ORAS that is used in Helm, which also provides a CLI, and can convert any file into OCI format and push it to a container registry that supports OCI.

$ echo "hello world" > artifact.txt$ oras push localhost:5000/hello-artifact:v1 --manifest-config /dev/null:application/vnd.acme.rocket.config ./artifact.txt* source: https://oras.land/cli/1_pushing/

In this way, it is possible to store the packaged Helm Chart in an OCI-compliant registry.

Below are the steps to push Helm packages to the Artifact Registry in CI. While not shown below, I run CI in the production environment and push to all the Artifact Registry for development, staging and production.

# Export to create an OCI-compliant Helm Package.
# Declare because we are using Helm 3.7.1.
$ export HELM_EXPERIMENTAL_OCI=1# login artifact registry
$ gcloud auth print-access-token | helm registry login -u oauth2accesstoken --password-stdin https://asia-northeast1-docker.pkg.dev# generate pkg
$ helm --debug package --version [CHART_VERSION] [CHART_PATH]# push pkg
$ helm push [CHART_NAME]_[CHART_VERSION].tgz oci://asia-northeast1-docker.pkg.dev/[PROJECT_ID]/[REGISTRY_NAME]

The version of Helm Chart should be semantic versioning, following the “X.X.X” format.

However, our team’s development speed is fast and the commit messages were not unified, making it difficult to manage semantic versioning strictly. (There are automation tools, but…)

Using the rule that it is acceptable to add strings after the hyphen, as in the example “version: 1.2.3-alpha.1+ef365”, we manage the version of the Helm package as “1.0.0-[Git_Commit_Hash]”.

* Attention: The method of adding strings after the hyphen is not officially permitted, as stated in the official Helm documentation, so please make sure they are supported by the tools you use and the registry you store them in.

Incidentally, AWS and Azure also have registries that support OCI, so you can use them in a similar way to store Helm packages.

Now, you have created the “Helm Chart Change Flow in the development phase”. The next topic is “GitOps workflow with ArgoCD”.

The manifest management of multiple repositories using ArgoCD.

3. Setting up ArgoCD

In this section, I’ll introduce how to get a Helm package from the Artifact Registry and implement GitOps. However, we will not discuss ArgoCD or GitOps themselves, or the App of Apps Pattern, as those are off topic.

In ArgoCD 2.2.3, when specifying a Helm in the Application resource, it is assumed that “the Helm Chart and values.yaml you want to deploy exist in the same repository”.

The following is a manifest from the official website. It is deployed by loading the values.yaml that exists in the same location as the Helm Chart specified in https://bitnami-labs.github.io/sealed-secrets.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  source:
    chart: sealed-secrets
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    targetRevision: 1.16.1
    helm:
      releaseName: sealed-secrets
  destination:
    server: "https://kubernetes.default.svc"
    namespace: kubeseal* source: https://argo-cd.readthedocs.io/en/latest/user-guide/helm/

But what we want to achieve this time is “how to load data from two locations”.

The Helm Chart located in the Artifact Registry.
values.yaml located in the git repository for infrastructure

There are several ways to achieve this.

With sidecar: The method of running an image of sidecar including Helm commands using config-management-plugin-v2(CMP), and generating the manifest by executing Helm commands of sidecar communicating gRPC through socket from argo-repo-server, then deploying it.
With docker image: The method of putting Helm in the docker image of argo-repo-server, using CMP to run Helm commands on argo-repo-server, generating a manifest, then deploying.
With Artifact Registry: The method of registering the Artifact Registry in the ArgoCD repository, then using Helm Dependencies to generate and deploy the manifest.

First, we tried the 1st method, but in version 2.2.3, there was a problem that the gRPC communication using socket between sidecar and argo-repo-server experiences a context timeout in 5 seconds, so we could not properly synchronize ArgoCD with the Git repository. (It seems that this problem has been resolved on the current main branch).

We haven’t tried the second method because we thought it is undesirable to modify the image provided by ArgoCD. The appeal of the 1st and 2nd methods is that it is possible to generate the manifest however we want, but it wasn’t fulfilling our requirements because we were using a general Helm.

We chose the 3rd method, and it was the simplest to achieve what we wanted using ArgoCD’s features.

Since our environment uses External Secret to get credentials from Secret Manager, we use the manifest as below instead of a normal Secret to make the Artifact Registry available to ArgoCD.

* Official Document: How to manage private repositories with ArgoCD

# helm registry
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: asia-northeast1-helm-registry
spec:
  backendType: gcpSecretsManager
  projectId: PROJECT_NAME
  data:
  - key: argocd-helm-login-key # secret manager name
    name: key
    version: latest
  template:
    metadata:
      namespace: argocd
      labels:
        argocd.argoproj.io/secret-type: repository
    stringData:
      enableOCI: "true"
      url: asia-northeast1-docker.pkg.dev
      name: asia-northeast1-helm-registry
      # https://cloud.google.com/artifact-registry/docs/helm/authentication#json-key
      username: _json_key_base64
      password: <%= data.key %>
      type: helm

With this configuration, ArgoCD can now access the registry where the Helm package is stored.

4. Synchronizing Manifest in the infrastructure repository

The next step is the preparation of the manifest for app1.

In the repository for infrastructure, we place the files in the following directory structure. In this article, we will focus on staging.

manifests/app1
├── production
│  ├── Chart.yaml
│  └── values.yaml
│
└── staging
   ├── Chart.yaml
   └── values.yaml

values.yaml only contains variables dedicated to staging, so I won’t explain it here. The important part is Chart.yaml.

In Chart.yaml, the version of the Helm package to be used is listed as dependencies. By this, you can get the necessary packages just by performing a “helm dependency update”.

apiVersion: v2
name: app1
type: application
version: 1.0.0dependencies:
  - name: app1
    version: "1.0.0-e5c1704"
    repository: "oci://asia-northeast1-docker.pkg.dev/[PROJECT_ID]/[REGISTRY_NAME]"

In the case of ArgoCD, the dependent charts from the private registry that you have just registered with this code will be downloaded. This allows us to generate a manifest from the data in two separate locations.

Now all we have to do is define the Application to give to ArgoCD.

This is just a setting to have ArgoCD watch the Git repository where values.yaml is stored. Note that you do not register the Artifact Registry that contains the Helm Chart.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app1
  namespace: argocd
  annotations:
    argocd.argoproj.io/manifest-generate-paths: ..
spec:
  project: defaultsource:
    repoURL: git@github.com:xxx/infra.git
    path: manifests/app1/staging
    targetRevision: main
    helm:
      valueFiles:
        - values.yamldestination:
    server: https://kubernetes.default.svc
    namespace: app1

A summary of the result of this will be:

Helm Chart is specified in Chart.yaml from Artifact Registry
values.yaml is specified as an application from the Git repository for infrastructure

Now we’ve created “GitOps flow using ArgoCD” !

Conclusion

The OCI scheme allows us to store Helm in the registry
Using ORAS, we can easily add anything you want
If you have multiple repositories, Helm + ArgoCD configuration is very flexible

Hopefully this is helpful!

Our tech ebook is available!

We have published “mixi tech note #07” for free at Tech Book Fest 12!
You can download it now! (Japanese only)

I wrote the 3rd chapter, titled “Let’s get the most out of Cloud Build!”.
Please feel free to download it HERE!

Contents List

Chapter 1: Building Web Apps with Bazel using Elm + Haskell
Chapter 2: Microservices and Consistency in the Gacha System
Chapter 3: Using Cloud Build to its Fullest!
Chapter 4: Cloud Spanner Tips I Learned in TIPSTAR Development
Chapter 5: My Experience as an Engineering Manager for 6 Months
Chapter 6: Feature Toggle Management Using Unleash

Official website: https://techbookfest.org/organization/54670001