Terraform your CICD Toolchain : GitHub

When you’re running and managing the CICD Toolchain of a big organization, you may want to establish some conventions and usage rules to avoid having to manage a big inconsistent mess. One possibility to avoid this, after you had established your various naming conventions and usages, is to use Terraform to maintain all of the objects constitutes your application projects.

For example, if your application’s sources are hosted on GitHub, using Actions for the build and SonarQube for the code quality analysis, you have to maintain both services : the repositories, the accesses, and the SonarQube project. And if along with this you have Argo-CD for deployment, or AWX for Ansible orchestration, etc, it’s getting more and more time costly for resources maintenance.

In this article, we will use a GitHub organization as an example with a very simple and basic Terraform procedure just for demonstration. Just a little warning : I’m more used to Ansible than Terraform, so please don’t throw me your knives, I’ve just wanted to try something else.

Terraform is not just a tool providing an Infrastructure as Code service that help you to deploy your Cloud resources on your favorite provider. Or Terraforming your ProxMox (article in French). There is tons of Providers in the registry that are able to target various systems and services, even not related to infrastructure. In our case, we will use the GitHub provider which is a wrapper of GitHub’s API.

Why using Terraform for this ? Well, that’s one choice among other ones. There are also some Ansible collections on Galaxy that address this question, as well as you can develop yourself your management tool… All of these are based on the official REST/GraphQL API after all.

Init the idea

I’ve made a quick example here on GitHub. Here is the repository content :

• main.tf : the main terraform file
• provider.tf : the settings for the GitHub provider
• vars.tf : some global variables for the project

In my example, I’ve created an Organization on GitHub named the Poochi Corp. In the Poochi Corp, the GitHub users cannot create themselves their repositories to ensure the compliance to our policies. In a more advanced idea, the self-service remains available for the users, but through the IT service management tool which use API to call the workflow.

Setup the provider

In the provider.tf file, put the provider’s definition :

terraform {
    required_providers {
        github = {
        source  = "integrations/github"
        version = "~> 5.0"
        }
    }
}

In the vars.tf file, I’ve set some global variables :

## The GITHUB_TOKEN PAT that will execute the actions
## It's an environment variable named TF_VARS_GITHUB_TOKEN that provide it
variable "GITHUB_TOKEN" {
    type = string
}

## The target organization
variable "organization" {
    default = "poochi-corp"
}

## The default repository template
variable "repository_template" {
    default = "std-template-repository"
}

## The teams roles to create for each repository
variable "teams_roles" {
    type = set(string)
    default = ["push", "admin"]
}

## The repositories list
variable "repositories" {
    type=set(string)
    default = [
        "dept-1-app-1",
        "dept-2-app-1",
        "dept-3-app-1",
        "dept-4-app-1",
    ]
}

As you may understood by reading this file, the repositories list will be a variable using the naming convention. Every objects will use this base.

Then, in main.tf I’ve set the provider and the main code execution. It’s very basic : we create the repositories, the teams, and assign them.

# Configure the GitHub Provider
provider "github" {
    token = var.GITHUB_TOKEN
    owner = var.organization
}

## create repositories declared in vars

## Create the repository

resource "github_repository" "repository" {
    for_each = var.repositories
    name = each.value
    visibility = "private"
    vulnerability_alerts = true

    template {
        owner = var.organization
        repository = var.repository_template
    }
}

## Netsted loop for repositories, teams and roles

locals {
    repositories = var.repositories
    teams_roles = var.teams_roles
    repository_team = flatten([
        for repository in local.repositories : [
            for role in local.teams_roles : {
                repository = repository
                role = role
            }
        ]
    ])
}

## create its teams

resource "github_team" "teams" {
    for_each = { for entry in local.repository_team: "${entry.repository}-team-${entry.role}" => entry }
    name = "${each.value.repository}-team-${each.value.role}"
    # create_default_maintainer = true
}

## Assign teams

resource "github_team_repository" "dept-1-app-1-team-repo" {
    for_each = { for entry in local.repository_team: "${entry.repository}-team-${entry.role}" => entry }
    team_id = "${each.value.repository}-${each.value.role}"
    permission = each.value.role
    repository = each.value.repository
}

Test the idea

First, we need a GitHub Personal Access Token able to administrate the Organization and accessing repositories. I used the new beta fine-grained PATv2, far better than the previous model. I’ve choose the following settings :

To use it, I’ve set an environment variable named TF_VARS_GITHUB_TOKEN, because Terraform can use any environment vars starting by TF_VARS. That’s why I have a variable named GITHUB_TOKEN in the definition.

First, we initialize the project.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding integrations/github versions matching "~> 5.0"...
- Installing integrations/github v5.12.0...
- Installed integrations/github v5.12.0 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)

(....)

Then, we use the plan command to calculate the actions.

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

(tons of objects and settings)

Plan: 14 to add, 0 to change, 0 to destroy.

All good, let’s apply !

$ terraform apply

(.....)

Plan: 14 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

github_team_repository.dept-2-app-1-team-repo["push"]: Creating...
github_team_repository.dept-2-app-1-team-repo["admin"]: Creating...
github_team.dept-1-app-1-team["admin"]: Creating...
github_team.dept-1-app-1-team["push"]: Creating...
github_team.dept-2-app-1-team["admin"]: Creating...
github_team.dept-2-app-1-team["push"]: Creating...
github_repository.dept-1-app-1: Creating...
github_repository.dept-2-app-1: Creating...

As I’ve wrote above, I had some errors at the end because the GraphQL API is not working with PATv2. However, we still have the idea, let’s check what it made.

We have the repositories !

And the teams ! I’ve disabled the default maintainer but it looks like it’s one of the settings the template could not apply because of my PAT v2.

And as expected, the accesses don’t work.

Anyway, we have the idea. As you may see, all objects are private. This is a choice I’ve made to avoid people seeing too many things and concentrate on their own. In real life, we didn’t do this because it’s better to let developers from various teams to inspire each others. We applied a policy saying the repository visibility must match to the application’s confidentiality level. “Internal” (every member of the Organization can see the repository content) for non critical, and private for business confidential.

This procedure is quick and dirty, the purpose of this article was to introduce the idea because this one has it flaws. The biggest issue this procedure has is if you remove a repository from the list, it’ll be destroyed by Terraform.

In a next entry, we will enhance and expand it by integrating SonarQube in the chain. The goal will be to create the GitHub repository and the SonarQube project in the same round.