Terraform Synced-State Validation

Working with Terraform creates a number of exciting challenges.

In this blog we are going to focus on two of them:

Keeping Terraform’s state consistent with the actual provider (i.e. a cloud provider) where resources can be quite a challenge. Such inconsistencies are usually a result of changes performed directly in the cloud environment.
Keeping the Terraform code synced with the Terraform state. When we want to apply changes in Terraform, we can:

Start by pushing the changes to the Terraform repository and then apply them. The Terraform state will be behind until the changes are affected.
Start by applying the changes locally, then push them. The Terraform state will be ahead until the code is pushed.

The first option can be mostly avoided if we don’t allow manual actions that could have been performed by Terraform, but not entirely.

The real issue with the second option is that after the first action (apply/push), Terraform will be out of sync. If the second action is delayed or not happening, the undesired inconsistent state will be kept.

To ensure eventual consistency here, you can automate the workflow with an automation framework, such as Jenkins.

However, the Fyber DevOps team, wanted to avoid such an automation process because we thought it would become a delaying factor, and we still manage to keep Terraform state synced with the code and the actual resources.

We do this with a little help from a Jenkins pipeline library script. The script initials, updates and executes a ‘Terraform Plan’ command. If Terraform isn’t synced, it reports the unsynced components to the dedicated Slack channel. The Jenkins job is scheduled to run in the first and penultimate working hour, on each of the Terraform environments. This enables us to be able to fix any inconsistency by the end of the day.

	def call(jenkinsKeyIdForRepo, repoUrl, terraformVersion, terraformBucket, bootstrapSshKey, jenkinsAwsKey, awsRegion, slackChannelName) {

	step([$class: 'WsCleanup', deleteDirs: true, notFailBuild: true])

	manager.addShortText(env.environment)

	stage('Checkout') {
	checkout changelog: false, poll: false, scm: [$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanBeforeCheckout']], submoduleCfg: [], userRemoteConfigs: [[credentialsId: jenkinsKeyIdForRepo, url: repoUrl]]]
	}

	def tfHome = tool name: terraformVersion, type: 'com.cloudbees.jenkins.plugins.customtools.CustomTool'
	env.PATH = "${tfHome}:${env.PATH}"

	// Get S3 credentials
	withEnv(["AWS_DEFAULT_REGION=${awsRegion}"]) {

	wrap([$class: 'AnsiColorBuildWrapper', colorMapName: 'xterm']) {

	try {
	withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: jenkinsAwsKey, secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
	withCredentials(bindings: [sshUserPrivateKey(credentialsId: "$bootstrapSshKey",
	keyFileVariable: "bootstrapSshKey")]) {
	stage ('Create tvars File') {

	writeFile file: 'terraform.tfvars', text: """terraform_bucket = $terraformBucket
	site_module_state_path = \"site/terraform.tfstate\"
	bootstrap_ssh_key_path = \"${bootstrapSshKey}\"
	"""
	}
	// Mark the code build 'plan'....
	stage ('Plan') {
	// Output Terraform version
	sh "terraform --version"
	//Remove the terraform state file so we always start from a clean state
	if (fileExists(".terraform/terraform.tfstate")) {
	sh "rm -rf .terraform/terraform.tfstate"
	}
	if (fileExists("status")) {
	sh "rm status"
	}

	sh "terraform get --update; terraform init"

	def exitCode = sh script: "terraform plan -detailed-exitcode > plan.out; echo \$?", returnStatus: true

	planFile = readFile('plan.out')
	echo planFile
	}
	stage('handle result') {
	if (exitCode == 0) {
	currentBuild.result = 'SUCCESS'
	} else if (exitCode == 1) {
	currentBuild.result = 'UNSTABLE'
	slackSend channel: slackChannelName, color: 'warning', message: "@everyone :shock: Terraform ${env.environment} plan returned an error. (<$BUILD_URL/console\|Job>)"

	} else if (exitCode == 2) {
	currentBuild.result = 'UNSTABLE'

	changes = sh(
	script: "grep module plan.out",
	returnStdout: true
	).trim().toString()

	slackSend channel: slackChannelName, color: 'warning', message: "@everyone :facepalm-skype: There are changes to apply in Terraform ${env.environment} \". \nPlan changes are: \n $changes \n (<$BUILD_URL/console\|Job>)"
	}
	}
	}
	}
	} catch (any) {
	slackSend channel: slackChannelName, color: 'danger', message: "@everyone :angry-skype: Terraform ${env.environment} Validation has been broken!!\n (<$BUILD_URL/console\|Job>)"
	throw any
	}
	}
	}
	}

view raw terrformPlanValidation.groovy hosted with ❤ by GitHub

It’s important to note that the script is useful when you’re working with the Terraform Recommended Workflow, specifically when having an environment file for each environment that defines the Terraform modules, while the resources lie in their own repository.

Dani Shemesh

Newsletter Sign-Up

Get our mobile expertise straight to your inbox.

Terraform Synced-State Validation

Newsletter Sign-Up

Explore More