Part:2 Terraform & Infrastructure as Code

Mastering Terraform: A Beginner’s Guide to Managing Cloud Infrastructure

Terraform has emerged as one of the most popular tools for managing cloud infrastructure. It allows DevOps engineers and cloud enthusiasts to automate the provisioning of resources in a consistent and repeatable way. In this blog, we’ll break down what Terraform is, the concepts behind it, and how to get started with it. So, whether you're deploying to AWS, Azure, or any other cloud platform, you’ll be ready to get started with Terraform!

• Terraform Use-Cases/

• Terraform Life Cycle

• Write your First Terraform Project

• Terraform State File

• Terraform Best practices

• Terraform Modules

• Problems with Terraform

• Terraform Interview Questions

What is a Target API in Terraform?

At its core, Terraform interacts with cloud providers through their APIs. When you write a Terraform configuration, you're specifying the resources you want to create or manage (like EC2 instances in AWS or virtual machines in Azure). Terraform translates your configurations into API calls that the cloud provider understands. For example, when using AWS as a provider, Terraform will convert your configuration into AWS API requests to provision resources like EC2 instances or S3 buckets.

Benefits of Terraform

1. 🔹 Manage Any Infrastructure

   Terraform is cloud-agnostic, meaning you can manage any infrastructure — whether on AWS, Azure, Google Cloud, or even on-premise environments — using the same configuration language. Once you configure the provider, Terraform converts your configuration files into the specific APIs of the target provider. This makes switching between cloud platforms simple and future-proofs your infrastructure for hybrid or evolving cloud environments.

   never store the state file on GitHub.

   🔹 Track Your Infrastructure

   Terraform uses a state file to keep track of the infrastructure it manages. This state file can be stored remotely (e.g., in S3, Terraform Cloud, or other backends), allowing you to audit, reference, and understand what infrastructure has been created — all without logging into the cloud console. This gives you full visibility into your deployed resources.

   

   🔹 Automate Changes

   With Terraform, infrastructure changes are automated. Instead of manually logging into cloud consoles to update resources, you simply modify the configuration files and Terraform automatically determines the necessary changes. It calculates a plan, shows you what will change, and applies updates reliably — eliminating human errors and making infrastructure updates consistent and repeatable.

   🔹 Standardize Configuration

   Terraform enforces consistency by using a common templating language (.tf files) across all cloud providers. Unlike manual provisioning, where different tools or approaches may be used for different clouds, Terraform provides a standard way to define infrastructure, ensuring predictable and repeatable deployments.

   🔹 Collaborate

   Terraform's code-based approach means your infrastructure is stored in version control systems like Git. Teams can collaborate by reviewing pull requests, tracking changes, and managing infrastructure just like application code. This not only improves teamwork but also enables peer reviews, audits, and a controlled deployment process.

   The Terraform Lifecycle

1. Writing Configuration Files
   The first step is writing the Terraform configuration files (typically .tf files). These files define the infrastructure resources you want to create or manage.

2. Core Terraform Commands

   • terraform init: Initializes your working directory with the necessary provider configurations.

   • terraform plan: Shows the changes Terraform will make to your infrastructure based on your configuration files.

   • terraform apply: Applies the changes, creating or modifying resources in your cloud provider.

   • terraform destroy: Deletes the resources that Terraform has created.

3. Dry Run (terraform plan)
   Terraform allows you to preview changes before applying them by using the terraform plan command. This helps ensure that you're making the right changes.

4. State Management
   Terraform maintains a state file (terraform.tfstate) that tracks your infrastructure’s current state. This file can be stored remotely (e.g., in an S3 bucket) for collaboration and backup.

Writing Terraform Files

Note: If you're new to writing your first Terraform file or installing Terraform, you can use the official HashiCorp Terraform documentation. It provides excellent examples with clear explanations. If you want to write Terraform files for any resource (like Lambda or S3), search the Terraform documentation for any cloud provider.

Terraform installation for AWS: Install Terraform

Terraform documentation: Terraform Registry

1. Provider Block
   The first block in your configuration files is the provider block. It specifies which cloud provider you are using (e.g., AWS, Azure, etc.) and its version. For example, to use AWS, you would define a provider block like this:

    provider "aws" {
      region = "us-west-2"
    }
   

2. Resource Block
   The resource block defines the specific cloud resources you want to create. Here’s an example of creating an EC2 instance on AWS:

    resource "aws_instance" "example" {
      ami           = "ami-123456"
      instance_type = "t2.micro"
    }
   

3. Variables and Outputs
   Use variables.tf to define input values and output.tf to define outputs like IP addresses or instance IDs. This keeps your configuration flexible and reusable.

Best Practices

1. Use Input and Output Files
   Instead of hardcoding values directly into resource blocks, use variables.tf for input values and output.tf for outputs. This makes your configuration more reusable and modular.

2. Version Terraform
   Always specify the required Terraform version in your configuration to ensure consistency across environments.

    terraform {
      required_version = ">= 1.0"
    }
   

Key Terraform Commands

Here’s a quick overview of the core Terraform commands you'll use frequently:

1. terraform init: Initializes the working directory and downloads required providers.

2. terraform plan: Shows the execution plan for the changes Terraform will make.

3. terraform apply: Applies the changes to your infrastructure.

4. terraform destroy: Removes the infrastructure resources.

Providers in Terraform: Single vs. Multiple Providers

In a typical Terraform configuration, you can either use a single provider (like AWS) or multiple providers (for a hybrid cloud environment).

For example, if you use both AWS and Azure in your setup, you would define both providers like this:

provider "aws" {
  region = "us-west-2"
}

provider "azurerm" {
  features {}
}

When you add or change a provider (like switching from AWS to Azure), you may need to reinitialize Terraform by running terraform init again.

Authentication with Cloud Providers

When you configure Terraform, you need to authenticate it to your cloud provider.

• AWS: You need to have AWS CLI configured on your machine with access keys. If you haven’t configured the AWS CLI, you can do so by running aws configure and providing your AWS credentials.

• Azure: You must configure an Azure Service Principal and authenticate Terraform with that.

Once authentication is set up, you can run basic CLI commands like aws s3 ls or aws —version to verify the setup.

Deploying Your First EC2 Instance with Terraform

Let’s walk through the process of creating an EC2 instance in AWS using Terraform.

1. Write Your Terraform Configuration
   Create a main.tf file with the necessary provider and resource definitions:

   

    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4.16"
        }
      }
   
      required_version = ">= 1.2.0"
    }
   
    provider "aws" {
      region  = "us-west-1"
    }
    resource "aws_instance" "tf_demo_server" {
      ami                     = "ami-04f7a54071e74f488"
      instance_type           = "t2.micro"
      tags = {
        Name = "Terraform_Demo"
      }
    }
   

2. Initialize Terraform
   Run terraform init to initialize your working directory and download the required provider.

   

3. Preview Changes
   Run terraform plan to see what resources Terraform will create.

   

4. Apply Changes
   Run terraform apply to create the EC2 instance. Terraform will prompt you to confirm with “yes.”

    terraform apply
   

   

   After confirmation, Terraform will provision the EC2 instance, and you’ll see the instance ID and other details in the output.

   

5. Verify Resource Creation
   Once the instance is created, you can log in to the AWS Management Console and verify that your EC2 instance has been created in the specified region.

   

6. Removes the infrastructure resources: terraform destroy

   

   

   ✅ What gets destroyed?

   • Only resources that were created through Terraform and are tracked in its state file (terraform.tfstate).

   • It reads the current state and sends deletion instructions to the cloud provider (like AWS) for only those items.

So for example, if you created:

• An EC2 instance

• A security group

• A VPC
  Using Terraform, then terraform destroy will delete those.

  

Key Learnings So Far

At this stage, you’ve executed your Terraform scripts and applied your infrastructure changes. Now, let’s take a deeper dive into what we’ve learned and best practices when working with Terraform:

1. Outputs in Terraform:

   • After running terraform apply, you got only the EC2 instance ID as output. However, a single ID is not enough for a user who doesn't have access to the AWS Console.

   • To give more useful information (such as private IP, public IP, instance ID, and key-value pairs), you need to use an output.tf file.

   • Best Practice: Always provide detailed outputs using outputs.tf to share relevant resource information with users.

Example:

    output "private_ip" {
      value = aws_instance.my_instance.private_ip
    }

    output "public_ip" {
      value = aws_instance.my_instance.public_ip
    }

2. Managing Variables:

   • Avoid hardcoding values in main.tf. Instead, use a variables.tf file to define variables such as AMI IDs, instance types, and other configurations.

   • Best Practice: Organize configurations using variables to improve reusability and manageability.

Example:

    variable "ami_id" {
      description = "The AMI ID for the EC2 instance"
      default     = "ami-12345678"
    }

    resource "aws_instance" "example" {
      ami           = var.ami_id
      instance_type = var.instance_type
    }

3. Terraform State Files:

   • The terraform.tfstate file tracks the resources that Terraform manages. Without this file, Terraform would have no knowledge of the resources it created.

   • Important: This state file should never be stored locally or in a Git repository because it contains sensitive information like resource IDs and other metadata.

Key Takeaways:

• The state file is your source of truth for infrastructure.

• Never upload the state file to Git.

• Always use remote backends (e.g., AWS S3, Azure Blob Storage, or GCP’s GCS) to store state files in a secure and shared location.

---

Working with Remote Backends

Why Remote Backends are Crucial

Terraform tracks and manages infrastructure changes using the state file. This file must be stored in a centralized location where multiple users can access and update it securely. Using a remote backend ensures that the state file is managed in a secure, consistent, and collaborative manner.

1. Remote Backend Options:

   • S3 (AWS), Azure Blob Storage, and GCS (Google Cloud) are the most common remote backends.

   • The state file must be stored in a secure and shared location. For example:

       terraform {
         backend "s3" {
           bucket = "my-terraform-state-bucket"
           key    = "path/to/my/terraform.tfstate"
           region = "us-east-1"
         }
       }
     

2. Why Remote Backends Over Local:

   • Collaboration: Multiple users can access the same state file, making it easier for teams to work together.

   • Security: Sensitive data in the state file is not exposed or misplaced in local files or Git.

   • Consistency: Centralized state ensures the infrastructure is consistently tracked and managed across different environments.

State Locking with DynamoDB

State locking is a crucial practice to prevent race conditions when multiple users attempt to apply changes to the same infrastructure simultaneously.

1. How State Locking Works:

   • In AWS, we can use DynamoDB to lock the state file.

   • When Terraform starts applying changes, DynamoDB puts a lock on the state file.

   • If another user tries to apply changes, they will get a message saying the state is locked by another user.

Example Configuration for State Locking:

    terraform {
      backend "s3" {
        bucket         = "my-terraform-state-bucket"
        key            = "terraform.tfstate"
        region         = "us-east-1"
        dynamodb_table = "my-terraform-locks"
      }
    }

Benefits of Remote State and Locking:

• Collaboration: Multiple users or teams can run Terraform simultaneously without stepping on each other’s toes.

• Security: Avoids exposing the state file locally, ensuring sensitive data is secure.

• Consistency: Every team member has the same view of the infrastructure.

---

Best Practices for Terraform

1. State Isolation:

   • If your organization uses multiple environments (e.g., dev, staging, prod), each should have its own Terraform configuration and separate state file.

   • Best Practice: Use different state files for different environments to avoid risks like accidental changes in production.

Example S3 bucket organization:

    s3://my-terraform-bucket/dev/terraform.tfstate
    s3://my-terraform-bucket/staging/terraform.tfstate
    s3://my-terraform-bucket/prod/terraform.tfstate

2. Terraform in CI/CD Pipelines:

   • Use CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps to automate the execution of Terraform scripts.

   • The CI/CD pipeline can automatically pull the latest Terraform configuration from Git and apply it to the cloud infrastructure.

   • Best Practice: Set up automated pipelines to manage infrastructure changes, avoiding manual intervention and reducing human errors.

3. No Manual Changes in Cloud Console:

   • Any manual change made directly in the cloud provider’s console (e.g., changing an EC2 instance’s configuration) will not be tracked by Terraform unless explicitly refreshed with terraform refresh.

   • Best Practice: Always make infrastructure changes via Terraform to maintain consistency between the code and the infrastructure.

---

Limitations of Terraform

While Terraform is a powerful tool, it has some limitations that you should be aware of, especially when working in large-scale, production environments.

1. No Rollback Mechanism:

   • Terraform does not support automatic rollback like tools such as Helm (in Kubernetes). If a resource creation fails halfway, Terraform won’t automatically revert the previously successful steps.

2. Lack of Retry Mechanism:

   • If a resource fails due to issues like rate limiting or throttling, Terraform does not automatically retry the failed step.

3. Manual Drift Detection:

   • Terraform cannot detect manual changes made directly in the cloud console unless you run terraform plan or terraform apply explicitly. This can lead to inconsistencies between your state file and actual infrastructure.

4. Limited Programming Logic:

   • Terraform’s scripting capabilities are limited. It supports only basic conditionals, loops (with count or for_each), and variables. Complex programming logic is not its strength.

5. State File as Single Source of Truth:

   • The state file can become corrupted or lost, leading to problems tracking and managing infrastructure.

6. Not GitOps-Friendly:

   • Terraform is not naturally integrated with GitOps tools. For true GitOps, tools like Flux or ArgoCD may be better suited.

---

Conclusion

To wrap up, here are the core points you should keep in mind when using Terraform at scale:

• Always use remote backends (e.g., S3, Azure Blob) to store your Terraform state.

• Implement state locking to prevent concurrent modifications using DynamoDB (in AWS).

• Use outputs.tf to provide important details about the created resources to end-users.

• Avoid manual changes in the cloud provider’s console; always make changes through Terraform.

• Be aware of limitations like lack of rollback, retry mechanisms, and drift detection.