Terraform and NREC: Part IV - Pairing with Ansible

Last changed: 2024-09-17

This document builds on Terraform Part 1, Part 2 and Part 3. In Part 3 we built an environment in NREC consisting of 4 frontend web servers and 1 backend database server, as well as security groups and an SSH key pair for access. However, we didn’t do anything inside the operating systems of our instances to make them act as web servers or database servers. Terraform only interacts with the Openstack API, and doesn’t do anything inside the instances. Here is where Ansible comes into play. Using Ansible, we’ll configure the web servers and the database server to make a real service.

This is not an introduction to Ansible. It is assumed that the reader has some knowledge and experience in using Ansible.

The files used in this document can be downloaded:

• terraform.yaml

• main.tf

• secgroup.tf

• variables.tf

• terraform.tfvars

• web.yaml

• db.yaml

The examples in this document have been tested and verified with Terraform version 1.9.5:

Terraform v1.9.5
on linux_amd64
+ provider registry.terraform.io/ansible/ansible v1.3.0
+ provider registry.terraform.io/terraform-provider-openstack/openstack v2.1.0

Installing Ansible

Ansible is available in most Linux distributions. If possible, use the Ansible version availble from the distribution. For Fedora, and for RHEL or equivalent with EPEL enabled, simply install using yum:

# yum install ansible

For Debian and Ubuntu:

# apt-get install ansible

When using Ansible with NREC and instances created with Terraform, we’ll often create and destroy instances multiple times. This depends on your workflow. It may be beneficial to add the following configuration to your ~/.ansible.cfg to prevent Ansible from halting on unknown SSH host keys:

[defaults]
host_key_checking = False

Installing Terraform Plugin for Ansible

Ansible has a Terraform collection that reads information from the Terraform state file terraform.tfstate. This needs to be installed for this to work. Run the following command to install it:

ansible-galaxy collection install cloud.terraform

On Linux, this installs the Terraform collection under:

~/.ansible/collections/ansible_collections

You can verify the Terraform collection like this:

$ ansible-galaxy collection list | grep terraform
cloud.terraform 3.0.0

Ansible Information in Terraform State

The Terraform collection for Ansible expects certain data in the Terraform state. We will provide this by adding some extra statements to our Terraform main.tf file. First we need to add the Ansible plugin:

main.tf

# Define required providers
terraform {
  required_version = ">= 1.0"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
    }
    ansible = {
      version = "~> 1.3.0"
      source  = "ansible/ansible"
    }
  }
}

Running terraform init will then add the Ansible plugin along with the Openstack plugin.

Next, we need to add statements that tells Ansible the relevant details of our hosts. We define our web and database hosts, as well as groups that contain these hosts:

main.tf

# Ansible web hosts
resource "ansible_host" "web" {
  count  = lookup(var.role_count, "web", 0)
  name   = "${var.region}-web-${count.index}"
  groups = ["osl_web"] # Groups this host is part of

  variables = {
    ansible_host = trim(openstack_compute_instance_v2.web_instance[count.index].access_ip_v6, "[]")
  }
}

# Ansible db hosts
resource "ansible_host" "db" {
  count  = lookup(var.role_count, "db", 0)
  name   = "${var.region}-db-${count.index}"
  groups = ["osl_db"] # Groups this host is part of

  variables = {
    ansible_host = trim(openstack_compute_instance_v2.db_instance[count.index].access_ip_v6, "[]")
  }
}

# Ansible web group
resource "ansible_group" "web_group" {
  name     = "web"
  children = ["osl_web"]
  variables = {
    ansible_user = "almalinux"
  }
}

# Ansible db group
resource "ansible_group" "db_group" {
  name     = "db"
  children = ["osl_db"]
  variables = {
    ansible_user = "ubuntu"
  }
}

Note the use of the trim() function for the ansible_host variable. This is needed when using the IPv6 address.

Ansible Inventory File

The whole point is that the inventory should be generated dynamically based on the Terraform state. But we need to tell Ansible where to locate what it needs. For this we create an inventory file that we call “terraform.yaml”:

terraform.yaml

plugin: cloud.terraform.terraform_provider
project_path: /path/to/project
# Terraform binary (available in the $PATH) or full path to the binary.
binary_path: terraform

You will need to edit this file and set the full path to the Terraform directory, i.e. where you terraform.tfstate is located.

Testing Ansible

Note

When using the cloud.terraform collection with an unsupported version of Ansible, we get a warning as seen in the examples below. It still works in our case (RHEL9).

With the terraform.yaml file in place, we can run ansible to test and verify that it is able to reach the instances over the network:

First, we’ll see that our inventory is correct:

$ ansible-inventory -i terraform.yaml --graph --vars
[WARNING]: Collection cloud.terraform does not support Ansible version 2.14.14
@all:
  |--@ungrouped:
  |--@db:
  |  |--@osl_db:
  |  |  |--osl-db-0
  |  |  |  |--{ansible_host = 2001:700:2:8201::10cf}
  |  |  |  |--{ansible_user = ubuntu}
  |  |--{ansible_user = ubuntu}
  |--@web:
  |  |--@osl_web:
  |  |  |--osl-web-0
  |  |  |  |--{ansible_host = 2001:700:2:8201::11ae}
  |  |  |  |--{ansible_user = almalinux}
  |  |  |--osl-web-1
  |  |  |  |--{ansible_host = 2001:700:2:8201::1414}
  |  |  |  |--{ansible_user = almalinux}
  |  |  |--osl-web-2
  |  |  |  |--{ansible_host = 2001:700:2:8201::1236}
  |  |  |  |--{ansible_user = almalinux}
  |  |  |--osl-web-3
  |  |  |  |--{ansible_host = 2001:700:2:8201::111d}
  |  |  |  |--{ansible_user = almalinux}
  |  |--{ansible_user = almalinux}

Next, we’ll ping the instances using Ansible:

$ ansible -i terraform.yaml all -m ping
[WARNING]: Collection cloud.terraform does not support Ansible version 2.14.14
osl-web-2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
osl-web-1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
osl-web-3 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
osl-web-0 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
osl-db-0 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

We can also run a command on the instances to verify that SSH connection is working:

$ ansible -i terraform.yaml all -m shell -a 'uname -sr'
[WARNING]: Collection cloud.terraform does not support Ansible version 2.14.14
osl-db-0 | CHANGED | rc=0 >>
Linux 6.8.0-41-generic
osl-web-0 | CHANGED | rc=0 >>
Linux 5.14.0-427.31.1.el9_4.x86_64
osl-web-2 | CHANGED | rc=0 >>
Linux 5.14.0-427.31.1.el9_4.x86_64
osl-web-1 | CHANGED | rc=0 >>
Linux 5.14.0-427.31.1.el9_4.x86_64
osl-web-3 | CHANGED | rc=0 >>
Linux 5.14.0-427.31.1.el9_4.x86_64

We have verified that Ansible and dynamic inventory from Terraform state works, and are ready to proceed.

Using Ansible

Important

This section includes simple playbooks to show how Ansible can be used for configuring the OS and services. In order to make this into a real service for production use, a lot more work needs to be done.

I order to configure the web and database servers, we have created two playbooks. They are named web.yaml and db.yaml, respectively. We’ll take a look at web.yaml first:

web.yaml

---
- hosts: web
  become: true

  tasks:
  - name: Install Apache
    ansible.builtin.yum:
      name: httpd

  - name: Install php
    ansible.builtin.yum:
      name: php

  - name: Install php-mysqlnd
    ansible.builtin.yum:
      name: php-mysqlnd

  - name: Set httpd_can_network_connect_db SELinux boolean
    ansible.builtin.seboolean:
      name: httpd_can_network_connect_db
      state: yes
      persistent: yes

  - name: Make sure Apache is running and enabled
    ansible.builtin.systemd:
      name: httpd
      state: started
      enabled: yes

In this playbook, we do the following:

• Install the Apache web server, PHP and the PHP MySQL bindings

• Make sure that SELinux allows Apache to connect to the database

• Make sure that the web service is enabled and running.

Next, lets take a look at db.yaml which we use to configure the database server:

db.yaml

---
- hosts: db
  become: true

  tasks:
  - name: Create XFS filesystem on /dev/sdb
    ansible.builtin.filesystem:
      fstype: xfs
      dev: /dev/sdb

  - name: Mount volume
    ansible.builtin.mount:
      path: /var/lib/mysql
      src: /dev/sdb
      fstype: xfs
      state: mounted

  - name: Install MariaDB, also starts the service
    ansible.builtin.apt:
      name: mariadb-server
      update_cache: yes

  - name: Set MariaDB bind-address
    ansible.builtin.ini_file:
      path: /etc/mysql/mariadb.conf.d/90-server.cnf
      section: mysqld
      option: bind-address
      value: "{{ ansible_default_ipv4.address }}"
      backup: yes
    notify:
    - restart MariaDB

  - name: Make sure MariaDB is running and enabled
    ansible.builtin.systemd:
      name: mariadb
      state: started
      enabled: yes

  - name: Install PyMySQL, needed by Ansible MySQL module
    ansible.builtin.apt:
      name: python3-pymysql


  handlers:
  - name: restart MariaDB
    ansible.builtin.systemd:
      name: "mariadb"
      state: restarted

In this playbook, we do the following:

• Create a filesystem on our volume, available as the /dev/sdb device, and mount it as /var/lib/mysql

• Install MariaDB (i.e. MySQL)

• Set the MariaDB bind address, i.e. the IP address that we want the database server to listen to. We use the internal, private IPv4 address for this. When using the IPv6 network in NREC, instances also get a private IPv4 address. We can use this address for communication between instances, which in our case will be communication between the web servers and the database.

• Make sure that the database service is enabled and running.

• Install the MySQL bindings for Python. This is needed if we want to use Ansible to communicate with the database server, e.g. for creating databases.

The db.yaml also includes a handler for restarting MariaDB if we have done configuration changes which require a restart to take effect.

The Ansible playbooks above would be run like this, from the Terraform workspace directory:

$ ansible-playbook -i terraform.yaml db.yaml
$ ansible-playbook -i terraform.yaml web.yaml

---

Complete example

A complete listing of the example files used in this document is provided below.

terraform.yaml

plugin: cloud.terraform.terraform_provider
project_path: /path/to/project
# Terraform binary (available in the $PATH) or full path to the binary.
binary_path: terraform

main.tf

# Define required providers
terraform {
  required_version = ">= 1.0"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
    }
    ansible = {
      version = "~> 1.3.0"
      source  = "ansible/ansible"
    }
  }
}

# Configure the OpenStack Provider
# Empty means using environment variables "OS_*". More info:
# https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs
provider "openstack" {}

# SSH key
resource "openstack_compute_keypair_v2" "keypair" {
  region     = var.region
  name       = "${terraform.workspace}-${var.name}"
  public_key = file(var.ssh_public_key)
}

# Web servers
resource "openstack_compute_instance_v2" "web_instance" {
  region      = var.region
  count       = lookup(var.role_count, "web", 0)
  name        = "${var.region}-web-${count.index}"
  image_name  = lookup(var.role_image, "web", "unknown")
  flavor_name = lookup(var.role_flavor, "web", "unknown")

  key_pair = "${terraform.workspace}-${var.name}"
  security_groups = [
    "default",
    "${terraform.workspace}-${var.name}-ssh",
    "${terraform.workspace}-${var.name}-web",
  ]

  network {
    name = "IPv6"
  }

  lifecycle {
    ignore_changes = [image_name,image_id]
  }

  depends_on = [
    openstack_networking_secgroup_v2.instance_ssh_access,
    openstack_networking_secgroup_v2.instance_web_access,
  ]
}

# Database servers
resource "openstack_compute_instance_v2" "db_instance" {
  region      = var.region
  count       = lookup(var.role_count, "db", 0)
  name        = "${var.region}-db-${count.index}"
  image_name  = lookup(var.role_image, "db", "unknown")
  flavor_name = lookup(var.role_flavor, "db", "unknown")

  key_pair = "${terraform.workspace}-${var.name}"
  security_groups = [
    "default",
    "${terraform.workspace}-${var.name}-ssh",
    "${terraform.workspace}-${var.name}-db",
  ]

  network {
    name = "IPv6"
  }

  lifecycle {
    ignore_changes = [image_name,image_id]
  }

  depends_on = [
    openstack_networking_secgroup_v2.instance_ssh_access,
    openstack_networking_secgroup_v2.instance_db_access,
  ]
}

# Volume
resource "openstack_blockstorage_volume_v3" "volume" {
  name = "database"
  size = var.volume_size
}

# Attach volume
resource "openstack_compute_volume_attach_v2" "attach_vol" {
  instance_id = openstack_compute_instance_v2.db_instance[0].id
  volume_id   = openstack_blockstorage_volume_v3.volume.id
}

# Ansible web hosts
resource "ansible_host" "web" {
  count  = lookup(var.role_count, "web", 0)
  name   = "${var.region}-web-${count.index}"
  groups = ["osl_web"] # Groups this host is part of

  variables = {
    ansible_host = trim(openstack_compute_instance_v2.web_instance[count.index].access_ip_v6, "[]")
  }
}

# Ansible db hosts
resource "ansible_host" "db" {
  count  = lookup(var.role_count, "db", 0)
  name   = "${var.region}-db-${count.index}"
  groups = ["osl_db"] # Groups this host is part of

  variables = {
    ansible_host = trim(openstack_compute_instance_v2.db_instance[count.index].access_ip_v6, "[]")
  }
}

# Ansible web group
resource "ansible_group" "web_group" {
  name     = "web"
  children = ["osl_web"]
  variables = {
    ansible_user = "almalinux"
  }
}

# Ansible db group
resource "ansible_group" "db_group" {
  name     = "db"
  children = ["osl_db"]
  variables = {
    ansible_user = "ubuntu"
  }
}

secgroup.tf

# Security group SSH + ICMP
resource "openstack_networking_secgroup_v2" "instance_ssh_access" {
  region      = var.region
  name        = "${terraform.workspace}-${var.name}-ssh"
  description = "Security group for allowing SSH and ICMP access"
}

# Security group HTTP
resource "openstack_networking_secgroup_v2" "instance_web_access" {
  region      = var.region
  name        = "${terraform.workspace}-${var.name}-web"
  description = "Security group for allowing HTTP access"
}

# Security group MySQL
resource "openstack_networking_secgroup_v2" "instance_db_access" {
  region      = var.region
  name        = "${terraform.workspace}-${var.name}-db"
  description = "Security group for allowing MySQL access"
}

# Allow ssh from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv4" {
  region            = var.region
  count             = length(var.allow_ssh_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = var.allow_ssh_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow ssh from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv6" {
  region            = var.region
  count             = length(var.allow_ssh_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = var.allow_ssh_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow icmp from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv4" {
  region            = var.region
  count             = length(var.allow_ssh_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "icmp"
  remote_ip_prefix  = var.allow_ssh_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow icmp from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv6" {
  region            = var.region
  count             = length(var.allow_ssh_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "ipv6-icmp"
  remote_ip_prefix  = var.allow_ssh_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow HTTP from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv4" {
  region            = var.region
  count             = length(var.allow_http_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = var.allow_http_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
}

# Allow HTTP from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv6" {
  region            = var.region
  count             = length(var.allow_http_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = var.allow_http_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
}

# Allow MySQL from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv4" {
  region            = var.region
  count             = length(var.allow_mysql_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_ip_prefix  = var.allow_mysql_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

# Allow MYSQL from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv6" {
  region            = var.region
  count             = length(var.allow_mysql_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_ip_prefix  = var.allow_mysql_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

# Allow MYSQL from web servers (IPv4)
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv4" {
  region            = var.region
  count             = 1
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_group_id   = openstack_networking_secgroup_v2.instance_web_access.id
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

# Allow MYSQL from web servers (IPv6)
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv6" {
  region            = var.region
  count             = 1
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_group_id   = openstack_networking_secgroup_v2.instance_web_access.id
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

variables.tf

# Variables
variable "region" {
}

variable "name" {
  default = "myproject"
}

variable "ssh_public_key" {
  default = "~/.ssh/id_rsa.pub"
}

variable "network" {
  default = "IPv6"
}

variable "volume_size" {
  default = 20
}

# Security group defaults
variable "allow_ssh_from_v6" {
  type    = list(string)
  default = []
}

variable "allow_ssh_from_v4" {
  type    = list(string)
  default = []
}

variable "allow_http_from_v6" {
  type    = list(string)
  default = []
}

variable "allow_http_from_v4" {
  type    = list(string)
  default = []
}

variable "allow_mysql_from_v6" {
  type    = list(string)
  default = []
}

variable "allow_mysql_from_v4" {
  type    = list(string)
  default = []
}

# Mapping between role and image
variable "role_image" {
  type = map(string)
  default = {
    "web" = "GOLD Alma Linux 9"
    "db"  = "GOLD Ubuntu 24.04 LTS"
  }
}

# Mapping between role and flavor
variable "role_flavor" {
  type = map(string)
  default = {
    "web" = "m1.small"
    "db"  = "m1.medium"
  }
}

# Mapping between role and number of instances (count)
variable "role_count" {
  type = map(string)
  default = {
    "web" = 4
    "db"  = 1
  }
}

terraform.tfvars

# Region
region = "osl"

# This is needed to access the instance over ssh
allow_ssh_from_v6 = [
  "2001:700:100:8070::/64",
  "2001:700:100:8071::/64",
]
allow_ssh_from_v4 = [
  "129.240.114.32/28",
  "129.240.114.48/28"
]

# This is needed to access the instance over http
allow_http_from_v6 = [
  "2001:700:200::/48",
  "2001:700:100::/41"
]
allow_http_from_v4 = [
  "129.177.0.0/16",
  "129.240.0.0/16"
]

# This is needed to access the instance over the mysql port
allow_mysql_from_v6 = [
  "2001:700:100:4003::43/128"
]
allow_mysql_from_v4 = [
  "129.240.130.12/32"
]

web.yaml

---
- hosts: web
  become: true

  tasks:
  - name: Install Apache
    ansible.builtin.yum:
      name: httpd

  - name: Install php
    ansible.builtin.yum:
      name: php

  - name: Install php-mysqlnd
    ansible.builtin.yum:
      name: php-mysqlnd

  - name: Set httpd_can_network_connect_db SELinux boolean
    ansible.builtin.seboolean:
      name: httpd_can_network_connect_db
      state: yes
      persistent: yes

  - name: Make sure Apache is running and enabled
    ansible.builtin.systemd:
      name: httpd
      state: started
      enabled: yes

db.yaml

---
- hosts: db
  become: true

  tasks:
  - name: Create XFS filesystem on /dev/sdb
    ansible.builtin.filesystem:
      fstype: xfs
      dev: /dev/sdb

  - name: Mount volume
    ansible.builtin.mount:
      path: /var/lib/mysql
      src: /dev/sdb
      fstype: xfs
      state: mounted

  - name: Install MariaDB, also starts the service
    ansible.builtin.apt:
      name: mariadb-server
      update_cache: yes

  - name: Set MariaDB bind-address
    ansible.builtin.ini_file:
      path: /etc/mysql/mariadb.conf.d/90-server.cnf
      section: mysqld
      option: bind-address
      value: "{{ ansible_default_ipv4.address }}"
      backup: yes
    notify:
    - restart MariaDB

  - name: Make sure MariaDB is running and enabled
    ansible.builtin.systemd:
      name: mariadb
      state: started
      enabled: yes

  - name: Install PyMySQL, needed by Ansible MySQL module
    ansible.builtin.apt:
      name: python3-pymysql


  handlers:
  - name: restart MariaDB
    ansible.builtin.systemd:
      name: "mariadb"
      state: restarted