Terraform and NREC: Part IV - Pairing with Ansible

Last changed: 2024-02-21

Important

NB! Does not work! Changes to Terraform and Ansible has rendered this documentation obsolete. We will review and update as soon as possible.

This document builds on Terraform Part 1, Part 2 and Part 3. In Part 3 we built an environment in NREC consisting of 4 frontend web servers and 1 backend database server, as well as security groups and an SSH key pair for access. However, we didn’t do anything inside the operating systems of our instances to make them act as web servers or database servers. Terraform only interacts with the Openstack API, and doesn’t do anything inside the instances. Here is where Ansible comes into play. Using Ansible, we’ll configure the web servers and the database server to make a real service.

This is not an introduction to Ansible. It is assumed that the reader has some knowledge and experience in using Ansible.

The files used in this document can be downloaded:

Installing Ansible

Ansible is available in most Linux distributions. If possible, use the Ansible version availble from the distribution. For Fedora, and for RHEL or CentOS with EPEL enabled, simply install using yum:

# yum install ansible

For Debian and Ubuntu:

# apt-get install ansible

When using Ansible with NREC and instances created with Terraform, we’ll often create and destroy instances multiple times. This depends on your workflow. It may be beneficial to add the following configuration to your ~/.ansible.cfg to prevent Ansible from halting on unknown SSH host keys:

[defaults]
host_key_checking = False

Ansible inventory from Terraform state

Terraform maintains a state in the working directory, and is also able to update its local state against the real resources in NREC. The local state is stored in terraform.tfstate, and we’re using a Python script that reads this file and produces an Ansible inventory dynamically.

To use the Python script we need to install and set it up:

  • Download terraform.py

  • Put it in ~/.local/bin and make sure it is executable:

    $ mv ~/Downloads/terraform.py ~/.local/bin/
    $ chmod a+x ~/.local/bin/terraform.py
    

This installs the Python script terraform.py into ~/.local/bin (e.g. your home directory). Usually, this directory should be in your shell path. If it isn’t, you can add it (for bash):

export PATH=$PATH:~/.local/bin

In your Terraform working directory, create a directory called “inventory”, and create a symbolic link “hosts” that points to the terraform.py script:

$ mkdir inventory
$ ln -s ~/.local/bin/terraform.py inventory/hosts

You can then run ansible from within your Terraform working directory to verify that dynamic inventory works:

$ ansible all -i inventory --list-hosts
  hosts (5):
    bgo-db-0
    bgo-web-0
    bgo-web-1
    bgo-web-2
    bgo-web-3

This only lists the hosts and verifies that the dynamic inventory works. Having Ansible actually connect to the hosts requires additional configuration as described in the next section.

Configuring Ansible connectivity

In order for Ansible to function correctly we need to tell Ansible additional information about the instances. First, we add a map in variables.tf to address the SSH user. Ansible needs to know which SSH user to connect as:

variables.tf
1# Mapping between role and SSH user
2variable "role_ssh_user" {
3  type = map(string)
4  default = {
5    "web" = "centos"
6    "db"  = "ubuntu"
7  }
8}

Next, we need to use those variables and add a metadata directive in the compute instance resource. The script terraform.py will use this metadata to correctly create inventory for Ansible. For the web servers (CentOS 8):

main.tf
1  metadata = {
2    ssh_user       = lookup(var.role_ssh_user, "web", "unknown")
3    prefer_ipv6    = 1
4    my_server_role = "web"
5  }

And for the database server (Ubuntu 21.04 LTS):

main.tf
1  metadata = {
2    ssh_user       = lookup(var.role_ssh_user, "db", "unknown")
3    prefer_ipv6    = 1
4    python_bin     = "/usr/bin/python3"
5    my_server_role = "database"
6  }

We have added this metadata:

  • ssh_user: Using the map variable created in variables.tf (see above).

  • prefer_ipv6: This tells terraform.py that we want Ansible to use the IPv6 address of the instance. This is needed in our case as we’re using the IPv6 network type in NREC. When using the dualStack network, this is usually not needed.

  • python_bin: This is only used on the database server (Ubuntu). Ansible needs a working Python binary to function, and in Ubuntu’s case there isn’t a /usr/bin/python and Ansible needs to be explicitly told which binary to use on the instance.

  • my_server_role: We use this to control how to identify the web servers and database servers in the Ansible inventory.

With these in place, having applied the configuration with terraform apply, we can run terraform.py to view our inventory:

$ terraform.py --root . --hostfile
## begin hosts generated by terraform.py ##
2001:700:2:8301::113b bgo-db-0
2001:700:2:8301::113f bgo-web-0
2001:700:2:8301::100a bgo-web-1
2001:700:2:8301::100b bgo-web-2
2001:700:2:8301::1129 bgo-web-3
## end hosts generated by terraform.py ##

And we run ansible to verify that it is able to reach the instances over the network:

$ ansible all -i inventory -m ping
bgo-web-3 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-web-1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-db-0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-web-2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-web-0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

We can also run a command on the instances to verify that SSH connection is working:

$ ansible all -i inventory -m shell -a 'uname -sr'
bgo-web-3 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-web-1 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-web-2 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-web-0 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-db-0 | CHANGED | rc=0 >>
Linux 4.15.0-70-generic

We have verified that Ansible and dynamic inventory from Terraform state works, and are ready to proceed.

Using Ansible

Important

This section includes simple playbooks to show how Ansible can be used for configuring the OS and services. In order to make this into a real service for production use, a lot more work needs to be done.

I order to configure the web and database servers, we have created two playbooks. They are named web.yaml and db.yaml, respectively. We’ll take a look at web.yaml first:

web.yaml
 1---
 2- hosts: os_metadata_my_server_role=web
 3  become: true
 4
 5  tasks:
 6  - name: Install Apache
 7    yum:
 8      name: httpd
 9
10  - name: Install php
11    yum:
12      name: php
13
14  - name: Install php-mysql
15    yum:
16      name: php-mysql
17
18  - name: Set httpd_can_network_connect_db SELinux boolean
19    seboolean:
20      name: httpd_can_network_connect_db
21      state: yes
22      persistent: yes
23
24  - name: Make sure Apache is running and enabled
25    systemd:
26      name: httpd
27      state: started
28      enabled: yes

In this playbook, we do the following:

  • Install the Apache web server, PHP and the PHP MySQL bindings

  • Make sure that SELinux allows Apache to connect to the database

  • Make sure that the web service is enabled and running.

Next, lets take a look at db.yaml which we use to configure the database server:

db.yaml
 1---
 2- hosts: os_metadata_my_server_role=database
 3  become: true
 4
 5  tasks:
 6  - name: Create XFS filesystem on /dev/sdb
 7    filesystem:
 8      fstype: xfs
 9      dev: /dev/sdb
10
11  - name: Mount volume
12    mount:
13      path: /var/lib/mysql
14      src: /dev/sdb
15      fstype: xfs
16      state: mounted
17
18  - name: Install MariaDB, also starts the service
19    apt:
20      name: mariadb-server
21
22  - name: Set MariaDB bind-address
23    ini_file:
24      path: /etc/mysql/mariadb.conf.d/90-server.cnf
25      section: mysqld
26      option: bind-address
27      value: "{{ ansible_default_ipv4.address }}"
28      backup: yes
29    notify:
30    - restart MariaDB
31
32  - name: Make sure MariaDB is running and enabled
33    systemd:
34      name: mariadb
35      state: started
36      enabled: yes
37
38  - name: Install PyMySQL, needed by Ansible MySQL module
39    apt:
40      name: python3-pymysql
41
42
43  handlers:
44  - name: restart MariaDB
45    systemd:
46      name: "mariadb"
47      state: restarted

In this playbook, we do the following:

  • Create a filesystem on our volume, available as the /dev/sdb device, and mount it as /var/lib/mysql

  • Install MariaDB (i.e. MySQL)

  • Set the MariaDB bind address, i.e. the IP address that we want the database server to listen to. We use the internal, private IPv4 address for this. When using the IPv6 network in NREC, instances also get a private IPv4 address. We can use this address for communication between instances, which in our case will be communication between the web servers and the database.

  • Make sure that the database service is enabled and running.

  • Install the MySQL bindings for Python. This is needed if we want to use Ansible to communicate with the database server, e.g. for creating databases.

The db.yaml also includes a handler for restarting MariaDB if we have done configuration changes which require a restart to take effect.

The Ansible playbooks above would be run like this, from the Terraform workspace directory:

$ ansible-playbook -i inventory db.yaml
$ ansible-playbook -i inventory web.yaml

Complete example

A complete listing of the example files used in this document is provided below.

main.tf
  1# Define required providers
  2terraform {
  3  required_version = ">= 1.0"
  4  required_providers {
  5    openstack = {
  6      source  = "terraform-provider-openstack/openstack"
  7    }
  8  }
  9}
 10
 11# Configure the OpenStack Provider
 12# Empty means using environment variables "OS_*". More info:
 13# https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs
 14provider "openstack" {}
 15
 16# SSH key
 17resource "openstack_compute_keypair_v2" "keypair" {
 18  region     = var.region
 19  name       = "${terraform.workspace}-${var.name}"
 20  public_key = file(var.ssh_public_key)
 21}
 22
 23# Web servers
 24resource "openstack_compute_instance_v2" "web_instance" {
 25  region      = var.region
 26  count       = lookup(var.role_count, "web", 0)
 27  name        = "${var.region}-web-${count.index}"
 28  image_name  = lookup(var.role_image, "web", "unknown")
 29  flavor_name = lookup(var.role_flavor, "web", "unknown")
 30
 31  key_pair = "${terraform.workspace}-${var.name}"
 32  security_groups = [
 33    "default",
 34    "${terraform.workspace}-${var.name}-ssh",
 35    "${terraform.workspace}-${var.name}-web",
 36  ]
 37
 38  network {
 39    name = "IPv6"
 40  }
 41
 42  metadata = {
 43    ssh_user       = lookup(var.role_ssh_user, "web", "unknown")
 44    prefer_ipv6    = 1
 45    my_server_role = "web"
 46  }
 47
 48  lifecycle {
 49    ignore_changes = [image_name,image_id]
 50  }
 51
 52  depends_on = [
 53    openstack_networking_secgroup_v2.instance_ssh_access,
 54    openstack_networking_secgroup_v2.instance_web_access,
 55  ]
 56}
 57
 58# Database servers
 59resource "openstack_compute_instance_v2" "db_instance" {
 60  region      = var.region
 61  count       = lookup(var.role_count, "db", 0)
 62  name        = "${var.region}-db-${count.index}"
 63  image_name  = lookup(var.role_image, "db", "unknown")
 64  flavor_name = lookup(var.role_flavor, "db", "unknown")
 65
 66  key_pair = "${terraform.workspace}-${var.name}"
 67  security_groups = [
 68    "default",
 69    "${terraform.workspace}-${var.name}-ssh",
 70    "${terraform.workspace}-${var.name}-db",
 71  ]
 72
 73  network {
 74    name = "IPv6"
 75  }
 76
 77  metadata = {
 78    ssh_user       = lookup(var.role_ssh_user, "db", "unknown")
 79    prefer_ipv6    = 1
 80    python_bin     = "/usr/bin/python3"
 81    my_server_role = "database"
 82  }
 83
 84  lifecycle {
 85    ignore_changes = [image_name,image_id]
 86  }
 87
 88  depends_on = [
 89    openstack_networking_secgroup_v2.instance_ssh_access,
 90    openstack_networking_secgroup_v2.instance_db_access,
 91  ]
 92}
 93
 94# Volume
 95resource "openstack_blockstorage_volume_v2" "volume" {
 96  name = "database"
 97  size = var.volume_size
 98}
 99
100# Attach volume
101resource "openstack_compute_volume_attach_v2" "attach_vol" {
102  instance_id = openstack_compute_instance_v2.db_instance[0].id
103  volume_id   = openstack_blockstorage_volume_v2.volume.id
104}
secgroup.tf
  1# Security group SSH + ICMP
  2resource "openstack_networking_secgroup_v2" "instance_ssh_access" {
  3  region      = var.region
  4  name        = "${terraform.workspace}-${var.name}-ssh"
  5  description = "Security group for allowing SSH and ICMP access"
  6}
  7
  8# Security group HTTP
  9resource "openstack_networking_secgroup_v2" "instance_web_access" {
 10  region      = var.region
 11  name        = "${terraform.workspace}-${var.name}-web"
 12  description = "Security group for allowing HTTP access"
 13}
 14
 15# Security group MySQL
 16resource "openstack_networking_secgroup_v2" "instance_db_access" {
 17  region      = var.region
 18  name        = "${terraform.workspace}-${var.name}-db"
 19  description = "Security group for allowing MySQL access"
 20}
 21
 22# Allow ssh from IPv4 net
 23resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv4" {
 24  region            = var.region
 25  count             = length(var.allow_ssh_from_v4)
 26  direction         = "ingress"
 27  ethertype         = "IPv4"
 28  protocol          = "tcp"
 29  port_range_min    = 22
 30  port_range_max    = 22
 31  remote_ip_prefix  = var.allow_ssh_from_v4[count.index]
 32  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
 33}
 34
 35# Allow ssh from IPv6 net
 36resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv6" {
 37  region            = var.region
 38  count             = length(var.allow_ssh_from_v6)
 39  direction         = "ingress"
 40  ethertype         = "IPv6"
 41  protocol          = "tcp"
 42  port_range_min    = 22
 43  port_range_max    = 22
 44  remote_ip_prefix  = var.allow_ssh_from_v6[count.index]
 45  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
 46}
 47
 48# Allow icmp from IPv4 net
 49resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv4" {
 50  region            = var.region
 51  count             = length(var.allow_ssh_from_v4)
 52  direction         = "ingress"
 53  ethertype         = "IPv4"
 54  protocol          = "icmp"
 55  remote_ip_prefix  = var.allow_ssh_from_v4[count.index]
 56  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
 57}
 58
 59# Allow icmp from IPv6 net
 60resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv6" {
 61  region            = var.region
 62  count             = length(var.allow_ssh_from_v6)
 63  direction         = "ingress"
 64  ethertype         = "IPv6"
 65  protocol          = "icmp"
 66  remote_ip_prefix  = var.allow_ssh_from_v6[count.index]
 67  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
 68}
 69
 70# Allow HTTP from IPv4 net
 71resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv4" {
 72  region            = var.region
 73  count             = length(var.allow_http_from_v4)
 74  direction         = "ingress"
 75  ethertype         = "IPv4"
 76  protocol          = "tcp"
 77  port_range_min    = 80
 78  port_range_max    = 80
 79  remote_ip_prefix  = var.allow_http_from_v4[count.index]
 80  security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
 81}
 82
 83# Allow HTTP from IPv6 net
 84resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv6" {
 85  region            = var.region
 86  count             = length(var.allow_http_from_v6)
 87  direction         = "ingress"
 88  ethertype         = "IPv6"
 89  protocol          = "tcp"
 90  port_range_min    = 80
 91  port_range_max    = 80
 92  remote_ip_prefix  = var.allow_http_from_v6[count.index]
 93  security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
 94}
 95
 96# Allow MySQL from IPv4 net
 97resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv4" {
 98  region            = var.region
 99  count             = length(var.allow_mysql_from_v4)
100  direction         = "ingress"
101  ethertype         = "IPv4"
102  protocol          = "tcp"
103  port_range_min    = 3306
104  port_range_max    = 3306
105  remote_ip_prefix  = var.allow_mysql_from_v4[count.index]
106  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
107}
108
109# Allow MYSQL from IPv6 net
110resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv6" {
111  region            = var.region
112  count             = length(var.allow_mysql_from_v6)
113  direction         = "ingress"
114  ethertype         = "IPv6"
115  protocol          = "tcp"
116  port_range_min    = 3306
117  port_range_max    = 3306
118  remote_ip_prefix  = var.allow_mysql_from_v6[count.index]
119  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
120}
121
122# Allow MYSQL from web servers (IPv4)
123resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv4" {
124  region            = var.region
125  count             = 1
126  direction         = "ingress"
127  ethertype         = "IPv4"
128  protocol          = "tcp"
129  port_range_min    = 3306
130  port_range_max    = 3306
131  remote_group_id   = openstack_networking_secgroup_v2.instance_web_access.id
132  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
133}
134
135# Allow MYSQL from web servers (IPv6)
136resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv6" {
137  region            = var.region
138  count             = 1
139  direction         = "ingress"
140  ethertype         = "IPv6"
141  protocol          = "tcp"
142  port_range_min    = 3306
143  port_range_max    = 3306
144  remote_group_id   = openstack_networking_secgroup_v2.instance_web_access.id
145  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
146}
variables.tf
 1# Variables
 2variable "region" {
 3}
 4
 5variable "name" {
 6  default = "myproject"
 7}
 8
 9variable "ssh_public_key" {
10  default = "~/.ssh/id_rsa.pub"
11}
12
13variable "network" {
14  default = "IPv6"
15}
16
17variable "volume_size" {
18  default = 20
19}
20
21variable "metadata" {
22  type    = list(string)
23  default = []
24}
25
26# Security group defaults
27variable "allow_ssh_from_v6" {
28  type    = list(string)
29  default = []
30}
31
32variable "allow_ssh_from_v4" {
33  type    = list(string)
34  default = []
35}
36
37variable "allow_http_from_v6" {
38  type    = list(string)
39  default = []
40}
41
42variable "allow_http_from_v4" {
43  type    = list(string)
44  default = []
45}
46
47variable "allow_mysql_from_v6" {
48  type    = list(string)
49  default = []
50}
51
52variable "allow_mysql_from_v4" {
53  type    = list(string)
54  default = []
55}
56
57# Mapping between role and image
58variable "role_image" {
59  type = map(string)
60  default = {
61    "web" = "GOLD CentOS 8"
62    "db"  = "GOLD Ubuntu 21.04 LTS"
63  }
64}
65
66# Mapping between role and flavor
67variable "role_flavor" {
68  type = map(string)
69  default = {
70    "web" = "m1.small"
71    "db"  = "m1.medium"
72  }
73}
74
75# Mapping between role and number of instances (count)
76variable "role_count" {
77  type = map(string)
78  default = {
79    "web" = 4
80    "db"  = 1
81  }
82}
83
84# Mapping between role and SSH user
85variable "role_ssh_user" {
86  type = map(string)
87  default = {
88    "web" = "centos"
89    "db"  = "ubuntu"
90  }
91}
terraform.tfvars
 1# Region
 2region = "osl"
 3
 4# This is needed to access the instance over ssh
 5allow_ssh_from_v6 = [
 6  "2001:700:100:12::7/128",
 7  "2001:700:100:118::67/128"
 8]
 9allow_ssh_from_v4 = [
10  "129.240.12.7/32",
11  "129.240.118.67/32"
12]
13
14# This is needed to access the instance over http
15allow_http_from_v6 = [
16  "2001:700:200::/48",
17  "2001:700:100::/41"
18]
19allow_http_from_v4 = [
20  "129.177.0.0/16",
21  "129.240.0.0/16"
22]
23
24# This is needed to access the instance over the mysql port
25allow_mysql_from_v6 = [
26  "2001:700:100:118::67/128"
27]
28allow_mysql_from_v4 = [
29  "129.240.118.67/32"
30]
web.yaml
 1---
 2- hosts: os_metadata_my_server_role=web
 3  become: true
 4
 5  tasks:
 6  - name: Install Apache
 7    yum:
 8      name: httpd
 9
10  - name: Install php
11    yum:
12      name: php
13
14  - name: Install php-mysql
15    yum:
16      name: php-mysql
17
18  - name: Set httpd_can_network_connect_db SELinux boolean
19    seboolean:
20      name: httpd_can_network_connect_db
21      state: yes
22      persistent: yes
23
24  - name: Make sure Apache is running and enabled
25    systemd:
26      name: httpd
27      state: started
28      enabled: yes
db.yaml
 1---
 2- hosts: os_metadata_my_server_role=database
 3  become: true
 4
 5  tasks:
 6  - name: Create XFS filesystem on /dev/sdb
 7    filesystem:
 8      fstype: xfs
 9      dev: /dev/sdb
10
11  - name: Mount volume
12    mount:
13      path: /var/lib/mysql
14      src: /dev/sdb
15      fstype: xfs
16      state: mounted
17
18  - name: Install MariaDB, also starts the service
19    apt:
20      name: mariadb-server
21
22  - name: Set MariaDB bind-address
23    ini_file:
24      path: /etc/mysql/mariadb.conf.d/90-server.cnf
25      section: mysqld
26      option: bind-address
27      value: "{{ ansible_default_ipv4.address }}"
28      backup: yes
29    notify:
30    - restart MariaDB
31
32  - name: Make sure MariaDB is running and enabled
33    systemd:
34      name: mariadb
35      state: started
36      enabled: yes
37
38  - name: Install PyMySQL, needed by Ansible MySQL module
39    apt:
40      name: python3-pymysql
41
42
43  handlers:
44  - name: restart MariaDB
45    systemd:
46      name: "mariadb"
47      state: restarted