Terraform and NREC: Part IV - Pairing with Ansible

Last changed: 2023-09-22

Important

NB! Does not work! Changes to Terraform and Ansible has rendered this documentation obsolete. We will review and update as soon as possible.

This document builds on Terraform Part 1, Part 2 and Part 3. In Part 3 we built an environment in NREC consisting of 4 frontend web servers and 1 backend database server, as well as security groups and an SSH key pair for access. However, we didn’t do anything inside the operating systems of our instances to make them act as web servers or database servers. Terraform only interacts with the Openstack API, and doesn’t do anything inside the instances. Here is where Ansible comes into play. Using Ansible, we’ll configure the web servers and the database server to make a real service.

This is not an introduction to Ansible. It is assumed that the reader has some knowledge and experience in using Ansible.

The files used in this document can be downloaded:

Installing Ansible

Ansible is available in most Linux distributions. If possible, use the Ansible version availble from the distribution. For Fedora, and for RHEL or CentOS with EPEL enabled, simply install using yum:

# yum install ansible

For Debian and Ubuntu:

# apt-get install ansible

When using Ansible with NREC and instances created with Terraform, we’ll often create and destroy instances multiple times. This depends on your workflow. It may be beneficial to add the following configuration to your ~/.ansible.cfg to prevent Ansible from halting on unknown SSH host keys:

[defaults]
host_key_checking = False

Ansible inventory from Terraform state

Terraform maintains a state in the working directory, and is also able to update its local state against the real resources in NREC. The local state is stored in terraform.tfstate, and we’re using a Python script that reads this file and produces an Ansible inventory dynamically.

To use the Python script we need to install and set it up:

  • Download terraform.py

  • Put it in ~/.local/bin and make sure it is executable:

    $ mv ~/Downloads/terraform.py ~/.local/bin/
$ chmod a+x ~/.local/bin/terraform.py

This installs the Python script terraform.py into ~/.local/bin (e.g. your home directory). Usually, this directory should be in your shell path. If it isn’t, you can add it (for bash):

export PATH=$PATH:~/.local/bin

In your Terraform working directory, create a directory called “inventory”, and create a symbolic link “hosts” that points to the terraform.py script:

$ mkdir inventory
$ ln -s ~/.local/bin/terraform.py inventory/hosts

You can then run ansible from within your Terraform working directory to verify that dynamic inventory works:

$ ansible all -i inventory --list-hosts
  hosts (5):
    bgo-db-0
    bgo-web-0
    bgo-web-1
    bgo-web-2
    bgo-web-3

This only lists the hosts and verifies that the dynamic inventory works. Having Ansible actually connect to the hosts requires additional configuration as described in the next section.

Configuring Ansible connectivity

In order for Ansible to function correctly we need to tell Ansible additional information about the instances. First, we add a map in variables.tf to address the SSH user. Ansible needs to know which SSH user to connect as:

variables.tf
1
2
3
4
5
6
7
8
# Mapping between role and SSH user
variable "role_ssh_user" {
  type = map(string)
  default = {
    "web" = "centos"
    "db"  = "ubuntu"
  }
}

Next, we need to use those variables and add a metadata directive in the compute instance resource. The script terraform.py will use this metadata to correctly create inventory for Ansible. For the web servers (CentOS 8):

main.tf
1
2
3
4
5
  metadata = {
    ssh_user       = lookup(var.role_ssh_user, "web", "unknown")
    prefer_ipv6    = 1
    my_server_role = "web"
  }

And for the database server (Ubuntu 21.04 LTS):

main.tf
1
2
3
4
5
6
  metadata = {
    ssh_user       = lookup(var.role_ssh_user, "db", "unknown")
    prefer_ipv6    = 1
    python_bin     = "/usr/bin/python3"
    my_server_role = "database"
  }

We have added this metadata:

  • ssh_user: Using the map variable created in variables.tf (see above).
  • prefer_ipv6: This tells terraform.py that we want Ansible to use the IPv6 address of the instance. This is needed in our case as we’re using the IPv6 network type in NREC. When using the dualStack network, this is usually not needed.
  • python_bin: This is only used on the database server (Ubuntu). Ansible needs a working Python binary to function, and in Ubuntu’s case there isn’t a /usr/bin/python and Ansible needs to be explicitly told which binary to use on the instance.
  • my_server_role: We use this to control how to identify the web servers and database servers in the Ansible inventory.

With these in place, having applied the configuration with terraform apply, we can run terraform.py to view our inventory:

$ terraform.py --root . --hostfile
## begin hosts generated by terraform.py ##
2001:700:2:8301::113b bgo-db-0
2001:700:2:8301::113f bgo-web-0
2001:700:2:8301::100a bgo-web-1
2001:700:2:8301::100b bgo-web-2
2001:700:2:8301::1129 bgo-web-3
## end hosts generated by terraform.py ##

And we run ansible to verify that it is able to reach the instances over the network:

$ ansible all -i inventory -m ping
bgo-web-3 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-web-1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-db-0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-web-2 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
bgo-web-0 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

We can also run a command on the instances to verify that SSH connection is working:

$ ansible all -i inventory -m shell -a 'uname -sr'
bgo-web-3 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-web-1 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-web-2 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-web-0 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64

bgo-db-0 | CHANGED | rc=0 >>
Linux 4.15.0-70-generic

We have verified that Ansible and dynamic inventory from Terraform state works, and are ready to proceed.

Using Ansible

Important

This section includes simple playbooks to show how Ansible can be used for configuring the OS and services. In order to make this into a real service for production use, a lot more work needs to be done.

I order to configure the web and database servers, we have created two playbooks. They are named web.yaml and db.yaml, respectively. We’ll take a look at web.yaml first:

web.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
---
- hosts: os_metadata_my_server_role=web
  become: true

  tasks:
  - name: Install Apache
    yum:
      name: httpd

  - name: Install php
    yum:
      name: php

  - name: Install php-mysql
    yum:
      name: php-mysql

  - name: Set httpd_can_network_connect_db SELinux boolean
    seboolean:
      name: httpd_can_network_connect_db
      state: yes
      persistent: yes

  - name: Make sure Apache is running and enabled
    systemd:
      name: httpd
      state: started
      enabled: yes

In this playbook, we do the following:

  • Install the Apache web server, PHP and the PHP MySQL bindings
  • Make sure that SELinux allows Apache to connect to the database
  • Make sure that the web service is enabled and running.

Next, lets take a look at db.yaml which we use to configure the database server:

db.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
---
- hosts: os_metadata_my_server_role=database
  become: true

  tasks:
  - name: Create XFS filesystem on /dev/sdb
    filesystem:
      fstype: xfs
      dev: /dev/sdb

  - name: Mount volume
    mount:
      path: /var/lib/mysql
      src: /dev/sdb
      fstype: xfs
      state: mounted

  - name: Install MariaDB, also starts the service
    apt:
      name: mariadb-server

  - name: Set MariaDB bind-address
    ini_file:
      path: /etc/mysql/mariadb.conf.d/90-server.cnf
      section: mysqld
      option: bind-address
      value: "{{ ansible_default_ipv4.address }}"
      backup: yes
    notify:
    - restart MariaDB

  - name: Make sure MariaDB is running and enabled
    systemd:
      name: mariadb
      state: started
      enabled: yes

  - name: Install PyMySQL, needed by Ansible MySQL module
    apt:
      name: python3-pymysql


  handlers:
  - name: restart MariaDB
    systemd:
      name: "mariadb"
      state: restarted

In this playbook, we do the following:

  • Create a filesystem on our volume, available as the /dev/sdb device, and mount it as /var/lib/mysql
  • Install MariaDB (i.e. MySQL)
  • Set the MariaDB bind address, i.e. the IP address that we want the database server to listen to. We use the internal, private IPv4 address for this. When using the IPv6 network in NREC, instances also get a private IPv4 address. We can use this address for communication between instances, which in our case will be communication between the web servers and the database.
  • Make sure that the database service is enabled and running.
  • Install the MySQL bindings for Python. This is needed if we want to use Ansible to communicate with the database server, e.g. for creating databases.

The db.yaml also includes a handler for restarting MariaDB if we have done configuration changes which require a restart to take effect.

The Ansible playbooks above would be run like this, from the Terraform workspace directory:

$ ansible-playbook -i inventory db.yaml
$ ansible-playbook -i inventory web.yaml

Complete example

A complete listing of the example files used in this document is provided below.

main.tf
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
# Define required providers
terraform {
  required_version = ">= 1.0"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
    }
  }
}

# Configure the OpenStack Provider
# Empty means using environment variables "OS_*". More info:
# https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs
provider "openstack" {}

# SSH key
resource "openstack_compute_keypair_v2" "keypair" {
  region     = var.region
  name       = "${terraform.workspace}-${var.name}"
  public_key = file(var.ssh_public_key)
}

# Web servers
resource "openstack_compute_instance_v2" "web_instance" {
  region      = var.region
  count       = lookup(var.role_count, "web", 0)
  name        = "${var.region}-web-${count.index}"
  image_name  = lookup(var.role_image, "web", "unknown")
  flavor_name = lookup(var.role_flavor, "web", "unknown")

  key_pair = "${terraform.workspace}-${var.name}"
  security_groups = [
    "default",
    "${terraform.workspace}-${var.name}-ssh",
    "${terraform.workspace}-${var.name}-web",
  ]

  network {
    name = "IPv6"
  }

  metadata = {
    ssh_user       = lookup(var.role_ssh_user, "web", "unknown")
    prefer_ipv6    = 1
    my_server_role = "web"
  }

  lifecycle {
    ignore_changes = [image_name]
  }

  depends_on = [
    openstack_networking_secgroup_v2.instance_ssh_access,
    openstack_networking_secgroup_v2.instance_web_access,
  ]
}

# Database servers
resource "openstack_compute_instance_v2" "db_instance" {
  region      = var.region
  count       = lookup(var.role_count, "db", 0)
  name        = "${var.region}-db-${count.index}"
  image_name  = lookup(var.role_image, "db", "unknown")
  flavor_name = lookup(var.role_flavor, "db", "unknown")

  key_pair = "${terraform.workspace}-${var.name}"
  security_groups = [
    "default",
    "${terraform.workspace}-${var.name}-ssh",
    "${terraform.workspace}-${var.name}-db",
  ]

  network {
    name = "IPv6"
  }

  metadata = {
    ssh_user       = lookup(var.role_ssh_user, "db", "unknown")
    prefer_ipv6    = 1
    python_bin     = "/usr/bin/python3"
    my_server_role = "database"
  }

  lifecycle {
    ignore_changes = [image_name]
  }

  depends_on = [
    openstack_networking_secgroup_v2.instance_ssh_access,
    openstack_networking_secgroup_v2.instance_db_access,
  ]
}

# Volume
resource "openstack_blockstorage_volume_v2" "volume" {
  name = "database"
  size = var.volume_size
}

# Attach volume
resource "openstack_compute_volume_attach_v2" "attach_vol" {
  instance_id = openstack_compute_instance_v2.db_instance[0].id
  volume_id   = openstack_blockstorage_volume_v2.volume.id
}
secgroup.tf
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
# Security group SSH + ICMP
resource "openstack_networking_secgroup_v2" "instance_ssh_access" {
  region      = var.region
  name        = "${terraform.workspace}-${var.name}-ssh"
  description = "Security group for allowing SSH and ICMP access"
}

# Security group HTTP
resource "openstack_networking_secgroup_v2" "instance_web_access" {
  region      = var.region
  name        = "${terraform.workspace}-${var.name}-web"
  description = "Security group for allowing HTTP access"
}

# Security group MySQL
resource "openstack_networking_secgroup_v2" "instance_db_access" {
  region      = var.region
  name        = "${terraform.workspace}-${var.name}-db"
  description = "Security group for allowing MySQL access"
}

# Allow ssh from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv4" {
  region            = var.region
  count             = length(var.allow_ssh_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = var.allow_ssh_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow ssh from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv6" {
  region            = var.region
  count             = length(var.allow_ssh_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = var.allow_ssh_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow icmp from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv4" {
  region            = var.region
  count             = length(var.allow_ssh_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "icmp"
  remote_ip_prefix  = var.allow_ssh_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow icmp from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv6" {
  region            = var.region
  count             = length(var.allow_ssh_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "icmp"
  remote_ip_prefix  = var.allow_ssh_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}

# Allow HTTP from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv4" {
  region            = var.region
  count             = length(var.allow_http_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = var.allow_http_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
}

# Allow HTTP from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv6" {
  region            = var.region
  count             = length(var.allow_http_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = var.allow_http_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
}

# Allow MySQL from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv4" {
  region            = var.region
  count             = length(var.allow_mysql_from_v4)
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_ip_prefix  = var.allow_mysql_from_v4[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

# Allow MYSQL from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv6" {
  region            = var.region
  count             = length(var.allow_mysql_from_v6)
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_ip_prefix  = var.allow_mysql_from_v6[count.index]
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

# Allow MYSQL from web servers (IPv4)
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv4" {
  region            = var.region
  count             = 1
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_group_id   = openstack_networking_secgroup_v2.instance_web_access.id
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}

# Allow MYSQL from web servers (IPv6)
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv6" {
  region            = var.region
  count             = 1
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 3306
  port_range_max    = 3306
  remote_group_id   = openstack_networking_secgroup_v2.instance_web_access.id
  security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}
variables.tf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
# Variables
variable "region" {
}

variable "name" {
  default = "myproject"
}

variable "ssh_public_key" {
  default = "~/.ssh/id_rsa.pub"
}

variable "network" {
  default = "IPv6"
}

variable "volume_size" {
  default = 20
}

variable "metadata" {
  type    = list(string)
  default = []
}

# Security group defaults
variable "allow_ssh_from_v6" {
  type    = list(string)
  default = []
}

variable "allow_ssh_from_v4" {
  type    = list(string)
  default = []
}

variable "allow_http_from_v6" {
  type    = list(string)
  default = []
}

variable "allow_http_from_v4" {
  type    = list(string)
  default = []
}

variable "allow_mysql_from_v6" {
  type    = list(string)
  default = []
}

variable "allow_mysql_from_v4" {
  type    = list(string)
  default = []
}

# Mapping between role and image
variable "role_image" {
  type = map(string)
  default = {
    "web" = "GOLD CentOS 8"
    "db"  = "GOLD Ubuntu 21.04 LTS"
  }
}

# Mapping between role and flavor
variable "role_flavor" {
  type = map(string)
  default = {
    "web" = "m1.small"
    "db"  = "m1.medium"
  }
}

# Mapping between role and number of instances (count)
variable "role_count" {
  type = map(string)
  default = {
    "web" = 4
    "db"  = 1
  }
}

# Mapping between role and SSH user
variable "role_ssh_user" {
  type = map(string)
  default = {
    "web" = "centos"
    "db"  = "ubuntu"
  }
}
terraform.tfvars
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Region
region = "osl"

# This is needed to access the instance over ssh
allow_ssh_from_v6 = [
  "2001:700:100:12::7/128",
  "2001:700:100:118::67/128"
]
allow_ssh_from_v4 = [
  "129.240.12.7/32",
  "129.240.118.67/32"
]

# This is needed to access the instance over http
allow_http_from_v6 = [
  "2001:700:200::/48",
  "2001:700:100::/41"
]
allow_http_from_v4 = [
  "129.177.0.0/16",
  "129.240.0.0/16"
]

# This is needed to access the instance over the mysql port
allow_mysql_from_v6 = [
  "2001:700:100:118::67/128"
]
allow_mysql_from_v4 = [
  "129.240.118.67/32"
]
web.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
---
- hosts: os_metadata_my_server_role=web
  become: true

  tasks:
  - name: Install Apache
    yum:
      name: httpd

  - name: Install php
    yum:
      name: php

  - name: Install php-mysql
    yum:
      name: php-mysql

  - name: Set httpd_can_network_connect_db SELinux boolean
    seboolean:
      name: httpd_can_network_connect_db
      state: yes
      persistent: yes

  - name: Make sure Apache is running and enabled
    systemd:
      name: httpd
      state: started
      enabled: yes
db.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
---
- hosts: os_metadata_my_server_role=database
  become: true

  tasks:
  - name: Create XFS filesystem on /dev/sdb
    filesystem:
      fstype: xfs
      dev: /dev/sdb

  - name: Mount volume
    mount:
      path: /var/lib/mysql
      src: /dev/sdb
      fstype: xfs
      state: mounted

  - name: Install MariaDB, also starts the service
    apt:
      name: mariadb-server

  - name: Set MariaDB bind-address
    ini_file:
      path: /etc/mysql/mariadb.conf.d/90-server.cnf
      section: mysqld
      option: bind-address
      value: "{{ ansible_default_ipv4.address }}"
      backup: yes
    notify:
    - restart MariaDB

  - name: Make sure MariaDB is running and enabled
    systemd:
      name: mariadb
      state: started
      enabled: yes

  - name: Install PyMySQL, needed by Ansible MySQL module
    apt:
      name: python3-pymysql


  handlers:
  - name: restart MariaDB
    systemd:
      name: "mariadb"
      state: restarted