Terraform and NREC: Part IV - Pairing with Ansible¶
Last changed: 2023-09-22
Important
NB! Does not work! Changes to Terraform and Ansible has rendered this documentation obsolete. We will review and update as soon as possible.
Contents
This document builds on Terraform Part 1, Part 2 and Part 3. In Part 3 we built an environment in NREC consisting of 4 frontend web servers and 1 backend database server, as well as security groups and an SSH key pair for access. However, we didn’t do anything inside the operating systems of our instances to make them act as web servers or database servers. Terraform only interacts with the Openstack API, and doesn’t do anything inside the instances. Here is where Ansible comes into play. Using Ansible, we’ll configure the web servers and the database server to make a real service.
This is not an introduction to Ansible. It is assumed that the reader has some knowledge and experience in using Ansible.
The files used in this document can be downloaded:
Installing Ansible¶
Ansible is available in most Linux distributions. If possible, use the Ansible version availble from the distribution. For Fedora, and for RHEL or CentOS with EPEL enabled, simply install using yum:
# yum install ansible
For Debian and Ubuntu:
# apt-get install ansible
When using Ansible with NREC and instances created with Terraform,
we’ll often create and destroy instances multiple times. This depends
on your workflow. It may be beneficial to add the following
configuration to your ~/.ansible.cfg
to prevent Ansible from
halting on unknown SSH host keys:
[defaults]
host_key_checking = False
Ansible inventory from Terraform state¶
Terraform maintains a state in the working directory, and is also able
to update its local state against the real resources in NREC. The
local state is stored in terraform.tfstate
, and we’re using a
Python script that reads this file and produces an Ansible inventory
dynamically.
To use the Python script we need to install and set it up:
Download
terraform.py
Put it in
~/.local/bin
and make sure it is executable:$ mv ~/Downloads/terraform.py ~/.local/bin/ $ chmod a+x ~/.local/bin/terraform.py
This installs the Python script terraform.py
into ~/.local/bin
(e.g. your home directory). Usually, this directory should be in your
shell path. If it isn’t, you can add it (for bash):
export PATH=$PATH:~/.local/bin
In your Terraform working directory, create a directory called
“inventory”, and create a symbolic link “hosts” that points to the
terraform.py
script:
$ mkdir inventory
$ ln -s ~/.local/bin/terraform.py inventory/hosts
You can then run ansible from within your Terraform working directory to verify that dynamic inventory works:
$ ansible all -i inventory --list-hosts
hosts (5):
bgo-db-0
bgo-web-0
bgo-web-1
bgo-web-2
bgo-web-3
This only lists the hosts and verifies that the dynamic inventory works. Having Ansible actually connect to the hosts requires additional configuration as described in the next section.
Configuring Ansible connectivity¶
In order for Ansible to function correctly we need to tell Ansible
additional information about the instances. First, we add a map in
variables.tf
to address the SSH user. Ansible needs to know which
SSH user to connect as:
1 2 3 4 5 6 7 8 | # Mapping between role and SSH user
variable "role_ssh_user" {
type = map(string)
default = {
"web" = "centos"
"db" = "ubuntu"
}
}
|
Next, we need to use those variables and add a metadata directive in the compute instance resource. The script terraform.py will use this metadata to correctly create inventory for Ansible. For the web servers (CentOS 8):
1 2 3 4 5 | metadata = {
ssh_user = lookup(var.role_ssh_user, "web", "unknown")
prefer_ipv6 = 1
my_server_role = "web"
}
|
And for the database server (Ubuntu 21.04 LTS):
1 2 3 4 5 6 | metadata = {
ssh_user = lookup(var.role_ssh_user, "db", "unknown")
prefer_ipv6 = 1
python_bin = "/usr/bin/python3"
my_server_role = "database"
}
|
We have added this metadata:
ssh_user
: Using the map variable created in variables.tf (see above).prefer_ipv6
: This tells terraform.py that we want Ansible to use the IPv6 address of the instance. This is needed in our case as we’re using the IPv6 network type in NREC. When using the dualStack network, this is usually not needed.python_bin
: This is only used on the database server (Ubuntu). Ansible needs a working Python binary to function, and in Ubuntu’s case there isn’t a/usr/bin/python
and Ansible needs to be explicitly told which binary to use on the instance.my_server_role
: We use this to control how to identify the web servers and database servers in the Ansible inventory.
With these in place, having applied the configuration with terraform
apply
, we can run terraform.py to view our inventory:
$ terraform.py --root . --hostfile
## begin hosts generated by terraform.py ##
2001:700:2:8301::113b bgo-db-0
2001:700:2:8301::113f bgo-web-0
2001:700:2:8301::100a bgo-web-1
2001:700:2:8301::100b bgo-web-2
2001:700:2:8301::1129 bgo-web-3
## end hosts generated by terraform.py ##
And we run ansible to verify that it is able to reach the instances over the network:
$ ansible all -i inventory -m ping
bgo-web-3 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-web-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-db-0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-web-2 | SUCCESS => {
"changed": false,
"ping": "pong"
}
bgo-web-0 | SUCCESS => {
"changed": false,
"ping": "pong"
}
We can also run a command on the instances to verify that SSH connection is working:
$ ansible all -i inventory -m shell -a 'uname -sr'
bgo-web-3 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-web-1 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-web-2 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-web-0 | CHANGED | rc=0 >>
Linux 3.10.0-1062.4.3.el7.x86_64
bgo-db-0 | CHANGED | rc=0 >>
Linux 4.15.0-70-generic
We have verified that Ansible and dynamic inventory from Terraform state works, and are ready to proceed.
Using Ansible¶
Important
This section includes simple playbooks to show how Ansible can be used for configuring the OS and services. In order to make this into a real service for production use, a lot more work needs to be done.
I order to configure the web and database servers, we have created two
playbooks. They are named web.yaml
and db.yaml
,
respectively. We’ll take a look at web.yaml
first:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | ---
- hosts: os_metadata_my_server_role=web
become: true
tasks:
- name: Install Apache
yum:
name: httpd
- name: Install php
yum:
name: php
- name: Install php-mysql
yum:
name: php-mysql
- name: Set httpd_can_network_connect_db SELinux boolean
seboolean:
name: httpd_can_network_connect_db
state: yes
persistent: yes
- name: Make sure Apache is running and enabled
systemd:
name: httpd
state: started
enabled: yes
|
In this playbook, we do the following:
- Install the Apache web server, PHP and the PHP MySQL bindings
- Make sure that SELinux allows Apache to connect to the database
- Make sure that the web service is enabled and running.
Next, lets take a look at db.yaml
which we use to configure the
database server:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
- hosts: os_metadata_my_server_role=database
become: true
tasks:
- name: Create XFS filesystem on /dev/sdb
filesystem:
fstype: xfs
dev: /dev/sdb
- name: Mount volume
mount:
path: /var/lib/mysql
src: /dev/sdb
fstype: xfs
state: mounted
- name: Install MariaDB, also starts the service
apt:
name: mariadb-server
- name: Set MariaDB bind-address
ini_file:
path: /etc/mysql/mariadb.conf.d/90-server.cnf
section: mysqld
option: bind-address
value: "{{ ansible_default_ipv4.address }}"
backup: yes
notify:
- restart MariaDB
- name: Make sure MariaDB is running and enabled
systemd:
name: mariadb
state: started
enabled: yes
- name: Install PyMySQL, needed by Ansible MySQL module
apt:
name: python3-pymysql
handlers:
- name: restart MariaDB
systemd:
name: "mariadb"
state: restarted
|
In this playbook, we do the following:
- Create a filesystem on our volume, available as the
/dev/sdb
device, and mount it as/var/lib/mysql
- Install MariaDB (i.e. MySQL)
- Set the MariaDB bind address, i.e. the IP address that we want the database server to listen to. We use the internal, private IPv4 address for this. When using the IPv6 network in NREC, instances also get a private IPv4 address. We can use this address for communication between instances, which in our case will be communication between the web servers and the database.
- Make sure that the database service is enabled and running.
- Install the MySQL bindings for Python. This is needed if we want to use Ansible to communicate with the database server, e.g. for creating databases.
The db.yaml
also includes a handler for restarting MariaDB if we
have done configuration changes which require a restart to take
effect.
The Ansible playbooks above would be run like this, from the Terraform workspace directory:
$ ansible-playbook -i inventory db.yaml
$ ansible-playbook -i inventory web.yaml
Complete example¶
A complete listing of the example files used in this document is provided below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | # Define required providers
terraform {
required_version = ">= 1.0"
required_providers {
openstack = {
source = "terraform-provider-openstack/openstack"
}
}
}
# Configure the OpenStack Provider
# Empty means using environment variables "OS_*". More info:
# https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs
provider "openstack" {}
# SSH key
resource "openstack_compute_keypair_v2" "keypair" {
region = var.region
name = "${terraform.workspace}-${var.name}"
public_key = file(var.ssh_public_key)
}
# Web servers
resource "openstack_compute_instance_v2" "web_instance" {
region = var.region
count = lookup(var.role_count, "web", 0)
name = "${var.region}-web-${count.index}"
image_name = lookup(var.role_image, "web", "unknown")
flavor_name = lookup(var.role_flavor, "web", "unknown")
key_pair = "${terraform.workspace}-${var.name}"
security_groups = [
"default",
"${terraform.workspace}-${var.name}-ssh",
"${terraform.workspace}-${var.name}-web",
]
network {
name = "IPv6"
}
metadata = {
ssh_user = lookup(var.role_ssh_user, "web", "unknown")
prefer_ipv6 = 1
my_server_role = "web"
}
lifecycle {
ignore_changes = [image_name]
}
depends_on = [
openstack_networking_secgroup_v2.instance_ssh_access,
openstack_networking_secgroup_v2.instance_web_access,
]
}
# Database servers
resource "openstack_compute_instance_v2" "db_instance" {
region = var.region
count = lookup(var.role_count, "db", 0)
name = "${var.region}-db-${count.index}"
image_name = lookup(var.role_image, "db", "unknown")
flavor_name = lookup(var.role_flavor, "db", "unknown")
key_pair = "${terraform.workspace}-${var.name}"
security_groups = [
"default",
"${terraform.workspace}-${var.name}-ssh",
"${terraform.workspace}-${var.name}-db",
]
network {
name = "IPv6"
}
metadata = {
ssh_user = lookup(var.role_ssh_user, "db", "unknown")
prefer_ipv6 = 1
python_bin = "/usr/bin/python3"
my_server_role = "database"
}
lifecycle {
ignore_changes = [image_name]
}
depends_on = [
openstack_networking_secgroup_v2.instance_ssh_access,
openstack_networking_secgroup_v2.instance_db_access,
]
}
# Volume
resource "openstack_blockstorage_volume_v2" "volume" {
name = "database"
size = var.volume_size
}
# Attach volume
resource "openstack_compute_volume_attach_v2" "attach_vol" {
instance_id = openstack_compute_instance_v2.db_instance[0].id
volume_id = openstack_blockstorage_volume_v2.volume.id
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | # Security group SSH + ICMP
resource "openstack_networking_secgroup_v2" "instance_ssh_access" {
region = var.region
name = "${terraform.workspace}-${var.name}-ssh"
description = "Security group for allowing SSH and ICMP access"
}
# Security group HTTP
resource "openstack_networking_secgroup_v2" "instance_web_access" {
region = var.region
name = "${terraform.workspace}-${var.name}-web"
description = "Security group for allowing HTTP access"
}
# Security group MySQL
resource "openstack_networking_secgroup_v2" "instance_db_access" {
region = var.region
name = "${terraform.workspace}-${var.name}-db"
description = "Security group for allowing MySQL access"
}
# Allow ssh from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv4" {
region = var.region
count = length(var.allow_ssh_from_v4)
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 22
port_range_max = 22
remote_ip_prefix = var.allow_ssh_from_v4[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}
# Allow ssh from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_ssh_access_ipv6" {
region = var.region
count = length(var.allow_ssh_from_v6)
direction = "ingress"
ethertype = "IPv6"
protocol = "tcp"
port_range_min = 22
port_range_max = 22
remote_ip_prefix = var.allow_ssh_from_v6[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}
# Allow icmp from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv4" {
region = var.region
count = length(var.allow_ssh_from_v4)
direction = "ingress"
ethertype = "IPv4"
protocol = "icmp"
remote_ip_prefix = var.allow_ssh_from_v4[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}
# Allow icmp from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_icmp_access_ipv6" {
region = var.region
count = length(var.allow_ssh_from_v6)
direction = "ingress"
ethertype = "IPv6"
protocol = "icmp"
remote_ip_prefix = var.allow_ssh_from_v6[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_ssh_access.id
}
# Allow HTTP from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv4" {
region = var.region
count = length(var.allow_http_from_v4)
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 80
port_range_max = 80
remote_ip_prefix = var.allow_http_from_v4[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
}
# Allow HTTP from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_http_access_ipv6" {
region = var.region
count = length(var.allow_http_from_v6)
direction = "ingress"
ethertype = "IPv6"
protocol = "tcp"
port_range_min = 80
port_range_max = 80
remote_ip_prefix = var.allow_http_from_v6[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_web_access.id
}
# Allow MySQL from IPv4 net
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv4" {
region = var.region
count = length(var.allow_mysql_from_v4)
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 3306
port_range_max = 3306
remote_ip_prefix = var.allow_mysql_from_v4[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}
# Allow MYSQL from IPv6 net
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_access_ipv6" {
region = var.region
count = length(var.allow_mysql_from_v6)
direction = "ingress"
ethertype = "IPv6"
protocol = "tcp"
port_range_min = 3306
port_range_max = 3306
remote_ip_prefix = var.allow_mysql_from_v6[count.index]
security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}
# Allow MYSQL from web servers (IPv4)
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv4" {
region = var.region
count = 1
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 3306
port_range_max = 3306
remote_group_id = openstack_networking_secgroup_v2.instance_web_access.id
security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}
# Allow MYSQL from web servers (IPv6)
resource "openstack_networking_secgroup_rule_v2" "rule_mysql_from_web_access_ipv6" {
region = var.region
count = 1
direction = "ingress"
ethertype = "IPv6"
protocol = "tcp"
port_range_min = 3306
port_range_max = 3306
remote_group_id = openstack_networking_secgroup_v2.instance_web_access.id
security_group_id = openstack_networking_secgroup_v2.instance_db_access.id
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 | # Variables
variable "region" {
}
variable "name" {
default = "myproject"
}
variable "ssh_public_key" {
default = "~/.ssh/id_rsa.pub"
}
variable "network" {
default = "IPv6"
}
variable "volume_size" {
default = 20
}
variable "metadata" {
type = list(string)
default = []
}
# Security group defaults
variable "allow_ssh_from_v6" {
type = list(string)
default = []
}
variable "allow_ssh_from_v4" {
type = list(string)
default = []
}
variable "allow_http_from_v6" {
type = list(string)
default = []
}
variable "allow_http_from_v4" {
type = list(string)
default = []
}
variable "allow_mysql_from_v6" {
type = list(string)
default = []
}
variable "allow_mysql_from_v4" {
type = list(string)
default = []
}
# Mapping between role and image
variable "role_image" {
type = map(string)
default = {
"web" = "GOLD CentOS 8"
"db" = "GOLD Ubuntu 21.04 LTS"
}
}
# Mapping between role and flavor
variable "role_flavor" {
type = map(string)
default = {
"web" = "m1.small"
"db" = "m1.medium"
}
}
# Mapping between role and number of instances (count)
variable "role_count" {
type = map(string)
default = {
"web" = 4
"db" = 1
}
}
# Mapping between role and SSH user
variable "role_ssh_user" {
type = map(string)
default = {
"web" = "centos"
"db" = "ubuntu"
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # Region
region = "osl"
# This is needed to access the instance over ssh
allow_ssh_from_v6 = [
"2001:700:100:12::7/128",
"2001:700:100:118::67/128"
]
allow_ssh_from_v4 = [
"129.240.12.7/32",
"129.240.118.67/32"
]
# This is needed to access the instance over http
allow_http_from_v6 = [
"2001:700:200::/48",
"2001:700:100::/41"
]
allow_http_from_v4 = [
"129.177.0.0/16",
"129.240.0.0/16"
]
# This is needed to access the instance over the mysql port
allow_mysql_from_v6 = [
"2001:700:100:118::67/128"
]
allow_mysql_from_v4 = [
"129.240.118.67/32"
]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | ---
- hosts: os_metadata_my_server_role=web
become: true
tasks:
- name: Install Apache
yum:
name: httpd
- name: Install php
yum:
name: php
- name: Install php-mysql
yum:
name: php-mysql
- name: Set httpd_can_network_connect_db SELinux boolean
seboolean:
name: httpd_can_network_connect_db
state: yes
persistent: yes
- name: Make sure Apache is running and enabled
systemd:
name: httpd
state: started
enabled: yes
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
- hosts: os_metadata_my_server_role=database
become: true
tasks:
- name: Create XFS filesystem on /dev/sdb
filesystem:
fstype: xfs
dev: /dev/sdb
- name: Mount volume
mount:
path: /var/lib/mysql
src: /dev/sdb
fstype: xfs
state: mounted
- name: Install MariaDB, also starts the service
apt:
name: mariadb-server
- name: Set MariaDB bind-address
ini_file:
path: /etc/mysql/mariadb.conf.d/90-server.cnf
section: mysqld
option: bind-address
value: "{{ ansible_default_ipv4.address }}"
backup: yes
notify:
- restart MariaDB
- name: Make sure MariaDB is running and enabled
systemd:
name: mariadb
state: started
enabled: yes
- name: Install PyMySQL, needed by Ansible MySQL module
apt:
name: python3-pymysql
handlers:
- name: restart MariaDB
systemd:
name: "mariadb"
state: restarted
|