terrarum

Building OpenStack Environments Part 3

Mon, 09 Oct 2017 00:00:00 -0600

Introduction

Continuing on with the series of building disposable OpenStack environments for testing purposes, this part will cover how to install services which are not supported by PackStack.

While PackStack does an amazing job at easily and quickly creating OpenStack environments, it only has the ability to install a subset of services under the OpenStack umbrella. However, almost all OpenStack services are supported by RDO, the overarching package library for RedHat/CentOS.

For this part in the series, I will show how to install and configure Designate, the OpenStack DNS service, using the RDO packages.

Introduction
Planning the Installation
Adding the Installation Steps
- Installing PowerDNS
- Installing Designate
Build the Image and Launch
Conclusion

Planning the Installation

PackStack spoils us by hiding all of the steps required to install an OpenStack service. Installing a service requires a good amount of planning, even if the service is only going to be used for testing rather than production.

To begin planning, first read over any documentation you can find about the service in question. For Designate, there is a good amount of documentation here.

The overview page shows that there are a lot of moving pieces to Designate. Whether or not you need to account for all of these is still in question since it's possible that the RDO packages provide some sort of base configuration.

The installation page gives some brief steps about how to install Designate. By reading the page, you can see that all of the services listed in the Overview do not require special configuration. This makes things more simple.

Keep in mind that if you were to deploy Designate for production use, you might have to tune all of these services to suit your environment. Determining how to tune these services is out of scope for this blog post. Usually it requires careful reading of the various Designate configuration files, looking for supplementary information on mailing lists, and often even reading the source code itself.

The installation page shows how to use BIND as the default DNS driver. However, I'm going to change things up here. Instead, I will show how to use PowerDNS. There are two reasons for this:

I'm allergic to BIND.
I had trouble piecing together everything required to run Designate with the new PowerDNS driver, so this will also serve as documentation to help others.

Adding the Installation Steps

Continuing with Part 1 and Part 2, you should have a directory called terraform-openstack-test on your workstation. The structure of the directory should look something like this:

$ pwd
/home/jtopjian/terraform-openstack-test
$ tree .
.
├── files
│   └── deploy.sh
├── key
│   ├── id_rsa
│   └── id_rsa.pub
├── main.tf
└── packer
    ├── files
    │   ├── deploy.sh
    │   ├── packstack-answers.txt
    │   └── rc.local
    └── openstack
        ├── build.json
        └── main.tf

deploy.sh is used to install and configure PackStack and then strip any unique information from the installation. Packer then makes an image out of this installation. Finally, rc.local does some post-boot configuration.

To install and configure Designate, you will want to add additional pieces to both deploy.sh and rc.local.

Installing PowerDNS

First, install and configure PowerDNS. To do this, add the following to deploy.sh:

  hostnamectl set-hostname localhost

  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
  packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

+ mysql -e "CREATE DATABASE pdns default character set utf8 default collate utf8_general_ci"
+ mysql -e "GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY 'password'"
+
+ yum install -y epel-release yum-plugin-priorities
+ curl -o /etc/yum.repos.d/powerdns-auth-40.repo https://repo.powerdns.com/repo-files/centos-auth-40.repo
+ yum install -y pdns pdns-backend-mysql
+
+ echo "daemon=no
+ allow-recursion=127.0.0.1
+ config-dir=/etc/powerdns
+ daemon=yes
+ disable-axfr=no
+ guardian=yes
+ local-address=0.0.0.0
+ local-ipv6=::
+ local-port=53
+ setgid=pdns
+ setuid=pdns
+ slave=yes
+ socket-dir=/var/run
+ version-string=powerdns
+ out-of-zone-additional-processing=no
+ webserver=yes
+ api=yes
+ api-key=someapikey
+ launch=gmysql
+ gmysql-host=127.0.0.1
+ gmysql-user=pdns
+ gmysql-dbname=pdns
+ gmysql-password=password" | tee /etc/pdns/pdns.conf
+
+ mysql pdns < /home/centos/files/pdns.sql
+ sudo systemctl restart pdns

  yum install -y wget git
  wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get github.com/gophercloud/gophercloud
  pushd ~/go/src/github.com/gophercloud/gophercloud
  go get -u ./...
  popd

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
  fi

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/github.com/gophercloud/gophercloud
      go test -v -tags "fixtures acceptance" -run "\$1" github.com/gophercloud/gophercloud/acceptance/openstack/\$2 | tee ~/gophercloud.log
      popd
    fi
  }
  EOF

  systemctl stop openstack-cinder-backup.service
  systemctl stop openstack-cinder-scheduler.service
  systemctl stop openstack-cinder-volume.service
  systemctl stop openstack-nova-cert.service
  systemctl stop openstack-nova-compute.service
  systemctl stop openstack-nova-conductor.service
  systemctl stop openstack-nova-consoleauth.service
  systemctl stop openstack-nova-novncproxy.service
  systemctl stop openstack-nova-scheduler.service
  systemctl stop neutron-dhcp-agent.service
  systemctl stop neutron-l3-agent.service
  systemctl stop neutron-lbaasv2-agent.service
  systemctl stop neutron-metadata-agent.service
  systemctl stop neutron-openvswitch-agent.service
  systemctl stop neutron-metering-agent.service

  mysql -e "update services set deleted_at=now(), deleted=id" cinder
  mysql -e "update services set deleted_at=now(), deleted=id" nova
  mysql -e "update compute_nodes set deleted_at=now(), deleted=id" nova
  for i in $(openstack network agent list -c ID -f value); do
    neutron agent-delete $i
  done

  systemctl stop httpd

  cp /home/centos/files/rc.local /etc
  chmod +x /etc/rc.local

There are four things begin done above:

A MySQL database is created for PowerDNS.
PowerDNS is then installed.
A configuration file is created.
A database schema is imported into the PowerDNS database.

You'll notice the schema is located in a file titled files/pdns.sql. Add the following to terraform-openstack-test/packer/files/pdns.sql:

CREATE TABLE domains (
  id                    INT AUTO_INCREMENT,
  name                  VARCHAR(255) NOT NULL,
  master                VARCHAR(128) DEFAULT NULL,
  last_check            INT DEFAULT NULL,
  type                  VARCHAR(6) NOT NULL,
  notified_serial       INT DEFAULT NULL,
  account               VARCHAR(40) DEFAULT NULL,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE UNIQUE INDEX name_index ON domains(name);


CREATE TABLE records (
  id                    BIGINT AUTO_INCREMENT,
  domain_id             INT DEFAULT NULL,
  name                  VARCHAR(255) DEFAULT NULL,
  type                  VARCHAR(10) DEFAULT NULL,
  content               VARCHAR(64000) DEFAULT NULL,
  ttl                   INT DEFAULT NULL,
  prio                  INT DEFAULT NULL,
  change_date           INT DEFAULT NULL,
  disabled              TINYINT(1) DEFAULT 0,
  ordername             VARCHAR(255) BINARY DEFAULT NULL,
  auth                  TINYINT(1) DEFAULT 1,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX recordorder ON records (domain_id, ordername);


CREATE TABLE supermasters (
  ip                    VARCHAR(64) NOT NULL,
  nameserver            VARCHAR(255) NOT NULL,
  account               VARCHAR(40) NOT NULL,
  PRIMARY KEY (ip, nameserver)
) Engine=InnoDB;


CREATE TABLE comments (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  name                  VARCHAR(255) NOT NULL,
  type                  VARCHAR(10) NOT NULL,
  modified_at           INT NOT NULL,
  account               VARCHAR(40) NOT NULL,
  comment               VARCHAR(64000) NOT NULL,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE INDEX comments_domain_id_idx ON comments (domain_id);
CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);


CREATE TABLE domainmetadata (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  kind                  VARCHAR(32),
  content               TEXT,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);


CREATE TABLE cryptokeys (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  flags                 INT NOT NULL,
  active                BOOL,
  content               TEXT,
  PRIMARY KEY(id)
) Engine=InnoDB;

CREATE INDEX domainidindex ON cryptokeys(domain_id);


CREATE TABLE tsigkeys (
  id                    INT AUTO_INCREMENT,
  name                  VARCHAR(255),
  algorithm             VARCHAR(50),
  secret                VARCHAR(255),
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);

Installing Designate

Now that deploy.sh will install and configure PowerDNS, add the steps to install and configure Designate:

  hostnamectl set-hostname localhost

  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
  packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

  mysql -e "CREATE DATABASE pdns default character set utf8 default collate utf8_general_ci"
  mysql -e "GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY 'password'"

  yum install -y epel-release yum-plugin-priorities
  curl -o /etc/yum.repos.d/powerdns-auth-40.repo https://repo.powerdns.com/repo-files/centos-auth-40.repo
  yum install -y pdns pdns-backend-mysql

  echo "daemon=no
  allow-recursion=127.0.0.1
  config-dir=/etc/powerdns
  daemon=yes
  disable-axfr=no
  guardian=yes
  local-address=0.0.0.0
  local-ipv6=::
  local-port=53
  setgid=pdns
  setuid=pdns
  slave=yes
  socket-dir=/var/run
  version-string=powerdns
  out-of-zone-additional-processing=no
  webserver=yes
  api=yes
  api-key=someapikey
  launch=gmysql
  gmysql-host=127.0.0.1
  gmysql-user=pdns
  gmysql-dbname=pdns
  gmysql-password=password" | tee /etc/pdns/pdns.conf

  mysql pdns < /home/centos/files/pdns.sql
  sudo systemctl restart pdns

+ openstack user create --domain default --password password designate
+ openstack role add --project services --user designate admin
+ openstack service create --name designate --description "DNS" dns
+ openstack endpoint create --region RegionOne dns public http://127.0.0.1:9001/
+
+ mysql -e "CREATE DATABASE designate CHARACTER SET utf8 COLLATE utf8_general_ci"
+ mysql -e "CREATE DATABASE designate_pool_manager"
+ mysql -e "GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'localhost' IDENTIFIED BY 'password'"
+ mysql -e "GRANT ALL PRIVILEGES ON designate_pool_manager.* TO 'designate'@'localhost' IDENTIFIED BY 'password'"
+ mysql -e "GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'localhost' IDENTIFIED BY 'password'"
+
+ yum install -y crudini
+
+ yum install -y openstack-designate\*
+
+ cp /home/centos/files/pools.yaml /etc/designate/
+
+ designate_conf="/etc/designate/designate.conf"
+ crudini --set $designate_conf DEFAULT debug True
+ crudini --set $designate_conf DEFAULT debug True
+ crudini --set $designate_conf DEFAULT notification_driver messaging
+ crudini --set $designate_conf service:api enabled_extensions_v2 "quotas, reports"
+ crudini --set $designate_conf keystone_authtoken auth_uri http://127.0.0.1:5000
+ crudini --set $designate_conf keystone_authtoken auth_url http://127.0.0.1:35357
+ crudini --set $designate_conf keystone_authtoken username designate
+ crudini --set $designate_conf keystone_authtoken password password
+ crudini --set $designate_conf keystone_authtoken project_name services
+ crudini --set $designate_conf keystone_authtoken auth_type password
+ crudini --set $designate_conf service:worker enabled true
+ crudini --set $designate_conf service:worker notify true
+ crudini --set $designate_conf storage:sqlalchemy connection mysql+pymysql://designate:password@127.0.0.1/designate
+
+ sudo -u designate designate-manage database sync
+
+ systemctl enable designate-central designate-api
+ systemctl enable designate-worker designate-producer designate-mdns
+ systemctl restart designate-central designate-api
+ systemctl restart designate-worker designate-producer designate-mdns
+
+ sudo -u designate designate-manage pool update

  yum install -y wget git
  wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get github.com/gophercloud/gophercloud
  pushd ~/go/src/github.com/gophercloud/gophercloud
  go get -u ./...
  popd

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
  fi

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/github.com/gophercloud/gophercloud
      go test -v -tags "fixtures acceptance" -run "\$1" github.com/gophercloud/gophercloud/acceptance/openstack/\$2 | tee ~/gophercloud.log
      popd
    fi
  }
  EOF

  systemctl stop openstack-cinder-backup.service
  systemctl stop openstack-cinder-scheduler.service
  systemctl stop openstack-cinder-volume.service
  systemctl stop openstack-nova-cert.service
  systemctl stop openstack-nova-compute.service
  systemctl stop openstack-nova-conductor.service
  systemctl stop openstack-nova-consoleauth.service
  systemctl stop openstack-nova-novncproxy.service
  systemctl stop openstack-nova-scheduler.service
  systemctl stop neutron-dhcp-agent.service
  systemctl stop neutron-l3-agent.service
  systemctl stop neutron-lbaasv2-agent.service
  systemctl stop neutron-metadata-agent.service
  systemctl stop neutron-openvswitch-agent.service
  systemctl stop neutron-metering-agent.service
+ systemctl stop designate-central designate-api
+ systemctl stop designate-worker designate-producer designate-mdns

  mysql -e "update services set deleted_at=now(), deleted=id" cinder
  mysql -e "update services set deleted_at=now(), deleted=id" nova
  mysql -e "update compute_nodes set deleted_at=now(), deleted=id" nova
  for i in $(openstack network agent list -c ID -f value); do
    neutron agent-delete $i
  done

  systemctl stop httpd

  cp /home/centos/files/rc.local /etc
  chmod +x /etc/rc.local

There are several steps happening above:

The openstack command is used to create a new service account called designate. A catalog endpoint is also created.
A database called designate is created.
A utility called crudini is installed. This is an amazing little tool to help modify ini files on the command-line.
Designate is installed.
A bundled pools.yaml file is copied to /etc/designate. I'll show the contents of this file soon.
crudini is used to configure /etc/designate/designate.conf.
The Designate database's schema is imported using the designate-manage command.
The Designate services are enabled in systemd.
designate-manage is again used, this time to update the DNS pools.
The Designate services are added to the list of services to stop before the image/snapshot is created.

These steps roughly follow what was pulled from the Designate Installation Guide linked to earlier.

As mentioned, a pools.yaml file is copied from the files directory. Create a file called terraform-openstack-test/packer/files/pools.yaml with the following contents:

---

- name: default
  description: Default PowerDNS Pool
  attributes: {}
  ns_records:
    - hostname: ns.example.com.
      priority: 1

  nameservers:
    - host: 127.0.0.1
      port: 53

  targets:
    - type: pdns4
      description: PowerDNS4 DNS Server
      masters:
        - host: 127.0.0.1
          port: 5354

      # PowerDNS Configuration options
      options:
        host: 127.0.0.1
        port: 53
        api_endpoint: http://127.0.0.1:8081
        api_token: someapikey

Finally, modify the rc.local file:

  #!/bin/bash
  set -x

  export HOME=/root

  sleep 60

  public_ip=$(curl http://169.254.169.254/latest/meta-data/public-ipv4/)
  if [[ -n $public_ip ]]; then
    while true ; do
      mysql -e "update endpoint set url = replace(url, '127.0.0.1', '$public_ip')" keystone
      if [[ $? == 0 ]]; then
        break
      fi
      sleep 10
    done

    sed -i -e "s/127.0.0.1/$public_ip/g" /root/keystonerc_demo
    sed -i -e "s/127.0.0.1/$public_ip/g" /root/keystonerc_admin
  fi

  systemctl restart rabbitmq-server
  while [[ true ]]; do
    pgrep -f rabbit
    if [[ $? == 0 ]]; then
      break
    fi
    sleep 10
    systemctl restart rabbitmq-server
  done

  systemctl restart openstack-cinder-api.service
  systemctl restart openstack-cinder-backup.service
  systemctl restart openstack-cinder-scheduler.service
  systemctl restart openstack-cinder-volume.service
  systemctl restart openstack-nova-cert.service
  systemctl restart openstack-nova-compute.service
  systemctl restart openstack-nova-conductor.service
  systemctl restart openstack-nova-consoleauth.service
  systemctl restart openstack-nova-novncproxy.service
  systemctl restart openstack-nova-scheduler.service
  systemctl restart neutron-dhcp-agent.service
  systemctl restart neutron-l3-agent.service
  systemctl restart neutron-lbaasv2-agent.service
  systemctl restart neutron-metadata-agent.service
  systemctl restart neutron-openvswitch-agent.service
  systemctl restart neutron-metering-agent.service
  systemctl restart httpd
+ systemctl restart designate-central designate-api
+ systemctl restart designate-worker designate-producer designate-mdns
+ systemctl restart pdns

  nova-manage cell_v2 discover_hosts

+ sudo -u designate designate-manage pool update
+
+ iptables -I INPUT -p tcp --dport 9001 -j ACCEPT
+ ip6tables -I INPUT -p tcp --dport 9001 -j ACCEPT
+
  iptables -I INPUT -p tcp --dport 80 -j ACCEPT
  ip6tables -I INPUT -p tcp --dport 80 -j ACCEPT
  cp /root/keystonerc* /var/www/html
  chmod 666 /var/www/html/keystonerc*

The above steps have been added:

The Designate services have been added to the list of services to be restarted during boot.
PowerDNS is also restarted
designate-manage is again used to update the DNS pools.
Port 9001 is opened for traffic.

Build the Image and Launch

With the above in place, you can regenerate your image using Packer and then launch a virtual machine using Terraform.

When the virtual machine is up and running, you'll find that your testing environment is now running OpenStack Designate.

Conclusion

This blog post covered how to add a service to your OpenStack testing environment that is not supported by PackStack. This was done by reviewing the steps to manually install and configure the service, translating those steps to automated commands, and adding those commands to the existing deployment scripts.

Building OpenStack Environments Part 2

Sat, 07 Oct 2017 00:00:00 -0600

Introduction

In the last post, I detailed how to create an all-in-one OpenStack environment in an isolated virtual machine for the purpose of testing OpenStack-based applications.

In this post, I'll cover how to create an image from the environment. This will allow you to launch virtual machines which already have the OpenStack environment installed and running. The benefits of this approach is that it reduces the time required to build the environment as well as pins the environment to a known working version.

In addition, I'll cover how to modify the all-in-one environment so that it can be accessed remotely. This way, testing does not have to be done locally on the virtual machine.

Note: I realize the title of this series might be a misnomer. This series is is not covering how to deploy OpenStack in general, but how to set up disposable OpenStack environments for testing purposes. Blame line wrapping.

Introduction
How to Generate the Image
Creating a Reusable OpenStack Image
Using the Image
Conclusion

How to Generate the Image

AWS and OpenStack (and any other cloud provider) provide the ability to create an image (whether AMI, qcow, etc) from a running virtual machine. This is commonly known as "snapshotting".

The process described here will use snapshotting, but it's not that simple. OpenStack has a lot of moving pieces and some of those pieces are dependent on unique configurations of the host: the hostname, the IP address(es), etc. These items must be accounted for and configured correctly on the new virtual machine.

With this in mind, the process of generating an image is roughly:

Launch a virtual machine.
Install an all-in-one OpenStack environment.
Remove any unique information from the OpenStack databases.
Snapshot.
Upon creation of a new virtual machine, ensure OpenStack knows about the new unique information.

Creating a Reusable OpenStack Image

Just like in Part 1, it's best to ensure this entire process is automated. Terraform works great to provision and deploy infrastructure, but it is not suited to provide a niche task such as snapshotting.

Fortunately, there's Packer. And even more fortunate is that Packer supports a wide array of cloud services.

If you haven't used Packer before, I recommend going through the intro before proceeding here.

In Part 1, I used AWS as the cloud being deployed to. For this part, I'll switch things up and use an OpenStack cloud.

Creating a Simple Image

To begin, you can continue using the same terraform-openstack-test directory that was used in Part 1.

First, create a new directory called packer/openstack:

$ pwd
/home/jtopjian/terraform-openstack-test
$ mkdir -p packer/openstack
$ cd packer/openstack

Next, create a file called build.json with the following contents:

{
  "builders": [{
    "type": "openstack",
    "image_name": "packstack-ocata",
    "reuse_ips": true,
    "ssh_username": "centos",

    "flavor": "{{user `flavor`}}",
    "security_groups": ["{{user `secgroup`}}"],
    "source_image": "{{user `image_id`}}",
    "floating_ip_pool": "{{user `pool`}}",
    "networks": ["{{user `network_id`}}"]
  }]
}

I've broken the above into two sections: the top section has hard-coded values while the bottom section requires input on the command-line. This is because the values will vary between your OpenStack cloud and my OpenStack cloud.

With the above in place, run:

$ packer build \
    -var 'flavor=m1.large' \
    -var 'secgroup=AllowAll' \
    -var 'image_id=9abadd38-a33d-44c2-8356-b8b8ae184e04' \
    -var 'pool=public' \
    -var 'network_id=b0b12e8f-a695-480e-9dc2-3dc8ac2d55fd' \
    build.json

Note the following: the image_id must be a CentOS 7 image and the Security Group must allow traffic from your workstation to Port 22.

This command will take some time to complete. When it has finished, it will print the UUID of a newly generated image:

==> Builds finished. The artifacts of successful builds are:
--> openstack: An image was created: 53ecc829-60c0-4a87-81f4-9fc603ff2a8f

That UUID will point to an image titled "packstack-ocata".

Congratulations! You just created an image.

However, there is virtually nothing different about "packstack-ocata" and the CentOS image used to create it. All Packer did was launch a virtual machine and create a snapshot of it.

In order for Packer to make changes to the virtual machine, you must configure "provisioners" in the build.json file. Provisioners are just like Terraform's concept of provisioners: steps that will execute commands on the running virtual machine. Before you can add some provisioners to the Packer build file, you first need to generate the scripts which will be run.

Generating an Answer File

In Part 1, PackStack was used to install an all-in-one OpenStack environment. The command used was:

$ packstack --allinone

This very simple command will use a lot of sane defaults and the result will be a fully functionall all-in-one environment.

However, in order to more easily make the OpenStack environment run correctly each time a virtual machine is created, the installation needs tuned. To do this, a custom "answer file" will be used when running PackStack.

An answer file is a file which contains each configurable setting within PackStack. This file is very large with lots of options. It's not something you want to write from scratch. Instead, PackStack can generate an answer file to be used as a template.

On a CentOS 7 virtual machine, which can even be the same virtual machine you created in Part 1, run:

$ packstack --gen-answer-file=packstack-answers.txt

Copy the file to your workstation using scp or some other means. Make a directory called files to store this answer file:

$ pwd
/home/jtopjian/terraform-openstack-test
$ mkdir packer/files
$ scp -i key/id_rsa centos@<ip>:packstack-answers.txt packer/files

Once stored locally, make the following changes:

First, locate the setting CONFIG_CONTROLLER_HOST. This setting will have the value of an IP address local to the virtual machine which generated this file:

CONFIG_CONTROLLER_HOST=10.41.8.200

Do a global search and replace of 10.41.8.200 with 127.0.0.1.

Next, use this opportunity to tune which services you want to enable for your test environment. For example:

- CONFIG_HORIZON_INSTALL=y
+ CONFIG_HORIZON_INSTALL=n
- CONFIG_CEILOMETER_INSTALL=y
+ CONFIG_CEILOMETER_INSTALL=n
- CONFIG_AODH_INSTALL=y
+ CONFIG_AODH_INSTALL=n
- CONFIG_GNOCCHI_INSTALL=y
+ CONFIG_GNOCCHI_INSTALL=n
- CONFIG_LBAAS_INSTALL=n
+ CONFIG_LBAAS_INSTALL=y
- CONFIG_NEUTRON_FWAAS=n
+ CONFIG_NEUTRON_FWAAS=y

These are all services I have personally changed the status of since either they are disabled by default and I want them enabled or they are enabled by default and I do not need them. Change the values to suit your needs.

You might notice that there are several embedded passwords and secrets in this answer file. Astute readers will realize that these passwords will all be used for every virtual machine created with this answer file. For production use, this is most definitely not secure. However, I consider this relatively safe since these OpenStack environments are temporary and only for testing.

Installing OpenStack

Next, begin building a deploy.sh script. You can re-use the deploy.sh script from Part 1 as a start, with one initial change:

  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
- packstack --allinone
+ packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

  yum install -y wget git
  wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get github.com/gophercloud/gophercloud
  pushd ~/go/src/github.com/gophercloud/gophercloud
  go get -u ./...
  popd

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
  fi

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/github.com/gophercloud/gophercloud
      go test -v -tags "fixtures acceptance" -run "\$1" github.com/gophercloud/gophercloud/acceptance/openstack/\$2 | tee ~/gophercloud.log
      popd
    fi
  }
  EOF

Next, alter packer/openstack/build.json with the following:

  {
    "builders": [{
      "type": "openstack",
      "image_name": "packstack-ocata",
      "reuse_ips": true,
      "ssh_username": "centos",

      "flavor": "{{user `flavor`}}",
      "security_groups": ["{{user `secgroup`}}"],
      "source_image": "{{user `image_id`}}",
      "floating_ip_pool": "{{user `pool`}}",
      "networks": ["{{user `network_id`}}"]
-   }]
+   }],
+   "provisioners": [
+     {
+       "type": "file",
+       "source": "../files",
+       "destination": "/home/centos/files"
+     },
+     {
+       "type": "shell",
+       "inline": [
+         "sudo bash /home/centos/files/deploy.sh"
+       ]
+     }
+   ]
  }

There are two provisioners being created here: one which will copy the files directory to /home/centos/files and one to run the deploy.sh script.

files was created outside of the openstack directory because these files are not unique to OpenStack. You can use the same files to build images in other clouds. For example, create a packer/aws directory and create a similar build.json file for AWS.

With that in place, run… actually, don't run yet. I'll save you a step. While the current configuration will launch an instance, install an all-in-one OpenStack environment, and create a snapshot, OpenStack will not work correctly when you create a virtual machine based on that image.

In order for it to work correctly, there are some more modifications which need made to make sure OpenStack starts correctly on a new virtual machine.

Removing Unique Data

In order to remove the unique data of the OpenStack environment, add the following to deploy.sh:

+ hostnamectl set-hostname localhost
+
  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
  packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

  yum install -y wget git
  wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get github.com/gophercloud/gophercloud
  pushd ~/go/src/github.com/gophercloud/gophercloud
  go get -u ./...
  popd

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
  fi

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/github.com/gophercloud/gophercloud
      go test -v -tags "fixtures acceptance" -run "\$1" github.com/gophercloud/gophercloud/acceptance/openstack/\$2 | tee ~/gophercloud.log
      popd
    fi
  }
  EOF
+
+ systemctl stop openstack-cinder-backup.service
+ systemctl stop openstack-cinder-scheduler.service
+ systemctl stop openstack-cinder-volume.service
+ systemctl stop openstack-nova-cert.service
+ systemctl stop openstack-nova-compute.service
+ systemctl stop openstack-nova-conductor.service
+ systemctl stop openstack-nova-consoleauth.service
+ systemctl stop openstack-nova-novncproxy.service
+ systemctl stop openstack-nova-scheduler.service
+ systemctl stop neutron-dhcp-agent.service
+ systemctl stop neutron-l3-agent.service
+ systemctl stop neutron-lbaasv2-agent.service
+ systemctl stop neutron-metadata-agent.service
+ systemctl stop neutron-openvswitch-agent.service
+ systemctl stop neutron-metering-agent.service
+
+ mysql -e "update services set deleted_at=now(), deleted=id" cinder
+ mysql -e "update services set deleted_at=now(), deleted=id" nova
+ mysql -e "update compute_nodes set deleted_at=now(), deleted=id" nova
+ for i in $(openstack network agent list -c ID -f value); do
+   neutron agent-delete $i
+ done
+
+ systemctl stop httpd

The above added 3 pieces to deploy.sh: set the hostname to localhost, stop all OpenStack services, and then delete all known agents for Cinder, Nova, and Neutron.

Now, with the above in place, run… no, not yet, either.

Remember the last step outlined in the beginning of this post:

Upon creation of a new virtual machine, ensure OpenStack knows about the new unique information.

How is the new virtual machine going to configure itself with new information? One solution is to create an rc.local file and place it in the /etc directory during the Packer provisioning phase. This way, when the virtual machine launches, rc.local is triggered and acts as a post-boot script.

Adding an rc.local File

First, add the following to deploy.sh:

  hostnamectl set-hostname localhost

  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
  packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

  yum install -y wget git
  wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get github.com/gophercloud/gophercloud
  pushd ~/go/src/github.com/gophercloud/gophercloud
  go get -u ./...
  popd

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
  fi

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/github.com/gophercloud/gophercloud
      go test -v -tags "fixtures acceptance" -run "\$1" github.com/gophercloud/gophercloud/acceptance/openstack/\$2 | tee ~/gophercloud.log
      popd
    fi
  }
  EOF

  systemctl stop openstack-cinder-backup.service
  systemctl stop openstack-cinder-scheduler.service
  systemctl stop openstack-cinder-volume.service
  systemctl stop openstack-nova-cert.service
  systemctl stop openstack-nova-compute.service
  systemctl stop openstack-nova-conductor.service
  systemctl stop openstack-nova-consoleauth.service
  systemctl stop openstack-nova-novncproxy.service
  systemctl stop openstack-nova-scheduler.service
  systemctl stop neutron-dhcp-agent.service
  systemctl stop neutron-l3-agent.service
  systemctl stop neutron-lbaasv2-agent.service
  systemctl stop neutron-metadata-agent.service
  systemctl stop neutron-openvswitch-agent.service
  systemctl stop neutron-metering-agent.service

  mysql -e "update services set deleted_at=now(), deleted=id" cinder
  mysql -e "update services set deleted_at=now(), deleted=id" nova
  mysql -e "update compute_nodes set deleted_at=now(), deleted=id" nova
  for i in $(openstack network agent list -c ID -f value); do
    neutron agent-delete $i
  done

  systemctl stop httpd

+ cp /home/centos/files/rc.local /etc
+ chmod +x /etc/rc.local

Next, create a file called rc.local inside the packstack/files directory:

#!/bin/bash
set -x

export HOME=/root

sleep 60

systemctl restart rabbitmq-server
while [[ true ]]; do
  pgrep -f rabbit
  if [[ $? == 0 ]]; then
    break
  fi
  sleep 10
  systemctl restart rabbitmq-server
done

nova-manage cell_v2 discover_hosts

The above is pretty simple: it's simply restarting RabbitMQ and running nova-manage to re-discover itself as a compute node.

Why restart RabbitMQ? I have no idea. I've found it needs to be done for OpenStack to work correctly.

I also mentioned I'll show how to to access the OpenStack services from outside the virtual machine, so you don't have to log in to the virtual machine to run tests.

To do that, add the following to rc.local:

  #!/bin/bash
  set -x

  export HOME=/root

  sleep 60

+ public_ip=$(curl http://169.254.169.254/latest/meta-data/public-ipv4/)
+ if [[ -n $public_ip ]]; then
+   while true ; do
+     mysql -e "update endpoint set url = replace(url, '127.0.0.1', '$public_ip')" keystone
+     if [[ $? == 0 ]]; then
+       break
+     fi
+     sleep 10
+   done

+   sed -i -e "s/127.0.0.1/$public_ip/g" /root/keystonerc_demo
+   sed -i -e "s/127.0.0.1/$public_ip/g" /root/keystonerc_admin
+ fi

  systemctl restart rabbitmq-server
  while [[ true ]]; do
    pgrep -f rabbit
    if [[ $? == 0 ]]; then
      break
    fi
    sleep 10
    systemctl restart rabbitmq-server
  done

+ systemctl restart openstack-cinder-api.service
+ systemctl restart openstack-cinder-backup.service
+ systemctl restart openstack-cinder-scheduler.service
+ systemctl restart openstack-cinder-volume.service
+ systemctl restart openstack-nova-cert.service
+ systemctl restart openstack-nova-compute.service
+ systemctl restart openstack-nova-conductor.service
+ systemctl restart openstack-nova-consoleauth.service
+ systemctl restart openstack-nova-novncproxy.service
+ systemctl restart openstack-nova-scheduler.service
+ systemctl restart neutron-dhcp-agent.service
+ systemctl restart neutron-l3-agent.service
+ systemctl restart neutron-lbaasv2-agent.service
+ systemctl restart neutron-metadata-agent.service
+ systemctl restart neutron-openvswitch-agent.service
+ systemctl restart neutron-metering-agent.service
+ systemctl restart httpd

  nova-manage cell_v2 discover_hosts

+ iptables -I INPUT -p tcp --dport 80 -j ACCEPT
+ ip6tables -I INPUT -p tcp --dport 80 -j ACCEPT
+ cp /root/keystonerc* /var/www/html
+ chmod 666 /var/www/html/keystonerc*

Three steps have been added above:

The first uses cloud-init to discover the virtual machine's public IP. Once the public IP is known, the endpoint table in the keystone database is updated with it. By default, PackStack sets the endpoints of the Keystone catalog to 127.0.0.1. This prevents outside interaction of OpenStack. Changing it to the public IP resolves this issue.

The keystonerc_demo and keystonerc_admin files are also updated with the public IP.

Why not just set the public IP in the PackStack answer file? Because the public IP will not be known until the virtual machine launches, which is after PackStack has run. And that's why 127.0.0.1 was used earlier: it's an easy placeholder to search and replace with and it will still create a working OpenStack environment if it wasn't replaced.

The second stop restarts all OpenStack services so they're aware of the new endpoints.

The third step copies the keystonerc_demo and keystonerc_admin files to /var/www/html/. This way, you can wget the files from http://public-ip/keystonerc_demo and http://public-ip/keystonerc_admin and save them to your workstation. You can then source them and begin interacting with OpenStack remotely.

Now, with all of that in place, re-run Packer:

$ pwd
/home/jtopjian/terraform-openstack-test/packer/openstack
$ packer build \
    -var 'flavor=m1.large' \
    -var 'secgroup=AllowAll' \
    -var 'image_id=9abadd38-a33d-44c2-8356-b8b8ae184e04' \
    -var 'pool=public' \
    -var 'network_id=b0b12e8f-a695-480e-9dc2-3dc8ac2d55fd' \
    build.json

Using the Image

When the build is complete, you will have a new image called packstack-ocata that you can create a virtual machine with.

As an example, you can use Terraform to launch the image:

variable "key_name" {}
variable "network_id" {}

variable "pool" {
  default = "public"
}

variable "flavor" {
  default = "m1.xlarge"
}

data "openstack_images_image_v2" "packstack" {
  name        = "packstack-ocata"
  most_recent = true
}

resource "random_id" "security_group_name" {
  prefix      = "openstack_test_instance_allow_all_"
  byte_length = 8
}

resource "openstack_networking_floatingip_v2" "openstack_acc_tests" {
  pool = "${var.pool}"
}

resource "openstack_networking_secgroup_v2" "openstack_acc_tests" {
  name        = "${random_id.security_group_name.hex}"
  description = "Rules for openstack acceptance tests"
}

resource "openstack_networking_secgroup_rule_v2" "openstack_acc_tests_rule_1" {
  security_group_id = "${openstack_networking_secgroup_v2.openstack_acc_tests.id}"
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 1
  port_range_max    = 65535
  remote_ip_prefix  = "0.0.0.0/0"
}

resource "openstack_networking_secgroup_rule_v2" "openstack_acc_tests_rule_2" {
  security_group_id = "${openstack_networking_secgroup_v2.openstack_acc_tests.id}"
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 1
  port_range_max    = 65535
  remote_ip_prefix  = "::/0"
}

resource "openstack_networking_secgroup_rule_v2" "openstack_acc_tests_rule_3" {
  security_group_id = "${openstack_networking_secgroup_v2.openstack_acc_tests.id}"
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "udp"
  port_range_min    = 1
  port_range_max    = 65535
  remote_ip_prefix  = "0.0.0.0/0"
}

resource "openstack_networking_secgroup_rule_v2" "openstack_acc_tests_rule_4" {
  security_group_id = "${openstack_networking_secgroup_v2.openstack_acc_tests.id}"
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "udp"
  port_range_min    = 1
  port_range_max    = 65535
  remote_ip_prefix  = "::/0"
}

resource "openstack_networking_secgroup_rule_v2" "openstack_acc_tests_rule_5" {
  security_group_id = "${openstack_networking_secgroup_v2.openstack_acc_tests.id}"
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "icmp"
  remote_ip_prefix  = "0.0.0.0/0"
}

resource "openstack_networking_secgroup_rule_v2" "openstack_acc_tests_rule_6" {
  security_group_id = "${openstack_networking_secgroup_v2.openstack_acc_tests.id}"
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "icmp"
  remote_ip_prefix  = "::/0"
}

resource "openstack_compute_instance_v2" "openstack_acc_tests" {
  name            = "openstack_acc_tests"
  image_id        = "${data.openstack_images_image_v2.packstack.id}"
  flavor_name     = "${var.flavor}"
  key_pair        = "${var.key_name}"
  security_groups = ["${openstack_networking_secgroup_v2.openstack_acc_tests.name}"]

  network {
    uuid = "${var.network_id}"
  }
}

resource "openstack_compute_floatingip_associate_v2" "openstack_acc_tests" {
  instance_id = "${openstack_compute_instance_v2.openstack_acc_tests.id}"
  floating_ip = "${openstack_networking_floatingip_v2.openstack_acc_tests.address}"
}

resource "null_resource" "rc_files" {
  provisioner "local-exec" {
    command = <<EOF
      while true ; do
        wget http://${openstack_compute_floatingip_associate_v2.openstack_acc_tests.floating_ip}/keystonerc_demo 2> /dev/null
        if [ $? = 0 ]; then
          break
        fi
        sleep 20
      done

      wget http://${openstack_compute_floatingip_associate_v2.openstack_acc_tests.floating_ip}/keystonerc_admin
    EOF
  }
}

The above Terraform configuration will do the following:

Search for the latest image titled "openstack-ocata".
Create a floating IP.
Create a security group with a unique name and six rules to allow all TCP, UDP, and ICMP traffic.
Launch an instance using the "openstack-ocata" image.
Associate the floating IP to the instance.
Poll the instance every 20 seconds to see if http://publicip/keystonerc_demo is available. When it is available, download it, along with keystonerc_admin.

To run this Terraform configuration, do:

$ terraform apply \
    -var "key_name=<keypair name>" \
    -var "network_id=<network uuid>" \
    -var "pool=<pool name>" \
    -var "flavor=<flavor name>"

Conclusion

This blog post detailed how to create a reusable image with OpenStack Ocata already installed. This allows you to create a standard testing environment in a fraction of the time that it takes to build the environment from scratch.

Building OpenStack Environments

Sat, 30 Sep 2017 00:00:00 -0600

Introduction

I work a number of OpenStack-based projects. In order to make sure they work correctly, I need to test them against an OpenStack environment. It's usually not a good idea to test on a production environment since mistakes and bugs can cause damage to production.

So the next best option is to create an OpenStack environment strictly for testing purposes. This blog post will describe how to create such an environment.

Introduction
Where to Run OpenStack
Provisioning a Virtual Machine
- Terraform
  - Deploying to AWS
Installing OpenStack
- DevStack
- PackStack
  - Installing OpenStack with PackStack
Testing with OpenStack
Automating the Process
Conclusion

Where to Run OpenStack

The first topic of consideration is where to run OpenStack.

VirtualBox

At a minimum, you can install VirtualBox on your workstation and create a virtual machine. This is quick, easy, and free. However, you're limited to the resources of your workstation. For example, if your laptop only has 4GB of memory and two cores, OpenStack is going to run slow.

AWS

Another option is to use AWS. While AWS offers a free tier, it's restricted (I think) to the t2.micro flavor. This flavor only supports 1 vCPU, which is is usually worse than your laptop. Larger instances will cost anywhere from $0.25 to $5.00 (and up!) per hour to run. It can get expensive.

However, AWS offers "spot-instances". These are virtual machines that cost a fraction of normal virtual machines. This is possible because spot instances run on spare, unused capacity in Amazon's cloud. The catch is that your virtual machine could be deleted when a higher paying customer wants to use the space. You certainly don't want to do this for production (well, you can, and that's a fun exercise on its own), but for testing, it's perfect.

With Spot Instances, you can run an m3.xlarge flavor, which consists of 4 vCPUs and 16GB of memory, for $0.05 per hour. An afternoon of work will cost you $0.20. Well worth the cost of 4 vCPUs and 16GB of memory, in my opinion.

Spot Pricing is constantly changing. Make sure you check the current price before you begin working. And make sure you do not leave your virtual machine running indefinitely!

Other Spot Pricing Clouds

Both Google and Azure offer spot instances, however, I have not had time to try them, so I can't comment.

Your Own Cloud

The best resource is your own cloud. Maybe you already have a home lab set up or your place of $work has a cloud you can use. This way, you can have a large amount of resources available to use for free.

Provisioning a Virtual Machine

Once you have your location sorted out, you need to decide how to interact with the cloud to provision a virtual machine.

At a minimum, you can use the standard GUI or console that the cloud provides. This works, but it's a hassle to have to manually go through all settings each time you want to launch a new virtual machine. It's always best to test with a clean environment, so you will be creating and destroying virtual machines a lot. Manually setting up virtual machines will get tedious and is prone to human error. Therefore, it's better to use a tool to automate the process.

Terraform

Terraform is a tool that enables you to declaratively create infrastructure. Think of it like Puppet or Chef, but for virtual machines and virtual networks instead of files and packages.

I highly recommend Terraform for this, though I admit I am biased because I spend a lot of time contributing to the Terraform project.

Deploying to AWS

As a reference example, I'll show how to use Terraform to deploy to AWS. Before you begin, make sure you have a valid AWS account and you have gone through the Terraform intro.

There's some irony about using AWS to deploy OpenStack. However, some readers might not have access to an OpenStack cloud to deploy to. Please don't turn this into a political discussion.

On your workstation, open a terminal and make a directory:

$ pwd
/home/jtopjian
$ mkdir terraform-openstack-test
$ cd terraform-openstack-test

Next, generate an SSH key pair:

$ pwd
/home/jtopjian/terraform-openstack-test
$ mkdir key
$ cd key
$ ssh-keygen -t rsa -N '' -f id_rsa
$ cd ..

Next, create a main.tf file which will house our configuration:

$ pwd
/home/jtopjian/terraform-openstack-test
$ vi main.tf

Start by creating a key pair:

provider "aws" {
  region = "us-west-2"
}

resource "aws_key_pair" "openstack" {
  key_name   = "openstack-key"
  public_key = "${file("key/id_rsa.pub")}"
}

With that in place, run:

$ terraform init
$ terraform apply

Next, create a Security Group. This will allow traffic in and out of the virtual machine. Add the following to main.tf:

  provider "aws" {
    region = "us-west-2"
  }

  resource "aws_key_pair" "openstack" {
    key_name   = "openstack-key"
    public_key = "${file("key/id_rsa.pub")}"
  }

+ resource "aws_security_group" "openstack" {
+   name        = "openstack"
+   description = "Allow all inbound/outbound traffic"
+
+   ingress {
+     from_port   = 0
+     to_port     = 0
+     protocol    = "tcp"
+     cidr_blocks = ["0.0.0.0/0"]
+   }
+
+   ingress {
+     from_port   = 0
+     to_port     = 0
+     protocol    = "udp"
+     cidr_blocks = ["0.0.0.0/0"]
+   }
+
+   ingress {
+     from_port   = 0
+     to_port     = 0
+     protocol    = "icmp"
+     cidr_blocks = ["0.0.0.0/0"]
+   }
+ }

Note: Don't include the +. It's used to highlight what has been added to the configuration.

With that in place, run:

$ terraform plan
$ terraform apply

If you log in to your AWS console through a browser, you can see that the key pair and security group have been added to your account.

You can easily destroy and recreate these resources at-will:

$ terraform plan
$ terraform destroy
$ terraform plan
$ terraform apply
$ terraform show

Finally, create a virtual machine. Add the following to main.tf:

  provider "aws" {
    region = "us-west-2"
  }

  resource "aws_key_pair" "openstack" {
    key_name   = "openstack-key"
    public_key = "${file("key/id_rsa.pub")}"
  }

  resource "aws_security_group" "openstack" {
    name        = "openstack"
    description = "Allow all inbound/outbound traffic"

    ingress {
      from_port   = 0
      to_port     = 0
      protocol    = "tcp"
      cidr_blocks = ["0.0.0.0/0"]
    }

    ingress {
      from_port   = 0
      to_port     = 0
      protocol    = "udp"
      cidr_blocks = ["0.0.0.0/0"]
    }

    ingress {
      from_port   = 0
      to_port     = 0
      protocol    = "icmp"
      cidr_blocks = ["0.0.0.0/0"]
    }

  }

+ resource "aws_spot_instance_request" "openstack" {
+   ami = "ami-0c2aba6c"
+   spot_price = "0.0440"
+   instance_type = "m3.xlarge"
+   wait_for_fulfillment = true
+   spot_type = "one-time"
+   key_name = "${aws_key_pair.openstack.key_name}"
+
+   security_groups = ["default", "${aws_security_group.openstack.name}"]
+
+   root_block_device {
+     volume_size = 40
+     delete_on_termination = true
+   }
+
+   tags {
+     Name = "OpenStack Test Infra"
+   }
+ }
+
+ output "ip" {
+   value = "${aws_spot_instance_request.openstack.public_ip}"
+ }

Above, an aws_sport_instance_request resource was added. This will launch a Spot Instance using the parameters we specified.

It's important to note that the aws_spot_instance_request resource also takes the same parameters as the aws_instance resource.

The ami being used is the latest CentOS 7 AMI published in the us-west-2 region. You can see the list of AMIs here. Make sure you use the correct AMI for the region you're deploying to.

Notice how this resource is referencing the other resources you created (the key pair, and the security group). Additionally, notice how you're specifying a spot_price. This is helpful to limit the amount of money that will be spent on this instance. You can get an accurate price by going to the Spot Request page and clicking on "Pricing History". Again, make sure you are looking at the correct region.

An output was also added to the main.tf file. This will print out the public IP address of the AWS instance when Terraform completes.

Amazon limits the amount of spot instances you can launch at any given time. You might find that if you create, delete, and recreate a spot instance too quickly, Terraform will give you an error. This is Amazon telling you to wait. You can open a support ticket with Amazon/AWS and ask for a larger spot quota to be placed on your account. I asked for the ability to launch 5 spot instances at any given time in the us-west-1 and us-west-2 region. This took around two business days to complete.

With all of this in place, run Terraform:

$ terraform apply

When it has completed, you should see output similar to the following:

Outputs:

ip = 54.71.64.171

You should now be able to SSH to the instance:

$ ssh -i key/id_rsa centos@54.71.64.171

And there you have it! You now have access to a CentOS virtual machine to continue testing OpenStack with.

Installing OpenStack

There are numerous ways to install OpenStack. Given that the purpose of this setup is to create an easy-to-deploy OpenStack environment for testing, let's narrow our choices down to methods that can provide a simple all-in-one setup.

DevStack

DevStack provides an easy way of creating an all-in-one environment for testing. It's mainly used to test the latest OpenStack source code. Because of that, it can buggy. I've found that even when using DevStack to deploy a stable version of OpenStack, there were times when DevStack failed to complete. Given that it takes approximately two hours for DevStack to install, having the installation fail has just wasted two hours of time.

Additionally, DevStack isn't suitable to run on a virtual machine which might reboot. When testing an application that uses OpenStack, it's possible that the application causes the virtual machine to be overloaded and lock up.

So for these reasons, I won't use DevStack here. That's not to say that DevStack isn't a suitable tool – after all, it's used as the core of all OpenStack testing.

PackStack

PackStack is also able to easily install an all-in-one OpenStack environment. Rather than building OpenStack from source, it leverages RDO packages and Puppet.

PackStack is also beneficial because it will install the latest stable release of OpenStack. If you are developing an application that end-users will use, these users will most likely be using an OpenStack cloud based on a stable release.

Installing OpenStack with PackStack

The PackStack home page has all necessary instructions to get a simple environment up and running. Here are all of the steps compressed into a shell script:

#!/bin/bash

systemctl disable firewalld
systemctl stop firewalld
systemctl disable NetworkManager
systemctl stop NetworkManager
systemctl enable network
systemctl start network

yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
yum install -y centos-release-openstack-ocata
yum-config-manager --enable openstack-ocata
yum update -y
yum install -y openstack-packstack
packstack --allinone

OpenStack Pike is available at the time of this writing, however, I have not had a chance to test and verify the instructions work. Therefore, I'll be using Ocata.

Save the file as something like deploy.sh and then run it in your virtual machine:

$ sudo bash deploy.sh

Consider using a tool like tmux or screen after logging into your remote virtual machine. This will ensure the deploy.sh script continues to run, even if your connection to the virtual machine was terminated.

The process will take approximately 30 minutes to complete.

When it's finished, you'll now have a usable all-in-one environment:

$ sudo su
$ cd /root
$ source keystonerc_demo
$ openstack network list
$ openstack image list
$ openstack server create --flavor 1 --image cirros test

Testing with OpenStack

Now that OpenStack is up and running, you can begin testing with it.

Let's say you want to add a new feature to Gophercloud. First, you need to install Go:

$ yum install -y wget
$ wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
$ chmod +x /usr/local/bin/gimme
$ eval "$(/usr/local/bin/gimme 1.8)"
$ export GOPATH=$HOME/go
$ export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

To make those commands permanent, add the following to your .bashrc file:

if [[ -f /usr/local/bin/gimme ]]; then
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
fi

Next, install Gophercloud:

$ go get github.com/gophercloud/gophercloud
$ cd ~/go/src/github.com/gophercloud/gophercloud
$ go get -u ./...

In order to run Gophercloud acceptance tests, you need to have several environment variables set. These are described here.

It would be tedious to set each variable for each test or each time you log in to the virtual machine. Therefore, embed the variables into the /root/keystonerc_demo and /root/keystonerc_admin files:

source /root/keystonerc_admin
nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
_NETWORK_ID=$(openstack network show private -c id -f value)
_SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
_EXTGW_ID=$(openstack network show public -c id -f value)
_IMAGE_ID=$(openstack image show cirros -c id -f value)

echo "" >> /root/keystonerc_admin
echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

echo "" >> /root/keystonerc_demo
echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

Now try to run a test:

$ source ~/keystonerc_demo
$ cd ~/go/src/github.com/gophercloud/gophercloud
$ go test -v -tags "fixtures acceptance" -run "TestServersCreateDestroy" \
  github.com/gophercloud/gophercloud/acceptance/openstack/compute/v2

That go command is long and tedious. A shortcut would be more helpful. Add the following to ~/.bashrc:

gophercloudtest() {
  if [[ -n $1 ]] && [[ -n $2 ]]; then
    pushd  ~/go/src/github.com/gophercloud/gophercloud
    go test -v -tags "fixtures acceptance" -run "$1" github.com/gophercloud/gophercloud/acceptance/openstack/$2 | tee ~/gophercloud.log
    popd
  fi
}

You can now run tests by doing:

$ source ~/.bashrc
$ gophercloudtest TestServersCreateDestroy compute/v2

Automating the Process

There's been a lot of work done since first logging into the virtual machine and it would be a hassle to have to do it all over again. It would be better if the entire process was automated, from start to finish.

First, create a new directory on your workstation:

$ pwd
/home/jtopjian/terraform-openstack-test
$ mkdir files
$ cd files
$ vi deploy.sh

In the deploy.sh script, add the following contents:

#!/bin/bash

systemctl disable firewalld
systemctl stop firewalld
systemctl disable NetworkManager
systemctl stop NetworkManager
systemctl enable network
systemctl start network

yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-3.noarch.rpm
yum install -y centos-release-openstack-ocata
yum-config-manager --enable openstack-ocata
yum update -y
yum install -y openstack-packstack
packstack --allinone

source /root/keystonerc_admin
nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
_NETWORK_ID=$(openstack network show private -c id -f value)
_SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
_EXTGW_ID=$(openstack network show public -c id -f value)
_IMAGE_ID=$(openstack image show cirros -c id -f value)

echo "" >> /root/keystonerc_admin
echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

echo "" >> /root/keystonerc_demo
echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

yum install -y wget
wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
chmod +x /usr/local/bin/gimme
eval "$(/usr/local/bin/gimme 1.8)"
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

go get github.com/gophercloud/gophercloud
pushd ~/go/src/github.com/gophercloud/gophercloud
go get -u ./...
popd

cat >> /root/.bashrc <<EOF
if [[ -f /usr/local/bin/gimme ]]; then
  eval "\$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=\$PATH:$GOROOT/bin:\$GOPATH/bin
fi

gophercloudtest() {
  if [[ -n \$1 ]] && [[ -n \$2 ]]; then
    pushd  ~/go/src/github.com/gophercloud/gophercloud
    go test -v -tags "fixtures acceptance" -run "\$1" github.com/gophercloud/gophercloud/acceptance/openstack/\$2 | tee ~/gophercloud.log
    popd
  fi
}
EOF

Next, add the following to main.tf:

  provider "aws" {
    region = "us-west-2"
  }

  resource "aws_key_pair" "openstack" {
    key_name   = "openstack-key"
    public_key = "${file("key/id_rsa.pub")}"
  }

  resource "aws_security_group" "openstack" {
    name        = "openstack"
    description = "Allow all inbound/outbound traffic"

    ingress {
      from_port   = 0
      to_port     = 0
      protocol    = "tcp"
      cidr_blocks = ["0.0.0.0/0"]
    }

    ingress {
      from_port   = 0
      to_port     = 0
      protocol    = "udp"
      cidr_blocks = ["0.0.0.0/0"]
    }

    ingress {
      from_port   = 0
      to_port     = 0
      protocol    = "icmp"
      cidr_blocks = ["0.0.0.0/0"]
    }

  }

  resource "aws_spot_instance_request" "openstack" {
    ami = "ami-0c2aba6c"
    spot_price = "0.0440"
    instance_type = "m3.xlarge"
    wait_for_fulfillment = true
    spot_type = "one-time"
    key_name = "${aws_key_pair.openstack.key_name}"

    security_groups = ["default", "${aws_security_group.openstack.name}"]

    root_block_device {
      volume_size = 40
      delete_on_termination = true
    }

    tags {
      Name = "OpenStack Test Infra"
    }
  }

+ resource "null_resource" "openstack" {
+  connection {
+    host        = "${aws_spot_instance_request.openstack.public_ip}"
+    user        = "centos"
+    private_key = "${file("key/id_rsa")}"
+  }
+
+  provisioner "file" {
+    source      = "files"
+    destination = "/home/centos/files"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo bash /home/centos/files/deploy.sh"
+    ]
+  }
+ }

  output "ip" {
    value = "${aws_spot_instance_request.openstack.public_ip}"
  }

The above has added a null_resource. null_resource is simply an empty Terraform resource. It's commonly used to store all provisioning steps. In this case, the above null_resource is doing the following:

Configuring the connection to the virtual machine.
Copying the files directory to the virtual machine.
Remotely running the deploy.sh script.

Now when you run Terraform, once the Spot Instance has been created, Terraform will copy the files directory to it and then execute deploy.sh. Terraform will now take approximately 30-40 minutes to finish, but when it has finished, OpenStack will be up and running.

Since a new resource type has been added (null_resource), you will need to run terraform init:

$ pwd
/home/jtopjian/terraform-openstack-test
$ terraform init

Then to run everything from start to finish, do:

$ terraform destroy
$ terraform apply

When Terraform is finished, you will have a fully functional OpenStack environment suitable for testing.

Conclusion

This post detailed how to create an all-in-one OpenStack environment that is suitable for testing applications. Additionally, all configuration was recorded both in Terraform and shell scripts so the environment can be created automatically.

Granted, if you aren't creating a Go-based application, you will need to install other dependencies, but it should be easy to figure out from the example detailed here.

While this setup is a great way to easily build a testing environment, there are still other improvements that can be made. For example, instead of running PackStack each time, an AMI image can be created which already has OpenStack installed. Additionally, multi-node environments can be created for more advanced testing. These methods will be detailed in future posts.

Waffles Year End Review 2015

Fri, 01 Jan 2016 00:00:00 -0700

Introduction

2015 was different for me. Instead of working on several small projects, I focused my time on two larger projects. One of them was Waffles.

Introduction
Status of Waffles
- Waffles and Terraform
- Waffles and Shell
Future Plans
Conclusion

Status of Waffles

Waffles continues to grow and evolve and I'm having a lot of fun with it.

You can see all changes in its CHANGELOG file, but to call out some of the most notable ones:

Git Profiles: Enables profiles to be stored in a remote Git repository and downloaded upon running Waffles.
Host Files: A special profile to store files for one particular host. These files are only ever copied to that one host. This is useful for SSL certificates.
Profile Data: Data specific to a profile, but still too generic for a site can be stored in profiles/profile_name/data.sh.
Lots of new resources such as Consul, Git, Python virtualenv and pip, and more.

Waffles and Terraform

Waffles and Terraform make a great pair. Terraform handles the creation of core IaaS resources and Waffles can be used to provision the compute-related resources. Because I find both Waffles and Teraform so useful, I recently created a Waffles Terraform Provisioner.

Waffles and Shell

Waffles provides a large library of helpful Bash functions. By sourcing lib/stdlib/system.sh in your shell, you can use any of them directly on the command line or in your own shell scripts. You can even use any of the Waffles resources directly on the command line!

Future Plans

Here are some of the ideas I have for Waffles in 2016:

Better "no-op" support: I want Waffles to be more intelligent about performing a no-operation command; maybe showing the diff of a file and maybe running a syntax check against requested changes.
Publishing profiles: I've been adamant about not creating a Waffles profile / module community. I don't want to see Waffles Profiles made into large, generic scripts that account for a wide-variety of environments. I'd rather see small, agile scripts that are targeted for specific environments. However, that doesn't mean that profiles can't be published to Github for others to find and modify on their own. I plan to publish my own profiles soon.

Conclusion

I dedicated a lot of time in 2015 working on Waffles and I'm happy to say that it really paid off. I'm able to use Waffles to sketch out experimental systems as well as deploy stable, production systems very quickly. I plan to continue supporting and working on Waffles throughout 2016.

Terraform Year End Review 2015

Fri, 01 Jan 2016 00:00:00 -0700

Introduction

As I mentioned in my Waffles year end review, I spent 2015 working on two major projects. Waffles was one project and Terraform was the other.

Introduction
Terraform
Project Review

Terraform

Terraform is an amazing tool. It can easily provision and connect resources across multiple clouds and providers. If you haven't tried it, I highly encourage you to do so.

And if you use Terraform but haven't dug into its internals, I encourage you to try that, too. Terraform's code is very clean and easy to learn. It's easy to add support for new resources: from virtual machines, to MySQL or PostgreSQL databases, to SSL certificates.

The number of resources that Terraform supports continues to grow. With new resources, new ways of using Terraform and interacting with resources are discovered. It's going to be very interesting to see how Terraform evolves.

Project Review

When Terraform was first releasted, I patiently waited for it to support OpenStack. Early in 2015, I got tired of waiting and began work on my own provider based off of the last known work. Coincidentally, two other parties began similar work. Rather than have a three-way duplication, everyone convened on a single code base. In the spring of 2015, the code was merged into Terraform proper and Terraform finally had native OpenStack support. Once merged, I continued to work on the OpenStack functionality: fixing bugs, adding features, and answering questions.

This type of project was very new to me. I'm a systems administrator by trade, so working on a large software project was something I wasn't familiar with at all. I'm still not, but I think I'm getting the hang of it. It's fun to play both sides, too: from a developer point of view, I have a much better appreciation for testing; from a sysadmin point of view, I'm mindful of making sure patches don't cause backwards incompatibilities.

I plan to continue working on Terraform's OpenStack support throughout 2016. I'd like to see more OpenStack projects added to Terraform (Trove, Designate, Glance) as well as other non-OpenStack projects (such as Cobbler).

Waffles

Thu, 11 Jun 2015 00:00:00 -0600

Introduction

Waffles is a simple configuration management system written in Bash. It's small, simple, and has allowed me to easily sketch out various environments that I want to experiment with. If you want to jump right in, head over to the homepage. The rest of this article will cover the history and some personal thoughts on why I created it.

Update: I apologize if Jekyll flags this as a new post in the RSS feed. After originally publishing this article, I thought of a few other items I wanted to mention. In addition, I noticed Waffles was posted to Reddit, so I wanted to address a few of the comments made.

Introduction
Defining the Problem
Attempted Solutions
Current Status
Is Waffles Competing?
- Is Waffles Better Than X?
Conclusion

Defining the Problem

The last article I wrote, Puppet Infrastructure 2015, was quite a beast. In a way, it was a cathartic exercise to write down all of my practices to sanely work with Puppet every day. After I published the article, I couldn't get over the fact that all of those practices are just for Puppet – not any of the services Puppet sets up and configures. It didn't sit right with me, so I began to look into why I found configuration management systems so complex.

"Complex" and "Simple" are subjective terms. One core focus was the fact that Omnibus and all-in-one packages are becoming a popular way to ease the installation and configuration of the configuration management system. In my opinion, when configuring your configuration management system becomes so involved that it's easier to just have an all-in-one installation, that's "complex". I wanted to see if it was possible to escape that. I wanted to see if it was possible to create a configuration management system that was able to work out of the box on modern Linux systems and able to be installed by simply cloning a git repository.

Secondly, I wanted to see if it was possible to strip "resource models" down to more simple components. Some systems have modeled resources in a specific language, like Python, while others have chosen to use a subprocess to interact with the best native command available. Both methods have their merits, though I favor the latter more. I wanted to take that method further: why not just use a Unix shell and related tools? After all, for decades, the standard way of interacting with a Unix-based system has been through the command-line.

That's not to say that resource models in configuration management systems are the way they are for no reason. Take Puppet's Types and Providers system, for example. Decoupling the "type" and "provider" has enabled Puppet to provide a common resource interface for a variety of platforms and backends. In addition, the Types and Providers system provides the user with a way to easily manage all resources of a given type on a system, provide input validation, and a lot of other features.

I have nothing but respect for the Types and Providers system. But I still wanted to see if it was possible to create a tool that provided the same core idea (abstract a resource into a simple model that allowed for easy management) in a more simple way. It's an experiment to see what happens when a robust catalog system, input validation system, etc were removed for something more bare. Would chaos ensue?

Third, I wanted to break out of the "compiled manifest" and "workflow" views of configuration management systems. Every configuration management system has some sort of DSL, and for a good reason. You can read about the decision to use a DSL with Puppet here. I have nothing against DSLs (you can see how plain Bash evolved into the Bash-based DSL in Waffles here), but I wanted to see if it was possible to not use a compiled manifest or workflow, and instead use a plain old top-down execution sequence.

With these thoughts in mind, I set off to see what I could build.

As a real-world test to validate my solution, I created a detailed list of steps that are required to build a simple, yet robust, Memcached service (this is the reason why there are so many references to Memcached in the Waffles documentation).

You can easily get away with installing Memcached by doing:

$ sudo apt-get install -y memcached

But what if you also need to:

Edit /etc/memcached.conf
Have other common packages installed (logwatch, Postfix, fail2ban, Sensu, Nagios, etc)
Have some standard users or SSH keys installed
Have some standard firewall rules installed

You can see how the idea of a "simple" Memcached service quickly becomes a first-class service in your environment. But does that mean the configuration management system must be complex to satisfy this?

Attempted Solutions

First Version

My initial solution was written in Golang. This was because I was doing a lot of work with OpenStack and Terraform and it minimized context switching to stay with Go. I ended up with a working system that compiled shell scripts together into a master shell script "role" (my term for a fully-defined node) and referenced data stored in a hierarchical YAML system (a very watered down Hiera).

It looked like this:

roles:
  memcached:
    server_types: [lxc]
    environments: [production, testing]

The program would then search a structured directory for shell scripts that contained memcached, lxc, production, and testing and create two master scripts:

memcached_lxc_production.sh
memcached_lxc_testing.sh

I didn't like it at all. The first thing I removed was the YAML data system. I felt that it carried too much overhead. Then I decided to get rid of Go altogether. The Go component was only being used to organize shell scripts. Why not just make everything in shell?

Second Version

The next major iteration was completely written in Bash. It used Bash variables to store data and Bash arrays to store a list of scripts that should be executed. Here's what a "role" looked like:

ATTRIBUTES=(
  [environment]="production"
  [location]="honolulu"
)

RUN=(
  "site/acng.sh"
  "site/packages.sh"
  "rabbitmq"
  "rabbitmq/repo.sh"
  "rabbitmq/server.sh"
  "openstack/rabbitmq.sh"
)

The ATTRIBUTES hash stored key/value pairs that described unique features of the role. The RUN array stored a list of scripts that would be executed in a top-down order. This was working very well and it enabled me to easily deploy all sorts of environments. I wasn't totally happy with the design, but I couldn't figure out what exactly I didn't like.

Roadblock

That got put on hold when I ran into a major roadblock: I was deploying a Consul cluster when I ran into the need to use a JSON-based configuration file. What would be the best way to handle it?

Static files meant data embedded inside the file and outside of the data system
Templates meant another layer of programming logic
External languages broke the Bash-only feature

I was stumped for a few days until I realized: Augeas! I've always had a lot of respect for the Augeas project but never had a reason to use it – until now. Even better, Augeas was able to cleanly parse and write JSON files. So I made Augeas an optional, but encouraged, component of Waffles.

Third Version

Now back to figuring out why I didn't like the current iteration. I realized I didn't like the format of the "role". I wanted the "role" to look more like a shell script and not like a metadata file that describes a system.

So I made some changes and was happier. The above RabbitMQ role now looked like:

stdlib.data common
stdlib.data openstack/rabbitmq

stdlib.profile site/acng
stdlib.profile site/packages
stdlib.module rabbitmq
stdlib.module rabbitmq/repo
stdlib.module rabbitmq/server
stdlib.module openstack/rabbitmq

Fourth and Final Version

There was still one part that bugged me: modules. In the above example, rabbitmq and openstack were both modules. I didn't like how I had profiles and modules mixed in the role. I refactored the above so that profiles were just an abstraction of modules, like in Puppet, but I didn't like that either. Finally, I decided to get rid of modules altogether. You can read more about this decision here.

So modules went away and the above turned into:

stdlib.data common
stdlib.data openstack/rabbitmq

stdlib.profile common/acng
stdlib.profile common/packages
stdlib.profile rabbitmq/repo
stdlib.profile rabbitmq/server
stdlib.profile openstack/rabbitmq

At that point I was very satisfied with how it looked. I used it for a week or two and was still happy and decided to finally release it publicly.

Current Status

Now I have a tool that I find fits my definition of "simple" and "intuitive".

But Waffles is nowhere near complete. The only milestone that has been reached is that I'm happy with its core design. Waffles certainly isn't at feature parity with other configuration management systems. For some features, this is only a matter of time. For other features, they just aren't applicable to Waffles's core ideas. For example, I want to have some kind of pull-based mechanism so clients can contact a central Waffles server. At the same time, I don't feel a PuppetDB or Etcd database belongs in Waffles.

Is Waffles Competing?

Waffles is just another tool. I have no intention to market Waffles as a competitor to other configuration management systems. The only reason Puppet has been called out throughout this article is because it's the system that I'm most knowledgeable with.

Is Waffles Better Than X?

Again, Waffles is just another tool that does a certain action in a certain way. The only thing that matters is what tool you find most useful. I have nothing but respect for every other configuration management system available today. If it wasn't for those projects, I wouldn't have been able to learn what I like and don't like in order to create my own.

As mentioned above, Puppet is called out a lot because it's the system I'm most knowledgeable with. Puppet is a great tool and I still use it everyday. I've also spent time with Chef, Ansible, Juju, and Salt and find them to be great tools, too.

I will make one comment about Ansible, though, because comparisons will be made (I'm assuming because Ansible is known to be a small and simple configuration management system): I feel Waffles and Ansible are two separate tools that do things very differently. I feel the biggest difference is that Ansible is a workflow-based system while Waffles is just a bunch of shell functions that execute from top to bottom.

I take no offence if you don't like Waffles. You can see from a lot of the other tools and code I've created that I do things very differently sometimes – that's just me.

Conclusion

This article gave the history and thought process behind a new project of mine: Waffles. Try it out and tell your friends. Feedback is welcome!

Puppet Infrastructure 2015

Thu, 01 Jan 2015 00:00:00 -0700

Preface

My previous article, Puppet Infrastructure was written back in 2013. It's one of the most popular articles on my blog and seems to have helped a good number of people (as well as draw criticism from others).

With it being 2015 and all, I thought I'd write an updated version and include some notes and thoughts from using Puppet throughout 2014.

Preface
Introduction
Prep-work
Install and configure PuppetDB
Setting Up Version Control
- What About Environments?
Building the Site Module
Committing Everything
Conclusion
Resources

Introduction

This tutorial will explain how to create a new Puppet environment using best practices such as version control, a site-local module, and roles & profiles. It will also include some practices that aren't considered "best", but have helped my sanity while using Puppet in production on a daily basis. It assumes the reader will be using Ubuntu. The instructions were verified with Ubuntu 14.04.

Note that this is not an intro to Puppet – this tutorial assumes that you have at least a beginner's knowledge of Puppet.

Prep-work

The Puppet Labs Package Repo

Install the Puppet Labs apt repo:

wget http://apt.puppetlabs.com/puppetlabs-release-trusty.deb
dpkg -i *.deb
rm *.deb
apt-get update

Install and Configure Puppet Server

This tutorial will use Puppet Server – the next generation of Puppet Master. You can read more about it here and here.

To install it, do:

apt-get install -y puppetserver

I haven't used Puppet Server extensively yet, but it's worked fairly well so far. I plan to use it throughout 2015 and if I run into any major issues, I'll update this tutorial.

Configure Puppet Server

By default, Puppet Server requires 2gb of RAM. If your server does not have 2gb, you can lower the amount by editing /etc/default/puppetserver:

sed -i -e 's/JAVA_ARGS="-Xms2g -Xmx2g -XX:MaxPermSize=256m"/JAVA_ARGS="-Xms1g -Xmx1g -XX:MaxPermSize=256m"/' /etc/default/puppetserver

Additionally, I configure my Puppet servers with the following two settings:

The Future Parser

puppet config set --section main parser future

I've been using the future parser since it was first announced in Puppet 3.2. In my opinion, iteration was a sorely missing feature in Puppet and I have abused the hell out of it since it became available.

However cool the future parser may be, keep in mind that the features it provides are not official. Future releases could remove them, or worse, break them and thus break your environment. This actually happened to me with the Puppet 3.5 release and I spent the afternoon recovering from an almost catastrophic puppet run.

Manifest Ordering

puppet config set --section main ordering manifest

I've included a list of great resources at the end of this article. One of them covers the "manifest ordering" feature in detail. It's a great read and I agree with its principles on why you shouldn't use it.

When I first started using Puppet, I hated dependency-based ordering, but since there was no other option, I had to live with it and eventually it became second nature.

However, if I could get back the ridiculous amount of hours I've spent trying to resolve ordering issues, I'd have an awesome vacation. Dependency-based ordering is great in theory, but unless you write all of your Puppet manifests and modules yourself and are very strict to follow dependency conventions, it's very difficult to get right in practice. Because of this, I've begun using manifest ordering.

I understand that by using manifest ordering, I'm not helping solve the larger issue of fixing dependency problems when I find them in others' modules. I wish I had more time to do that, but unfortunately I'm not paid to work on Puppet all day – sometimes I need to get back to work.

And to play devil's advocate: really, where else am I going to have to use a dependency-based system? Everything else I use executes in a top-down format. And has for loops.

Generate an SSL Certificate

Since the Puppet Server has just been installed, it hasn't generated an SSL cert yet. It will do this the first time you start the service, but the cert will be needed in the following step. To generate the cert, do:

puppet cert generate $(puppet config print certname)

Install and configure PuppetDB

PuppetDB is a complementary service to Puppet. It's not required, but it's very useful. Unfortunately, this tutorial will only cover the installation of PuppetDB and not what makes it so useful.

To install and configure it:

cd /etc/puppet/modules
puppet module install puppetlabs/puppetdb
apt-get -y install puppetdb
cd /root
echo include puppetdb > pdb.pp
echo include puppetdb::master::config >> pdb.pp
puppet apply --verbose pdb.pp
rm pdb.pp
rm -rf /etc/puppet/modules/*

Note that the certificate generated in the previous step must exist for PuppetDB to install cleanly.

Setting Up Version Control

It's a best practice to keep all the configuration management information in a version control provider such as git. Since the repository will probably contain sensitive information about your environment, it's recommended to use an internal git server. Managing one is very easy by using gitolite.

Seriously. Do not use Github, or any other public git host, if your repository will contain sensitive information. And if you accidentally check-in sensitive information, delete the entire repository immediately. Don't just delete the sensitive information and commit the changes. It sounds like common sense, I know, but it happens. Also consider using some type of encryption on your repo like hiera-gpg or blackbox.

Some people keep the entire /etc/puppet directory in the repository. There's nothing wrong with this and if this is what you'd like to do, the following should work:

cd /etc/puppet
git init
git add auth.conf fileserver.conf puppet.conf manifests/ modules/
git commit -m "Initial commit"
git remote add origin <remote repo location>
git push -u origin master

Another way of keeping all Puppet configuration in a repository is to create a module that will only be used for the particular Puppet environment being created. This method will be used for this tutorial.

Begin by creating a module:

cd /etc/puppet/modules
mkdir site
cd site
mkdir manifests ext data
touch data/.keep
touch manifests/.keep
cd ext
touch site.pp
ln -s /etc/puppet/modules/site/ext/site.pp /etc/puppet/manifests/site.pp
cd ..
git init
git add .
git commit -m "Initial commit"
git remote add origin <remote repo location>

Three directories were created inside the site module:

manifests: where your site-local manifests will go
data: where your Hiera data will go
ext: where your "extra" files will go. These files are complementary or supplementary to your environment as well as the main site.pp file.

What About Environments?

I experimented with environments throughout 2013 and 2014 and ultimately decided to not use them. By not using environments, my production sites now have multiple Puppet servers deployed: one for each project or domain of responsibility. If I used environments effectively, all of those Puppet servers could be combined into a single server.

However, I found that using environments in production caused some issues:

If the location that housed the main Puppet server was down (lost power, etc), no nodes anywhere could talk to Puppet.
If the filesystem on the central Puppet server became corrupt, it could affect all nodes. This happened to me in 2014, but the damage was restricted to only one project.
A single main Puppet server would require all projects to work on the same version of Puppet. This is not possible for some projects.
Similarly, upgrading Puppet means upgrading across the entire "federation".
Having to type extra characters and tabs to reach the environment got tedious (cd /etc/puppet/env<tab>/project-name<tab>/mod<tab>/site). Though this was not a significant reason, it was a sense of relief to just go back to /etc/puppet/mod<tab>/site.

With regard to development and testing deployments, it's way too easy for me to use Vagrant to fire up a small Puppet server than to configure a central Puppet server with environments.

This isn't to say that environments are a useless feature. I've just found that they haven't worked well for my specific use-cases.

Building the Site Module

At this point, Puppet is installed and an empty site module exists. Now we'll begin using Puppet to configure the Puppet server itself as well as any other server you place under Puppet control.

NTP

Things begin to break when two servers with skewed times try communicating with each other. To ensure this doesn't happen, NTP will be installed and configured. First, install the puppetlabs/ntp Puppet module:

cd /etc/puppet/modules
puppet module install puppetlabs/ntp

A Note About the Module Subcommand

The puppet command has a built-in subcommand to install modules. It's able to find the module by looking it up at the Forge.

There are pros and cons to this. On one hand, it provides an easy way to install a module plus any other modules it depends on. On the other hand, if you install a module that has a conflicting dependency with another module, the command will break. Additionally, sometimes the version of the module hosted at the Forge is outdated. When this happens, you need to manually download the module from its designated home – usually github.

I usually clone the modules directly from Github. The puppet module command was included in this tutorial as an example.

Keeping Track of Modules

The previous version of this article showed how to create a bash script that will re-download all modules you use. There's nothing wrong with that method and it works great.

The previous version also mentioned librarian-puppet. I tried to use it, but found its dependency resolution and module metadata checks to be way too strict.

Then I started using r10k which you can read about here. It's another great tool, but it didn't make a lot of sense to keep using it once I stopped using environments.

Dan Bode has a tool called librarian-puppet-simple that is a stripped down version of librarian-puppet. It simply installs a set of modules that you list in a Puppetfile – no dependency or metadata checks. Like Dan, librarian-puppet-simple is really awesome, and you should check it out. This is what I've been using for quite a while now and don't see a reason to stop.

Configuring NTP

Now that the puppetlabs/ntp module is installed, it can be used to install and configure NTP on any server under Puppet control. The puppetlabs/ntp module is a simple module and rarely needs any parameters.

In the site.pp file, add the following:

node 'puppet.example.com' {
  class { '::ntp': }
}

Roles and Profiles

There's an issue with this, though. This class will need applied to every server:

node 'puppet.example.com' {
  class { '::ntp': }
}

node 'www.example.com' {
  class { '::ntp': }
}

node 'db.example.com' {
  class { '::ntp': }
}

There's a lot of repeated configuration here and it'll only get worse as more modules are added. A better way to apply modules to nodes is to use the "Roles and Profiles" pattern. The end of this article has some links that will describe this pattern in detail.

A good profile to start with is the "base" profile. This profile will be applied to all servers, so it's important that this profile contains very generic and global settings. To start, create a new manifest called /etc/puppet/modules/site/manifests/profiles/base.pp with the following contents:

class site::profiles::base {
  class { '::ntp': }
}

Next, create a role:

class site::roles::puppet_server {
  contain site::profiles::base
}

Finally, apply the role to the node:

node 'puppet.example.com' {
  contain site::roles::puppet_server
}

With only the ntp module being used in site::profiles::base, this actually seems more complicated. To better show the usefulness of profiles, add the following to the base profile:

$packages = ['git', 'vim']
package { $packages: ensure => latest }

Before, you would have had to add those two lines to each node. Now you just add it to one profile and it gets applied to every node that has the base profile applied to.

All Your Base

Since my "base" settings are so common across different Puppet-controlled environments, I have started creating an actual "base" module called bass. I haven't yet decided to pronounce it bass as in "base" or bass as in the fish.

Class or Include or Contain?

There's a lot of great documentation on this subject that is written better than I ever could. See the end of this article for links. Once you've read everything and understand the history between these three keywords, here's my $0.02:

I use contain in my roles and nodes. This is because I enforce a no-parameter policy in roles and nodes.
I use class in my profiles since those do use parameters. I still sometimes use Anchors and explicit ordering, but I'm finding that these are not needed (as much) in Puppet 3.7+ and Puppet Server and by using manifest ordering.
Sometimes I'll throw a contain or include in the profiles, though, if I'm positive that I'll never need to add parameters to them and that their ordering is stable.

The First Puppet Run

At this point, Puppet can be run for the first time. If all goes well, NTP will be installed and running when Puppet has finished:

puppet apply --verbose /etc/puppet/manifests/site.pp

Configuring the Firewall

Next, the puppetlabs/firewall module will be used to build the basis of a deny-by-default firewall.

Install the puppetlabs/firewall module by cloning it from github:

cd /etc/puppet/modules
git clone https://github.com/puppetlabs/puppetlabs-firewall firewall

Add it to your Puppetfile, too:

mod 'firewall',
  :git => 'https://github.com/puppetlabs/puppetlabs-firewall'

Next, a new manifest called /etc/puppet/modules/site/manifests/firewall.pp will be created:

class site::firewall {
  firewall { '000 accept all icmp':
    proto  => 'icmp',
    action => 'accept',
  }

  firewall { '001 accept all to lo interface':
    proto   => 'all',
    iniface => 'lo',
    action  => 'accept',
  }

  firewall { '002 accept related established rules':
    proto   => 'all',
    ctstate => ['RELATED', 'ESTABLISHED'],
    action  => 'accept',
  }

  firewall { '999 drop all':
    proto  => 'all',
    action => 'drop',
  }
}

Next, add the following to the site::profiles::base profile, before the base packages are applied:

class { '::firewall': }
class { '::site::firewall': }

Now all servers will have a deny-by-default firewall applied. I wouldn't recommend applying this configuration yet because you'll be locked out if you're working on this server remotely.

Hiera and the Firewall

This is a good place to introduce Hiera - a tool to store structured configuration data outside of Puppet manifests. Hiera is installed by default with the Puppet package, so installing a hiera package is not needed. However, to utilize Hiera's merging feature, the deep_merge gem needs to be installed:

gem install deep_merge

To configure Hiera, create /etc/puppet/modules/site/ext/hiera.yaml with the following contents:

---
:backends:
  - yaml

:hierarchy:
  - "common"

:yaml:
  :datadir: /etc/puppet/modules/site/data

:merge_behavior: deeper

Next, link this configuration file to two locations:

ln -s /etc/puppet/modules/site/ext/hiera.yaml /etc/
ln -s /etc/puppet/modules/site/ext/hiera.yaml /etc/puppet

The first location is so you can use the hiera command-line tool. The second location is for Puppet itself.

At the moment, the only hierarchy configured is common. This means that Hiera will only read data from a single file: /etc/puppet/modules/site/data/common.yaml.

Add the following to that file:

---
trusted_networks:
  - '192.168.1.0/24'
  - '10.255.0.0/24'

Add any other networks or hosts (/32) to this list that you need.

Next, add the following before the 999 rule in site::firewall:

$trusted_networks = hiera_array('trusted_networks')
$trusted_networks.each |$network| {
  firewall { "003 allow all traffic from ${network}":
    proto  => 'all',
    source => $network,
    action => 'accept',
  }
}

This block of Puppet code uses Puppet's iteration feature from the future parser, so you'll need to make sure you have it enabled in puppet.conf.

Now the next time you apply the Puppet configuration, a deny-by-default firewall will be enabled with explicit allow rules for each trusted network you specified in Hiera.

My Hiera Structure

Over the past year+, I have standardized on the following Hiera hierarchy:

:hierarchy:
  - "node/%{::hostname}"
  - "location/%{::location}"
  - "role/%{::role}"
  - common

"node" is the hostname (or fqdn) of the node. While the idea of having node-specific settings goes against the "Pets v Cattle" argument, it's sometimes unavoidable. In my production environments, only a small percentage of nodes have their own node-specific settings, and even then it's maybe one or two values.
"location" is an arbitrary fact to group nodes:

echo location=honolulu > /etc/facter/facts.d/location.txt
echo location=maui > /etc/facter/facts.d/location.txt
echo location=dc1 > /etc/facter/facts.d/location.txt

"role" is another fact that matches the role that the node will have applied:

echo role=puppet_server > /etc/facter/facts.d/role.txt

Be careful about using Facter to categorize nodes on the node itself! Let's say a node with a role of dns was compromised and the intruder understood that they could change the role to mysql by replacing the fact in /etc/facter/facts.d/role.txt. On the next Puppet run, MySQL will be installed, which would apply sensitive information to the node that the intruder now has access to. It might also break your DNS server.

What Goes in Hiera and What Uses Hiera?

My personal rules of Hiera data are:

Hiera data is used only in profiles. If I write my own module that will be located in /etc/puppet/modules, I still use class parameters.
Since I only use Hiera in profiles, that means all Hiera data is site-specific. So the deciding question becomes: "what information needs stripped from this module that will allow it to work in another environment?" That data is then moved to Hiera.

Finishing up the Puppet Server Role

Up until now, broad configurations that could be applied to any node have been used. Now we'll create a more specific role and profile to configure the Puppet Server.

In order to do this, several modules will be needed:

mod 'concat',
  :git => 'https://github.com/puppetlabs/puppetlabs-concat',
  :ref => '1.1.2'

mod 'inifile',
  :git => 'https://github.com/puppetlabs/puppetlabs-inifile',
  :ref => '792d35cdb48fc2cba08ab578c1b7bc42ef3a0ace'

mod 'puppet',
  :git => 'https://github.com/jtopjian/puppet-puppet',
  :ref => 'puppetserver'

mod 'puppetdb',
  :git => 'https://github.com/puppetlabs/puppetlabs-puppetdb',
  :ref => '4.1.0'

mod 'postgresql',
  :git => 'https://github.com/puppetlabs/puppetlabs-postgresql',
  :ref => '4.1.0'

mod 'stdlib',
  :git => 'https://github.com/puppetlabs/puppetlabs-stdlib',
  :ref => '4.4.x'

Ref?

In production, I mark all modules in my Puppetfile with a reference. This reference is the known working release or commit of that module. This means that if I ever want to test an updated module, I'll need to actually create a test environment, rebuild everything, and confirm it works. But the alternative of just going cowboy and deploying all new releases to the production environment will only cause a lot of pain (and downtime).

Creating the Puppet Server Profile

Create /etc/puppet/modules/site/manifests/profiles/puppet/server.pp with the following contents:

class site::profiles::puppet::server {

  # Hiera
  $main_settings           = hiera('site::puppet::settings::main')
  $agent_settings          = hiera('site::puppet::settings::agent')
  $master_settings         = hiera('site::puppet::settings::master')
  $server_default_settings = hiera('site::puppet::settings::server_default')
  $puppet_package_ensure   = hiera('site::puppet::puppet_package_ensure')
  $server_package_ensure   = hiera('site::puppet::server_package_ensure')

  # Resources
  class { '::puppet':
    server                  => true,
    main_settings           => $main_settings,
    agent_settings          => $agent_settings,
    master_settings         => $master_settings,
    server_default_settings => $server_default_settings,
    puppet_package_ensure   => $server_package_ensure,
  }

  class { 'puppetdb': }
  class { 'puppetdb::master::config': }

}

This example profile is much more fleshed out than the previous site::profiles::base profile to show the format I'm currently using in my production profiles.

In the common.yaml Hiera file, add this:

# Puppet
site::puppet::puppet_package_ensure: 'latest'
site::puppet::server_package_ensure: 'latest'
site::puppet::settings::main:
  server: 'puppet'
  parser: 'future'
  ordering: 'manifest'
  pluginsync: true
  logdir: '/var/log/puppet'
  vardir: '/var/lib/puppet'
  ssldir: '/var/lib/puppet/ssl'
  rundir: '/var/run/puppet'
site::puppet::settings::agent:
  certname: "%{::fqdn}"
  show_diff: true
  splay: false
  configtimeout: 360
  usecacheonfailure: true
  report: true
  environment: "%{::environment}"
site::puppet::settings::server_default:
  JAVA_ARGS: '-Xms1g -Xmx1g -XX:MaxPermSize=256m'
site::puppet::settings::master:
  ca: true
  ssldir: '/var/lib/puppet/ssl'
puppetdb::master::config::restart_puppet: false

You can see how each Hiera item matches the corresponding section in the profile except for puppetdb::master::config::restart_puppet. This is because puppetdb::master::config::restart_puppet is an Automatic Paramter Lookup. Declaring this in Hiera is the same as if you did:

class { 'puppetdb::master::config':
  restart_puppet => false,
}

Now create the /etc/puppet/modules/site/manifests/profiles/puppet/agent.pp profile:

class site::profiles::puppet::agent {

  # Hiera
  $main_settings         = hiera('site::puppet::settings::main')
  $agent_settings        = hiera('site::puppet::settings::agent')
  $puppet_package_ensure = hiera('site::puppet::puppet_package_ensure')

  # Resources
  class { '::puppet':
    main_settings         => $main_settings,
    agent_settings        => $agent_settings,
    puppet_package_ensure => $server_package_ensure,
  }

}

Now build a role for the Puppet server:

class site::roles::puppet_server {
  contain site::profiles::base
  contain site::profiles::puppet::server
}

With all of this in place, run Puppet:

puppet apply --verbose /etc/puppet/manifests/site.pp

When everything has finished, you should now be able to switch to using puppet agent instead of puppet apply:

puppet agent -t --noop

My Opinion on Automatic Parameter Lookups

I think they're a great idea, but ulimately they're too "magical" and unintuitive. There's no easy way to tell if they're being used by reading the Puppet manifests – you have to read both the manifests and Hiera data and correlate the two data sources.

If there are any automatic lookups in my Hiera data, it's because I got lazy. It happens.

Committing Everything

A lot of work has been done here. To see all of the changes that were made, do the following:

cd /etc/puppet/modules/site
git status

All of this should be committed into git:

git add .
git commit -m "Created base profile, puppet server profile and role, configured hiera."
git push -u origin master

Conclusion

This tutorial was an updated version of my 2013 Puppet Infrastructure tutorial. It described how to install and configure Puppet Server, PuppetDB, and Hiera as well as how to lay the foundation of a maintainable Puppet environment.

In addition, I've included notes of my experiences learned over the past year with Puppet.

Resources

And as mentioned throughout this article:

Gary Larizza's blog. Read everything here. Twice.
Designing Puppet - Roles and Profiles
Puppet Roles and Profiles with a Simple Module Structure (part 2)
Roles and Profiles slidedeck
Puppet Containment
Puppet Automatic Parameters Lookup

Masterless Puppet with Vagrant

Fri, 25 Apr 2014 00:00:00 -0600

Introduction

In a previous article, I described how to create a Masterless Puppet workflow with Capistrano. In this article, I'll show how you can use Vagrant for Masterless Puppet.

Since Capistrano works over SSH, it can be used to control any type of server – from bare metal to LXC containers. However, the workflow can be a bit complex. Vagrant provides a simpler workflow, but you're limited to the server providers compatible with Vagrant. Pros and Cons.

Introduction
Prerequisites and Dependencies
Vagrant's Puppet Provisioners
Configuring Vagrant
Vagrant Up
Conclusion

Prerequisites and Dependencies

For this article, I'm using Vagrant 1.5.2 and vagrant-openstack-plugin to enable Vagrant to launch virtual machines inside OpenStack. The virtual machine that Vagrant launches will be running Ubuntu 14.04.

Despite these choices, though, the methods described in this article should be applicable to any Vagrant-based server as long as Vagrant's Puppet provisioner is compatible with it.

Vagrant's Puppet Provisioners

Out of the box, Vagrant supports provisioning servers with both Puppet Apply and Puppet Agent. Since this is a Masterless Puppet workflow, I'm going to be using Puppet Apply.

Vagrant's documentation does an excellent job at describing how to use the Puppet Apply provisioner. The only gotcha I found is that Puppet must be installed on the Vagrant-based virtual machine before Puppet Apply can be used. Maybe I just missed that bit in the documentation.

Configuring Vagrant

For this article, I'll create a simple Vagrant environment that will create a virtual machine and have Apache installed onto it. The virtual machine will be launched inside OpenStack, have Puppet installed via the Shell provisioner, and have Apache installed by the Puppet Apply provisioner along with Hiera and the Puppetlabs Apache module.

Vagrantfile

Below is the completed Vagrantfile:

require 'vagrant-openstack-plugin'

Vagrant.configure("2") do |config|
  # "dummy" box because we're using Glance
  config.vm.box = "dummy"

  # SSH
  config.ssh.private_key_path = "/path/to/id_rsa"

  # Basic OpenStack options
  # Note that an openrc file needs sourced before using
  config.vm.provider :openstack do |os|
    os.username        = ENV['OS_USERNAME']
    os.api_key         = ENV['OS_PASSWORD']
    os.tenant          = ENV['OS_TENANT_NAME']
    os.flavor          = /m1.tiny/
    os.image           = 'cc4a5014-1a99-4d65-811a-8c0184b15dd7'
    os.endpoint        = "#{ENV['OS_AUTH_URL']}/tokens"
    os.keypair_name    = "home"
    os.ssh_username    = "ubuntu"
    os.security_groups = ["default"]
  end

  config.vm.define "p.example.com"

  config.vm.provision "shell", :inline => <<-SHELL
    apt-get update
    apt-get install -y puppet
  SHELL

  config.vm.provision "puppet" do |puppet|
    puppet.hiera_config_path = "hiera.yaml"
    puppet.module_path = "modules"
  end

end

Hiera

The Vagrantfile defines a Hiera configuration called hiera.yaml located in the same directory as the Vagrantfile.

hiera.yaml

Here's the contents of hiera.yaml:

---
:backends:
  - yaml

:hierarchy:
  - "common"

:yaml:
  :datadir: '/vagrant/hiera'

This tells hiera to look for a file called common.yaml in the /vagrant/hiera directory on the newly provisioned virtual machine. On the Vagrant-side, everything is in the same directory as the Vagrantfile.

common.yaml

The contents of common.yaml will simply be:

---
apache::serveradmin: 'joe@example.com'
my_message: 'Hello, World!'

The first line sets the serveradmin setting for Apache and the second line sets an arbitrary message for later use.

Modules

The Vagrant Puppet Apply provisioner supports hosting modules. In the Vagrantfile, the module path is configured as ./modules.

To install and configure Apache via Puppet, I'm going to use the puppetlabs/apache module. This module requires the puppetlabs/stdlib and puppetlabs/concat modules, so in total, three modules will be used:

$ mkdir modules
$ cd modules
$ git clone https://github.com/puppetlabs/puppetlabs-apache apache
$ git clone https://github.com/puppetlabs/puppetlabs-stdlib stdlib
$ git clone https://github.com/puppetlabs/puppetlabs-concat concat

default.pp

The final component is an actual Puppet manifest. By default, Vagrant looks for ./manifests/default.pp which is exactly what I'll use:

include apache
notify { 'my_message':
  message => hiera('my_message'),
}

The first line installs and configures apache using the default settings from the puppetlabs/apache module except for the serveradmin setting which was defined in Hiera as joe@example.com.

The second line prints a message stored in Hiera just as another way to show that data is successfully being pulled from Hiera.

The Final Directory Structure

With everything in place, the directory structure should look like this:

$ tree -L 2
.
├── Vagrantfile
├── hiera
│   └── common.yaml
├── hiera.yaml
├── manifests
│   └── default.pp
└── modules
    ├── apache
    ├── concat
    └── stdlib

Vagrant Up

Now simply run vagrant up and watch Vagrant go to work!

If you're using a provider other than VirtualBox, like the OpenStack provider used in this article, don't forget to specify it on the command line:

$ vagrant up --provider=openstack

Conclusion

This article showed how Vagrant can provide a Masterless Puppet workflow. Besides the non-default OpenStack provider used, all other features shown in this article are native to Vagrant. This makes it very easy and accessible to use Puppet within Vagrant. The downside, though, is that the servers you can provision in this way are limited to the providers available for Vagrant. If this is a blocker for you, consider using something like Capistrano and building your own Masterless Puppet workflow.

Building an LXC Server - Ubuntu 14.04 Edition

Sat, 19 Apr 2014 00:00:00 -0600

Introduction

This article is a basic step-by-step HOWTO to create a server capable of hosting LXC-based containers.

Introduction
Prerequisites and Dependencies
Install LXC
Configure LXC
- Back to ZFS
Using LXC
- Creating a Container
- Testing ZFS Deduplication
Port Forwarding
- lxc-nat
Conclusion

Prerequisites and Dependencies

This server will be using Ubuntu 14.04. As 14.04 has just been released, some steps might change in the future.

apt Update

First, make sure all of the base packages are up to date:

$ sudo apt-get update
$ sudo apt-get dist-upgrade

Open vSwitch

The previous version of this article advocated Open vSwitch. I have since stopped using OVS as I've been able to configure Linux Bridge with the exact same features by using newer kernels.

ZFS

The newer LXC builds support ZFS as a backing store. This means that deduplication, compression, and snapshotting can all be taken advantage of. To install ZFS on Ubuntu 14.04, do:

$ sudo apt-add-repository ppa:zfs-native/daily
$ sudo apt-get update
$ sudo apt-get install ubuntu-zfs

Install LXC

Ubuntu 14.04 provides LXC 1.0.3, which is the latest version. I'm not sure if Ubuntu 14.04 will continue providing up-to-date versions of LXC, given it being an LTS release. If it falls behind, it might be beneficial to switch to the ubuntu-lxc/daily ppa.

To install LXC, just do:

$ sudo apt-get install lxc

Configure LXC

Back to ZFS

By default, LXC will look for a zpool titled lxc:

$ sudo zpool create -f tank /dev/vdc

Make sure deduplication and compression are turned on:

$ sudo zfs set dedup=on tank
$ sudo zfs set compression=on tank

LXC can use ZFS's native snapshot features. To make sure you can see snapshots when running zfs list, do the following:

$ sudo zpool set listsnapshots=on tank

To configure LXC to use ZFS as the backing store and set the default LXC path, add the following to /etc/lxc/lxc.conf:

lxc.lxcpath = /tank/lxc/containers
lxc.bdev.zfs.root = tank/lxc/containers

Ensure /tank/lxc/containers, or whichever path you choose, exists:

$ sudo zfs create tank/lxc
$ sudo zfs create tank/lxc/containers

Using LXC

Creating a Container

Create the first container by doing:

$ sudo lxc-create -t ubuntu -n test1 -B zfs -- -S /root/.ssh/id_rsa.pub

When the command has finished, you'll see that LXC has created a new ZFS partition:

$ sudo zfs list
$ df -h

Testing ZFS Deduplication

You can see the ZFS dedup stat by doing:

$ sudo zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank  99.5G   186M  99.3G     0%  1.01x  ONLINE  -

With that number in mind, create a second container:

$ sudo lxc-create -t ubuntu -n test2 -B zfs -- -S /root/.ssh/id_rsa.pub

When the command has finished, review the ZFS stat:

$ sudo zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank  99.5G   187M  99.3G     0%  2.02x  ONLINE  -

The dedup ratio doubled. This effectively means that no new disk space was consumed when the new container was created!

Port Forwarding

By default, LXC uses the veth networking mode for containers. This is the most robust networking mode. Other modes exist and I highly recommend this article for a detailed look at them.

veth mode can be thought of as a form of NAT and the LXC server is now acting as a NAT'd gateway for all of the containers running on the server. If you want the containers to be accessible from the public internet, you will need to do some port forwarding.

lxc-nat

Update: Zan Loy has made a much better version of the lxc-nat script mentioned below. The improved version can be found here.

Update 2: Daniël created a Python version of lxc-nat which can be found here.

I have put together a small script called lxc-nat that will configure port forwarding based on entries made in /etc/lxc/lxc-nat.conf.

For example, if you have Apache running in a container called www, create the following entry:

10.0.0.1:80 -> www:80

Or if you want to access www via SSH:

10.0.0.1:2201 -> www:22

Conclusion

This article showed the steps used to configure a server to host LXC-based containers on a ZFS storage backend.

Masterless Puppet with Capistrano

Mon, 24 Feb 2014 00:00:00 -0700

Introduction

This article will describe how to create a masterless Puppet workflow with Capistrano. A MP workflow uses Puppet, but without a Puppet Master server – hence masterless. The benefit of this workflow is a reduction in the amount of required services needed to run a Puppet environment: the Puppet Master, PuppetDB, and certificate management.

Introduction
Capistrano Installation and Configuration
Capistrano Modules
- Utils
- Hiera
- Puppet
Conclusion

Capistrano Installation and Configuration

The first step is to install and configure Capistrano. The documentation found on Capistrano's homepage is excellent.

Capistrano Modules

I have written a series of "modules" for Capistrano that can be found here. "Modules" is quoted because there is no official pluggable set of libraries for Capistrano called modules. These are just tasks and libraries that I've tried to make easily distributable.

The README files found in each of the modules should be clear enough for you to successfully install, configure, and use each module. However, if something is not clear, please let me know.

Utils

The first module required is the utils module. This module contains some helper functions for rendering templates and uploading files.

Hiera

The next module to configure is the hiera module.

By the time you have finished configuring this module, you should have a set of servers listed in a stage's data source file similar to this example.

Puppet

The final module to configure is the puppet module. The README file for this module has a specific section for a masterless Puppet workflow. Again, if the documentation is not clear, please let me know.

Conclusion

This article covered the different components required for using a masterless Puppet workflow with Capistrano. You should now be able to easily use Puppet to configure remote servers without the need of a Puppet Master.