Part 2: Using Docker

 

This is second part in our Docker tutorials, we strongly suggest to Part 1 before continuing here. At this point you should be familiar with basic Docker commands, how to run containers with cool random names. But you need more, to do something really useful with Docker. So today we will learn how to build basic Web application with couple of containers that will look like this at the end:

As you can see we will have lot of different solutions at the end of this tutorial. This structure is not production wise, it's just to show what Docker can do.

Docker Hub / Database

I've picked pgSQL for this tutorial, bust you must look at this more as concept, because you can do it with any DB (SQL and noneSQL).

With Docker your life should be easy, you don't have to install machines and DB's anymore, you just download image you need and use it. Most of the software have images ready in Docker Hub and lot of them already have preconfigured images made by users. But what is Docker Hub? It's official public images repository - https://hub.docker.com. You can register and upload your images to the hub for personal and public use. If you will continue use Docker you will meet personal registries (own by companies or private people), they work same way (later in our tutorials we will even install registry server).

One you logged in to Hub (but you don't have too) you can search for postgres. I've found 10 repositories - first one is official repository and 9 repositories made by users:

It's always suggested to take official repositories and make modifications for them, but if you're lazy and there is user repository that fit your needs (check what images include). If you're using user repository it's also important to check last update. Sometimes it's good to use old but stable versions, but sometime you need updated one.

Inside the repository you will see tags information (usually tags represent versions) and how to use the repository.

Now let's go to your machine and start building containers

As always we will use latest image and version, just because it's tutorial and we don't really care about it :)

Actually if you sure that you going to use common software you don't have to look in repository, just pull the name of it... The repository can be useful for usage, but also not must, as you can view image configuration... Anyway, let's pull our repo:

davidg@linux-cpl8:~> docker pull postgres:latest
latest: Pulling from postgres
104de4492b99: Pull complete 
065218d54d7d: Pull complete 
6d342ad75f37: Pull complete 
9433e325f9ad: Pull complete 
7c38e9491f7e: Pull complete 
d9a636286bd1: Pull complete 
4020db192fff: Pull complete 
b93ccbbdcc22: Pull complete 
13af7ae40c45: Pull complete 
e6826f7776c8: Pull complete 
5c67b212f3da: Pull complete 
1e87f75b5751: Pull complete 
24a73f6adf68: Pull complete 
effe3b6a83fc: Pull complete 
65482096da78: Pull complete 
089cc1d86ef7: Pull complete 
d4c43025a271: Pull complete 
41684070b967: Pull complete 
f969e36858c2: Pull complete 
a2c56c0927fc: Pull complete 
7bf0ec35adaf: Already exists 
postgres:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:0b2d2e463174edacb17d976d72adc839c032bfcfdf6da6799e288014d59998f8
Status: Downloaded newer image for postgres:latest 

And run your first Postgres container:

~ $ docker run --name postgres_test -e POSTGRES_PASSWORD=d0ckerul3z -d postgres
415a2a8734845a8d9188e959bb7acf90ecf21da1479f8213a6df4e2ac096430a
~ $ docker ps -a
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
415a2a873484        postgres:latest     "/docker-entrypoint.   5 minutes ago       Up 5 minutes        5432/tcp            postgres_test

So now we have  DB with default database, let's connect to it! But how? 'postgres_test' is name of the container but not DNS name. How does Docker network works?
I'll short the explanation of the network in few words. If you will run ifconfig you will notice new device:

~ $ ifconfig
docker0   Link encap:Ethernet  HWaddr 56:84:7A:FE:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:304 (304.0 b)  TX bytes:6496 (6.3 Kb)

 Device docker0 is created when docker service is started and it's Virtual Ethernet Bridge. All you containers traffic goes through it, it's the gateway. This mean we can connect to container via it's IP:

~ $ docker inspect postgres_test | grep IPAddress
        "IPAddress": "172.17.0.1",

If you will run 'docker inspect [CONTAINER_NAME]' without the 'grep' you will all the info about the container, try doing it. When you done let's connect to our DB:

~ $ psql -h 172.17.0.1 -U postgres postgres
Password for user postgres: d0ckerul3z
psql (9.3.6, server 9.4.4)
WARNING: psql major version 9.3, server major version 9.4.
         Some psql features might not work.
Type "help" for help.

postgres=# 

 Now you have container with fully functioned DB

Application Container

Next part is to create an application that will connect to our DB and use it.
I'm using simple flask application for the basic things, but it can be anything you want:

import psycopg2
import psycopg2.extras
import sys
from flask import Flask

app = Flask(__name__)

@app.route("/")
def test():
  con = None
  result = '<h1>The Guardians of the Galaxy</h1><table border="1"><tr><th>&nbsp;</th><th>Character</th><th>Real Name</th></tr>'
  try:
    con = psycopg2.connect("host='postgres_test' dbname='postgres' user='postgres' password='d0ckerul3z'") 
    cursor = con.cursor(cursor_factory=psycopg2.extras.DictCursor)
    cursor.execute("SELECT * FROM guardians")
    rows = cursor.fetchall()
    for row in rows:
      if row['teamleader']:
        result += "<tr><td>%s</td><td><b>%s</b></td><td><b>%s</b></td></tr>" % (row["id"], row["character"], row["realname"])
      else:
        result += "<tr><td>%s</td><td>%s</td><td>%s</td></tr>" % (row["id"], row["character"], row["realname"])
    result += '</table>'
  except psycopg2.DatabaseError, e:
    result =  'Error %s' % e    
  finally:
    if con:
      con.close()
  return result

if __name__ == "__main__":
  db_data = (
    ('Adam Warlock', 'Him', 'false'),
    ('Drax the Destroyer', 'Arthur Sampson Douglas', 'false'),
    ('Gamora', 'Gamora', 'false'),
    ('Quasar a.k.a. Martyr', 'Phyla-Vell', 'false'),
    ('Rocket Raccoon', 'Rocket Raccoon', 'true'),
    ('Star-Lord', 'Peter Quill', 'true'),
    ('Groot', 'Groot', 'false'),
    ('Mantis', 'Mantis', 'false'),
    ('Major Victory', 'Vance Astro', 'false'),
    ('Bug', 'Bug', 'false'),
    ('Jack Flag', 'Jack Harrison', 'false'),
    ('Cosmo the Spacedog', 'Cosmo', 'false'),
    ('Moondragon', 'Heather Douglas', 'false'),
)

  con = None
  try:
    con = psycopg2.connect("host='postgres_test' dbname='postgres' user='postgres' password='d0ckerul3z'")   
    cur = con.cursor()  
    cur.execute("DROP TABLE IF EXISTS Guardians")
    cur.execute("CREATE TABLE Guardians(Id SERIAL PRIMARY KEY, Character TEXT, RealName TEXT, TeamLeader BOOLEAN)")
    query = "INSERT INTO Guardians (Character, RealName, TeamLeader) VALUES (%s, %s, %s)"
    cur.executemany(query, db_data)
    con.commit()
  except psycopg2.DatabaseError, e:
    if con:
        con.rollback()
    print 'Error %s' % e    
    sys.exit(1)
  finally:
    if con:
      con.close()
  app.run(host= '0.0.0.0')

And now for the interesting part, get this to container! You can always start simple CentOS container and copy files there and start them, but why use Docker for this?

In Docker we make it simpler and more automatic - we make and image, just like image you just downloaded!
How do we do it? Lets start with making new directory where new image files will be stored:

~ $ mkdir docker_image_1
~ $ cd docker_image_1/
~/docker_image_1 $ 

Save the code as 'guardians.py'. Create new file called 'Dockerfile' with you favorite editor and insert this:

FROM centos:latest
MAINTAINER David Golovan 
LABEL Description="guardians:1 - WebApp to print Guardians list" Vendor="Forthscale" Version="1.0"

RUN yum -y update && yum -y install epel-release 
RUN yum -y install python-pip gcc postgresql postgresql-devel python-devel
RUN pip install flask psycopg2
COPY guardians.py /opt/guardians/
CMD ["python", "/opt/guardians/guardians.py"] 

Save the scripts to the new directory and before we proceed, what we are doing here?

• FROM - Image name + tag. This is the image on which our image is based, so it will contain everything from it.
• MAINTAINER - Just info about image owner if it goes public
• LABEL - Information about the image and it's version
• RUN - Execute shell command. We are executing yum install and pip install for packages, but it can be any command
• ADD - Copy file/directory from local server to the image. We are coping our scripts to /root/ of the image
• CMD - Command to execute once container is running. When this commands stops(error or just exit) the container will also stop, so always make sure to run here commands that don't exit :)

Build our image and we can start containers using it. Note: I've not pulled CentOS images before so our image build will pull on first build:

~/docker_image_1 $ docker build -t forthscale/guardians:1.0 .
Sending build context to Docker daemon 5.632 kB
Sending build context to Docker daemon 
Step 0 : FROM centos:latest
latest: Pulling from centos
f1b10cd84249: Pull complete 
c852f6d61e65: Pull complete 
7322fbe74aa5: Already exists 
centos:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:a4627c43bafc86705af2e8a5ea1f0ed34fbf27b6e7a392e5ee45dbd4736627cc
Status: Downloaded newer image for centos:latest
 ---> 7322fbe74aa5
Step 1 : MAINTAINER David Golovan <davidg@forthscale.com>
 ---> Running in cded0075df8b
 ---> b582a22635b5
Removing intermediate container cded0075df8b
Step 2 : LABEL Description "guardians:1 - WebApp to print Guardians list" Vendor "Forthscale" Version "1.0"
 ---> Running in e8ba10cb9536
 ---> 3779b4331b2e
Removing intermediate container e8ba10cb9536
Step 3 : RUN yum -y update && yum -y install epel-release
 ---> Running in 2bab13a17bf5
Loaded plugins: fastestmirror
Determining fastest mirrors
.....
 ---> b7207970f3b8
Removing intermediate container 841af65c665a
Step 4 : RUN yum -y install python-pip gcc postgresql postgresql-devel python-devel
 ---> Running in ffbb682e8747
Loaded plugins: fastestmirror
.......
 
 ---> 534c270b560b
Removing intermediate container ffbb682e8747
Step 5 : RUN pip install flask psycopg2
 ---> Running in 7f380d6f374c
Downloading/unpacking flask
....... 
Downloading/unpacking psycopg2
Successfully installed flask psycopg2 Werkzeug Jinja2 itsdangerous markupsafe
Cleaning up...
 ---> a7038e07d3fc
Removing intermediate container b066cc5e4b21
Step 6 : COPY guardians.py /opt/guardians/
 ---> a98ee8d4bc21
Removing intermediate container ffb5d8030f25
Step 7 : CMD python /opt/guardians/guardians.py
 ---> Running in c2c8da611dca
 ---> 70df63860b6e
Removing intermediate container c2c8da611dca
Successfully built 70df63860b6e

Our image is ready!

~/docker_image_1 $ docker images
REPOSITORY                            TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
forthscale/guardians                  1.0                   70df63860b6e        About a minute ago   379.2 MB
centos                                latest              7322fbe74aa5        2 weeks ago          172.2 MB
postgres                              latest              7bf0ec35adaf        2 weeks ago          213.9 MB
ubuntu                                latest              6d4946999d4f        3 weeks ago          188.3 MB
ubuntu                                trusty              6d4946999d4f        3 weeks ago          188.3 MB
ubuntu                                14.04               6d4946999d4f        3 weeks ago          188.3 MB

You can use it same way as any other containers, but we need to let it know about the DB container. On previous step we found the IP manually and it's not good for production environment, specially when the address are random. To handle this problem Docker allow to link containers on same host (and actually on different hosts using Ambassador) by just using flags. Second problem is that usually all Docker container ports are closed to public and we need to open it (if you look at postgres container you will see it opened 5432/tcp port):

~/docker_image_1 $ docker run -p 8080:5000 --link postgres_test:postgres_test --name guardian -d forthscale/guardians:1.0
fb2d7143af007fccdad3bf74c500a55562757c4a0fedc4ecdd9e9b35d6c22b99

• -p 8080:5000 - Open port and redirect it. 80 is public port and 5000 is port our application is listening (default flask port)
• --link - Linking to container. First write container name (that's why it's important to name your containers) and second write what name to use for it inside the container (I prefer to use same name).

Open your browser and go to http://localhost:8080

But wait, that's not all, you can make even more! We can have shared directory for all the containers and for example server static files from there.
In the Dockerfile add this lines before the CMD:

RUN mkdir /opt/guardians/static 
VOLUME ["/opt/guardians/static"] 

Change this lines in guradians.py

app = Flask(__name__) 
result = '<h1>The Guardians of the Galaxy</h1><table border="1"><tr><th>&nbsp;</th><th>Character</th><th>Real Name</th></tr>'

To

app = Flask(__name__, static_url_path = "/static", static_folder = "static")
result = '<img src="static/small_h.png" /><br /><h1>The Guardians of the Galaxy</h1><table border="1"><tr><th>&nbsp;</th><th>Character</th><th>Real Name</th></tr>' 

Build the image again, now you can use 1.1 tag:

~/docker_image_1 $ docker build -t forthscale/guardians:1.1

Remove old container and run it again with new flag:

~/docker_image_1 $ docker stop guardian && docker rm guardian 
~/docker_image_1 $ docker build -t forthscale/guardians:1.1 .

Now create the shared directory and put Docker logo there:

~/docker_image_1 $ mkdir /tmp/guardians
~/docker_image_1 $ wget -P /tmp/guardians https://www.docker.com/sites/default/files/legal/small_h.png

And start new container:

~/docker_image_1 $ docker run -p 8080:5000 --link postgres_test:postgres_test -v /tmp/guardians:/opt/guardians/static --name guardian -d forthscale/guardians:1.1

Test your page again :) You can start another container with same image, just make sure they have different ports, and they will use same DB and same directory for static image. Here is explanation what you did in this steps:

• In the Docker file you've added command to create new folder and make read/write with VOLUME command
• In guardins.py you've allowed static files in flask and their location and added HTML tag to show the image
• Build image was fast because it use cache for most of the parts of the file
• In docker run command flag -v allow to map any local directory to any read/write directory of the image. When running  docker inspect [CONTAINER] it will list volumes with read/write permissions that you can rewrite.

Now you can build much more advanced containers for you applications

Provided by:Forthscale systems, cloud experts

How to install PGPool II on PostgreSQL Servers in master-slave architecture + PGPoolAdmin web managment

General Information

PGPool can run on same server along with PostgreSQL DB or on stand alone server(recommended). In this article we will install PGPool on stand alone server, but the only difference is connection ports on PGPool and PostgreSQL.

We will install PGPool II 3.1 on PostgreSQL 9.1.

Basic architecture:

┏───────────────┓

│                │

│    pgpool-1    │

│  pgpool server │

│                │

┗───────────────┛

//             \\

//               \\

//                 \\

\\//                \\//

\/                  \/

┏───────────────┓           ┏───────────────┓

│                │           │                │

│    pgsql-1     │ streaming │     pgsql-2    │

│  pgsql master  │══════════>│  pgsql slave   │

│     server     │replication│     server     │

│                │           │                │

┗───────────────┛           ┗───────────────┛

Fail cases:

Slave fails

In case slave server will fail PGPool will start failover.sh script and will mark server as Down (state 3). It'll reconnect all open connections to this server to the master server, at this case users that have been on this connection will be disconnection.

When the server will be fixed and started up you will need manually start recovery process. To do this the PostgreSQL need to be turned off on the server. Recovery process will take backup from the master and restore it on slave. When it will finish restore process PGPool will connect it back. Streaming replication process will restore all the data inserted since the backup. No data lost.

Master fails

In case master server will fail PGPool will start failover.sh script, that will notice that the failed server is master and set trigger to slave server. When slave server notice the trigger it will promote itself to master. All the connections to master will be reconnected to slave and user will be disconnected. All the data inserted to master and not yet replicated to slave will be lost. When the server will be fixed and started you will need manually start recovery process. To do this PostgreSQL need to be turned off on the server. Recovery process will take backup from the master and restore it on slave. When it will finish restore process PGPool will connect it back. Streaming replication process will restore all the data inserted since the backup.

PGPool fails

PGPool is SPOF and if it fails there will be no connection to client. To fix this problem you need to create PGPool HA cluster. This can be done with cluster software like Pacemaker or Linux Heartbeat.

PGPool II setup

PGPool can be installed from source of from OS packages, I recommended installing from source, as we will need some files from the tar package. Source files can be downloaded from PGPool official site: http://pgpool.net/mediawiki/index.php/Downloads#pgpool-II

Before compiling PGPool you might need to install missing packages:

• For Debian/Ubuntu: libpq-dev
• For RH/CentOS: postgresql-libs

Download needed version of PGPool, open and install:

$ tar xfz /some/where/pgpool-II-3.1.1.tar.gz

$ cd pgpool-II-3.1.1

$ ./configure

$ make

$ make install

You can configure PGPool install path using –prefix flag, in this article we will use default configurations, which will install PGPool configurations to /usr/local/share/etc

If you installing from OS software manager you can't configure the path, but you won't need installing missing packages. Default path will for configuration will be /etc/pgpool2/
After installing PGPool we need to do some basic configuration, here's the sample pgpool.conf file with basic configuration needed to run PGPool in master-slave with streaming replication:
link for configuration

Make sure you changing this configuration to ones you need:

• PGPool client connections port

port = 5432

• PostgreSQL Master server

backend_hostname0 = 'pgpool-1'

• PostgreSQL connection port

backend_port0 = 5432

• Load balancing weight

backend_weight0 = 0

• PostgreSQL data path(on PostgreSQL server)

backend_data_directory0 = '/var/lib/postgresql/9.1/main'

• PostgreSQL username and password (super user)

sr_check_user = 'postgres'

sr_check_password = ''

• Health check for PostgreSQL nodes period in seconds

health_check_period = 10

• PostgreSQL username and password (super user)

health_check_user = 'postgres'

health_check_password = ''

• User to run recovery script on PostgreSQL server

recovery_user = 'postgres'

recovery_password = ''

Next we will setup user for PGPool Administration tool called PCP(later can be used in PGPoolAdmin). We need to encrypt user password to md5:

$ pg_md5 password

Edit pcp.conf and insert the new password along with username to the end of the file:

username:5f4dcc3b5aa765d61d8327deb882cf99

Next we need to add script for failover, that already configured in pgpool.conf. This script will check if the PostgreSQL server that failed is master or slave. In case of master fail it will put trigger on the slave and it will promote itself to master. Put this script at same location as all configuration files /usr/local/share/etc and give it execute permissions:

link to failover.sh

Create necessary directories

$ mkdir /var/run/pgpool

$ mkdir /var/log/pgpool

Set permissions to this directories for user that will run PGPool.

Now PGPool is fully configured and can be started. To start it just run PGPool command. By default it will run at same shell as your user and will print the log to STDOUT. Here is sample of command how to start PGPool as deamon and save the log to file:

$ pgpool -n -d > /tmp/pgpool.log 2>&1 &

You may not start the PGPool now if you want to install PGPoolAdmin.

PGPoolAdmin setup(optional)

PGPoolAdmin allows to manage PGPool: start/stop PGPool, edit configurations, add/remove/ recovery/promote PostgreSQL nodes.

To install PGPoolAdmin you will need to install first Apache service to the server(I hope you know how to do it).

Download PGPoolAdmin from http://pgpool.net/mediawiki/index.php/Downloads#pgpoolAdmin.

Open the tar file and put it content into Apache data directory:

$ tar xfz /some/where/pgpoolAdmin-3.1.1.tar.gz

$ mv /some/where/pgpoolAdmin-3.1.1/pgpooladmin /var/www/html/

Now we need to set permissions to apache user to edit configuration files

$ chown apache /var/run/pgpool

$ chown apache /var/log/pgpool

$ chown apache /share/local/etc/pgpool.conf

$ chown apache /share/local/etc/pool_hba.conf

$ chown -R apache /var/www/html/pgpooladmin

PGPool recovery script must login to PostgreSQL server and create trigger file, to do it must login to server via ssh from user apache. To increase security you might want to set firewall to allow to connection only from PGPool server.

PGPoolAdmin is ready to installed and used. Open in you browser http://serverIP/pgpooladmin/install/index.php follow the steps. At the end PGPoolAdmin will start PGPool on the server. To login use PCP user and password configured before.

PostgreSQL setup

At this point we have PGPool running and accepting connections, but it might not work properly as PostgreSQL might not accept those connections. At this step we will re-configure your PostgreSQL server. This step must be done to all new node you want to add to PGPool, before running recovery process.

Edit postgresql.conf configuration file with following variables:

listen_address = '*'

hot_standby = on

wal_level = hot_standby

max_wal_senders = 1

Edit pg_hba.conf configuration file:

host all all 0.0.0.0/0 trust

host replication postgres 0.0.0.0/0 trust

This mean that the server will accept connections from all IP's without authentication for any user. It's possible to put your subnet mask or IP of PGPool server. Note, md5 authentication is not working with PGPool master-slave configuration.

Second allows to do streaming replication between any server without authentication for user postgres. This can be changed for needed subnet or IP of second server and if you want to use other user for replication(User must have REPLICATION or SUPERUSER permission on the second server).

Now we need to setup some PGPool recovery functions. If you have installed PGPool from source you have them inside tar file. If you installed it from software manager download the source file and follow this step.

Inside the source file you have sql directory. You need to compile it each one of them and run the compiled SQL file on postgres DB and template1. If you installed PostgreSQL for software manager you will need to install additional package:

• For Debian/Ubuntu: postgresql-server-dev
• For RH/CentOS: postgresql-devel

Run in each directory:

$ make install

$ psql -f pgpool-*****.sql postgres

$ pgsql -f pgpool-*****.sql template1

PGPool recovery process is using 2 scripts, one on master and second on slave. First script called basebackup.sh, it's executed from the master and creates backup, then it sends it to the slave. This script need to be in PostgreSQL data directory(default /var/lib/postgresql/9.1/main) with execute permissions Next user that will run recovery process on the nodes need to be able to the login without password to the each other(again, I hope you can do it by yourself).

Second script allows PGPool to start PostgreSQL service. This script uses pg_ctl command, check that the path is correct. If you don't have pg_ctl change the path to /etc/init.d/postgresql and remove from the command “-w -D $DESTDIR”.

Create trigger directory and give postgres permissions on it

$ mkdir -p /var/log/pgpool/trigger

$ chown postgres /var/log/pgpool/trigger

Finishing the setup

Now you should have running PGPool and configured PostgreSQL master server. Last step is to configure the slave and test everything. Do all the configuration on the slave same as on the master and at the end stop the PostgreSQL service. If you installed PGPoolAdmin login into the interface, go to “Edit pgpool.conf” and then to “Backends” and press “Add”. Fill all the info for new backend and change “backend_weight” on both backends to 1. Save the configuration and reload PGPool. When PGPool will discover new node press “Recovery” near it. After recovery

Have questions? Just contact us right away and we will be happy to assist

Powered by 123ContactForm | Report abuse