How to Install Terraform 0.12 on Ubuntu 18.04

To this day (8/8/2019) Terraform is not packaged in an official apt repository. There is an option to install it with Snap but be careful it will probably be an older version. When i checked it was v0.11.11
If you do want to install it with snap, run:

$  snap install terraform

To install the latest version follow this procedure.

You might want to update your system just in case:

$ sudo apt-get update

Now since you are getting a Terraform binary from official Hashicorp site, you will need both wget and unzip packages unless already installed:

$ sudo apt-get install wget unzip

Last step would be to download an unzip Terraform package (you can find latest here).

$ wget https://releases.hashicorp.com/terraform/0.12.6/terraform_0.12.6_linux_amd64.zip
$ sudo unzip ./terraform_0.12.6_linux_amd64.zip -d /usr/local/bin/

check that it is installed:

$ terraform -v

you are all done.

Provided by:Forthscale systems, cloud experts

Part 1: Starting with Docker

If you haven’t heard about Docker by now, you are probably hanging out with the wrong crowd. As we speak, Docker is next thing after Cloud computing and if you don’t understand yet how significant they are, last April they joined the Unicorn Club.

And what this post is all about? It is a first in a series of posts about Docker usage. I’ll start with how to start using Docker (YA newbie tutorial) in a common infrastructure. Next posts will concentrate on more advanced Docker usage.

Basics. What is Docker?

Docker is an open platform for building, shipping and running distributed applications. It gives programmers, development teams and operations engineers the common toolbox they need to take advantage of the distributed and networked nature of modern applications

Docker.io

Basically Docker allows to use application packaged in containers on machine to simplify deployment. More simple explanation is that you can run virtual OS on any Linux and soon Microsoft powered machine. This not exact explanation, since container is not a real VM, you just receive ready to use virtual environment. What is a Container? Container is a Linux Kernel level virtual environment. Sort of evolution of chroot jail environments.

If you want to compare LXC/Docker to any other VM that you can run on your computer that the biggest difference will be the lightweight of the containers. Each VM need it’s own disk space and run full OS on it while the containers use shared OS and layered FS(AuFS). So it’s lighter, faster and easier.

Why Docker?

The news that docker worth more than 1B$ is not the reason to use, but the Moby Dock whale logo may just be :). Seriously, Docker is open source platform and if you’re at this blog you know the benefits of it. They are not the first to make the service, but they are working on it for two years and at this point Docker leadership become a de-facto standard. Docker can run on any Linux OS and it’s has native support of big cloud providers - AWS, Google Engine, OpenStack, ProfitBricks and more are joining every day. Deploying Docker is really easy, I’ve started using Docker while I was going on train with bad WiFi and after 1 hour I had small application cloud with web server and DB.

Actually that’s the main reason we’ve chosen to use Docker to deploy our own infrastructure manager - Heili :) But there are few other products that might have similar functionality but they are not as advanced, neither solid as Docker.

Let’s start

Let`s leave theory here, if you want to know more check Docker site or Google.

Let’s start and build containers! As I already said Docker can run on a lot of different OS’s and it doesn’t really matter what OS you run once you installed it, that’s why I decided not to explain how to install it and just send you to official guide for installation:

Now, once you have Docker installed let’s make a first container, but first we need to have OS. Docker has ready images of common OS and we will use Ubuntu for this tutorial

~ $ docker pull ubuntu:14.04
14.04: Pulling from ubuntu
428b411c28f0: Pull complete
435050075b3f: Pull complete 
9fd3c8c9af32: Pull complete 
6d4946999d4f: Pull complete 
ubuntu:14.04: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:59662c823007a7a6fbc411910b472cf4ed862e8f74603267ddfe20d5af4f9d79
Status: Downloaded newer image for ubuntu:14.04

Let’s see what this command does:

• pull - Pull image from repository(by default Docker Hub)
• ubuntu - Image name
• :14.04 - Image tag, in this case it’s OS version

After sometime (depends on your internet connection speed) you will have Ubuntu 14.04 image ready for use. And now you can start the container :)

~ $ docker run -t -i ubuntu:14.04
root@eb029641a22c:/#

• run - Run a command in new container
• -t - Allocate pseudo terminal, so we will see the output
• -i - Interactive, you will be able to use STDIN
• ubuntu:14.04 - Image name

That’s all, you have running container with Ubuntu! Of course that’s not the way we use container, because we don’t want them to be another kind of VM, we want the container to run command/application, that’s why we have run command in Docker. So how do we do it?

The command docker run also expect to receive a command to run once the container is running:

~ $ docker run -t -i ubuntu:14.04 echo “Hello World”
Hello World
~ $

Cool! our container printed “Hello World” and we are back to our original OS, But why?

Docker container is designed to run one command and once this command is done the container is closed. So how can we run something and keep container running? Just execute command that will never exit, until something happens, in our case simple loop (press Ctrl+C to stop it):

~ $ docker run -t -i ubuntu:14.04  sh -c "while :; do echo \"Hello World\"; sleep 10; done"
Hello World
Hello World
^C~ $

So we have simple bash loop that will run until the world collapse but we don’t want to see the output all the time, because if we do, why do we need to use container? This mode is really helpful for debug for image(more explanation in the next posts) and our production container will be demonized:

~ $ docker run -d ubuntu:14.04  sh -c "while :; do echo \"Hello World\"; sleep 10; done"
dde9009d78b7b6f58a57eb7737bea01913552164c2f4125bb1e9fe6e8336b0db
~ $

Voila! Something happened and we have now long strange string, but where is the container? Is it running? Docker has command that allow us to see all containers, let’s check if we can see something with it’s help:

~ $ docker ps -a
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
46671d396781        ubuntu:14.04        "\"sh -c 'while :; d   5 minutes ago      Exited (130) 5 minutes ago                       mad_thompson        
4460b0b7a5f7        ubuntu:14.04        "echo “Hello World     8 minutes ago      Exited (0) 8 minutes ago                        gloomy_pasteur      
50292108ff3b        ubuntu:14.04        "/bin/bash"            12 minutes ago      Exited (0) 12 minutes ago                        silly_lumiere       
dde9009d78b7        ubuntu:14.04        "\"sh -c 'while :; d   3 minutes ago       Up 3 minutes                            cranky_turing

Let’s start with the command:

• ps - List containers
• -a - Show all containers, without this flag we will see only running containers

And what we have at the output?

• Container ID - Each container receive unique ID once it started, for easy management. The string we received earlier is the full ID and here we can see the shorter ID
• Image - Name of the image that has been used by this container
• Command - Command that was executed on this container
• Created - How long it’s here?
• Status - What is the current status and how long is this status
• Ports - What ports are open(explanation in the next post)
• Names - Each container can have a unique name, so we will not have to use the ID all the time. If name is not provided, Docker will generate random name for us(this is the 1st Docker issue submitted on Git)

We have here “mad” Ken Thompson(Unix and C inventor), “gloomy” Louis Pasteur(vaccination discoverer), “silly” Lumiere brothers(first film makers) and “cranky” Alan Truing(founder father of computer since, also spotted as Sherlock) - sounds like the Avengers(probably you have another team members). You can keep using this cool name generator or next time when you run the container give --name flag with desired unique name.

We see that our latest container is up and running but we also can see 3 container that are exited. This are the container that  we used on previous steps, you can see it by the command. This containers are exist, so this mean that this names are reserved, so if you use random names you’re alright, but if you have your own unique name, you can use it again. To be able to use it again we need to remove the stopped containers:

~ $ docker rm mad_thompson gloomy_pasteur silly_lumiere

or

~ $ docker rm $(docker ps -q -f status=exited)
46671d396781
4460b0b7a5f7
50292108ff3b
~ $ docker ps -a
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
dde9009d78b7        ubuntu:14.04        "\"sh -c 'while :; d   5 minutes ago      Up 5 minutes                           cranky_turing

Turing is alone now, it’s it should print “Hello World” in loop for 5 minutes, every 10 seconds = 30 messages. Let’s check the output of the container (make sure to change to your container name):

~ $ docker logs cranky_turing
Hello World
Hello World
Hello World
Hello World
...
Hello World
~ $

As you can see our container is running as expected. To stop just execute:

~ $ docker stop cranky_turing

And delete the container

~ $ docker rm cranky_turing

This is the basics of using the Docker containers, you can play more with it while the next post is written.

The next part will explain more advanced containers usage, containers network and how to build small web application on Docker container!

Provided by:Forthscale systems, cloud experts

Backup to AWS S3 with duply on Ubuntu

1. Install duply and dependences (as root).
sudo apt-get install duplicity duply python-boto

2. Create backup profile (as root).  
sudo duply bck_profile create

3. Edit profile /root/.duply/bck_profile/conf like as:
GPG_PW='___YOUR GENERATED PASSWORD____'
TARGET='s3://s3-endpoint-name.amazonaws.com/bucket-name/directory/for/backup/'
TARGET_USER='AWS_KEY_ID'
TARGET_PASS='AWS_SECRET_KEY'
SOURCE='/'
MAX_AGE=6M
#MAX_FULL_BACKUPS=1
#VERBOSITY=5
TEMP_DIR=/content
#DUPL_PARAMS="$DUPL_PARAMS --time-separator _ "
#DUPL_PARAMS="$DUPL_PARAMS --short-filenames "
MAX_FULLBKP_AGE=1W
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
VOLSIZE=200
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

4. Edit /root/.duply/bck_profile/excludes to include the list of non wanted directories such as:
- /dev
- /home/*/.cache
- /home/*/.ccache
- /lost+found
- /media
- /mnt
- /proc
- /root/.cache
- /root/.ccache
- /run
- /selinux
- /sys
- /tmp
- /var/cache/*/*
- /var/log
- /var/run
- /var/tmp

5. Tun a full backup test:
sudo duply bck_profile backup

6. Configure crone for daily backup:
sudo crontab -e 
and  add:
0 2 * * * duply bck_profile backup

7. Some useful commands to operate your backup:
To get current list of present backups.
sudo duply status
To restore all files from last backup to /target_dir
sudo duply restore /target_dir
To restore directory etc from last backup to /target_dir
sudo duply fetch etc /target_dir

Note:
 All backups early than MAX_FULLBKP_AGE=1W will be incremental, every MAX_FULLBKP_AGE=1W will create full backup.

Provided by:Forthscale systems, cloud experts

Creating a RAID device on Amazon AWS Elastic

Following procedure is for creating a software (OS managed) RAID disk devices.
EBS (Amazon web services elastic block storage) does not provide any RAID abilities by itself.
But you can use Linux SW raid option to increase either speed or device redundancy.
This tutorial will show you how.

First install RAID management tool on target EC2 instances

For Debian based (Debian, Ubuntu) execute: 
apt-get install mdadm
For Red Hat based (RHEL, Oracle linux, CentOS) execute:
yum install mdadm
For Gentoo execute:
emerge mdadm

Now prepare the EBS devices.

Create desired EBS volume
Attach new EBS volumes to EC2 instance and write down the device name (for example xvdf and xvdg)

Now you can create the RAID device from them.

Set up RAID 0 on this EBS instance with the following command:
  mdadm --create --verbose --auto=yes /dev/md0 --chunk=256 --level=0 --raid-devices=2 /dev/xvdf /dev/xvdg
   blockdev --setra 65536 /dev/md0

Now verify that the raid device /dev/md0 exists
cat /proc/mdstat

Next step is to add devices to the mdadm.conf file (main sw raid configuration file)
   echo DEVICE /dev/xvdf /dev/xvdg | sudo tee /etc/mdadm/mdadm.conf
 
Add the other device info about /dev/md0 to the mdadm.conf file so that it comes back on reboot
   mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Filesystem

One RAID is set up , you can create a file-system on it, we usually propose XFS, but you can use any file system (ext3/4, zfs or reiserfs)
  
To create XFS file system, you will need to install support for it first.
For Debian based (Debian, Ubuntu) execute: 
apt-get install xfsprogs
For Red Hat based (RHEL, CentOS) execute:
yum install xfsprogs
For Oracle Linux:
Log in to ULN, and subscribe your system to the ol6_x86_64_latest channel.
Then run:
yum install xfsprogs xfsdump 
For Gentoo execute:
emerge xfsprogs 

Once installed, execute
mkfs.xfs -f /dev/md0
to create an actual filesystem on your new RAID device.
  
Now you can mount the file system
   mkdir /MOUNTPOINT (for example /raid)
   mount /dev/md0 /MOUNTPOINT (for example /raid)
 
Verify the volume exists and the size you expected
df -h 
 
Edit /etc/fstab to make sure mount point comes back on reboot
/dev/md0 /raid xfs noatime,noexec,nodiratime 0 0

You have the new RAID based partition in your system.
Same procedure will work on both public EC2 cloud and VPC.

Provided by:Forthscale systems, cloud experts

Fixing Fujitsu LIFEBOOK UH572 Touchpad on OpenSuSE 13.1

1. open a terminal or change to /etc/default/ in a file manager
2. run # edit as a superuser file /etc/default/grub
3. go to the line containing: GRUB_CMDLINE_LINUX
4. add "i8042.notimeout i8042.nomux" to the string in quotes
5. save and exit
6. run # sudo grub2-mkconfig -o /boot/grub2/grub.cfg
7. init 6 (reboot)

Provided by:Forthscale systems, cloud experts

Accessing NoMachine NX Server as root

IN a default installation on NX server root access is disabled. To allow root login just follow those simple steps.

Edit file:
/usr/NX/etc/server.cfg

and substitute line reading

#EnableAdministratorLogin = "0" 
to:
EnableAdministratorLogin = "1"
save the file and exit.

and run in shell as a root user:

/usr/NX/bin/nxserver --useradd root

this is it. Changes will take affect immediately.

Setting up updates with public yum server for Oracle Enterprise Linux

Oracle provides a free and public yum server to update it`s Enterprise Linux distribution.
It is easy to set-up and it supports versions 4, 5 and 6.

To set up public repositories for different versions, download configuration file (for ex. with wget)

In Oracle Enterprise Linux 6.x

# cd /etc/yum.repos.d
# wget https://public-yum.oracle.com/public-yum-ol6.repo

In Oracle Enterprise Linux 5.x
# cd /etc/yum.repos.d
# wget https://public-yum.oracle.com/public-yum-el5.repo



In Oracle Enterprise Linux 4, Update 6 or Newer
# cd /etc/yum.repos.d
# if you have old repo file then:
# mv Oracle-Base.repo Oracle-Base.repo.disabled
# wget https://public-yum.oracle.com/public-yum-el4.repo

You can verify new configuration with:
#  yum list

And execute update with :

# yum update

Provided by: Forthscale systems, scalable infrastructure experts

Vertica installation tutorial

        

How to install Vertica Analytic Database

Download Vertica

Download Vertica RPM from the site http://www.vertica.com/

For this tutorial we are using Community Edition 6.1.2-0, which is the latest version so far. This version has bug, that need to fixed manually in multiple nodes install, the bug will be fixed in next versions.

Server preparation

Before installing Vertica few things must be done on the server and some of the are optional(but strongly suggested). All this steps must be done on all nodes if installing multi node solution.        

Must configurations

• Install Linux OS - we are using Centos 6.4 for this tutorial
• Check that the server is has at least 1 GB RAM free(minimum for install, we suggest     to have more memory for normal usage)

[root@vertica01 ~]# free -m

            total       used       free     shared    buffers     cached

Mem:          1877        465       1412          0         14        343

-/+ buffers/cache:        106       1770

Swap:         3039          0       3039

• Server need at least 2GB of swap

[root@vertica01 ~]# free -m

            total       used       free     shared    buffers     cached

Mem:          1877        465       1412          0         14        343

-/+ buffers/cache:        106       1770

Swap:         3039          0       3039

• Disable SELinux.

[root@vertica01 ~]# vi /etc/sysconfig/selinux

SELINUX=disabled

• Disable firewall (for multiple nodes install). If you system must have firewall ensure that this ports are open and not used:

Port	Protocol	Description
22	TCP	SSH
5433	TCP	Vertica Client
5433	UDP	Vertica Spread
5434	TCP	Vertica cluster communication
5444	TCP	Vertica Management Console
5450	TCP	Vertica Management Console
4803	TCP/UDP	Spread
4804	UDP	Spread
4805	UDP	Spread

• Verify that pam_limits.so module is configured for su command

[root@vertica01 ~]# vi /etc/pam.d/su

session required pam_limits.so

• For multiple node install add nodes names and IP’s to /etc/hosts file for name resolution. Add it even if you’re using DNS server for faster resolution. Also add master host name and IP to it self, Vertica is not checking on what host it’s running

[root@vertica01 ~]# vi /etc/hosts

192.168.122.01  vertica01

192.168.122.02  vertica02

192.168.122.03  vertica03

• For multiple node install make sure that root and DB management (default is     dbadmin) are able to ssh between the nodes without a password. The root user ssh is used at the install only

http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/                    

Suggested configurations

• Configure and     start NTP service
• Disable CPU Frequency Scaling in BIOS
• Configure I/O scheduler to deadline, noop or cfq

[root@vertica01 ~]# vi /boot/grub/grub.conf

Add elevator=<name> to kernel line

Single Node

Once the system is ready for installation you may install the RPM you have downloaded from the Vertica

[root@vertica01 ~]# rpm -ivh vertica-6.1.2-0.x86_64.RHEL5.rpm

RPM will add new directory under /opt with Vertica installation and management scripts. For the basic install run the script with one parameter that indicates DB admin user, with this parameter the script will try to recreate it:

[root@vertica01 ~]# /opt/vertica/sbin/install_vertica -u dbadmin

When the script finish to run you have Vertica installed on single node. You can use admin tools to create new DB:

[root@vertica01 ~]# /opt/vertica/bin/adminTools

Multiple Nodes

Multi node installation uses the same script that is ran only on one server(master) that will install Vertica on the rest of the nodes. To install installation script, install same RPM

[root@vertica01 ~]# rpm -ivh vertica-6.1.2-0.x86_64.RHEL5.rpm

The script will copy the RPM to the rest of the nodes so it's important to have passwordless SSH between the nodes for root user and configure all the nodes in the /etc/hosts. Run the installation script with few more parameters:

-s     – nodes list comma-seperated

-r     – path to the Vertica installation RPM file, to install on rest of the nodes

-u    – DB admin user with password less SSH login between the nodes

-T     – point-to-point nodes communication, used when nodes are not on the same subnet or when nodes are virtual machines

[root@vertica01 ~]# /opt/vertica/sbin/install_vertica -s node01,node02,node03 -r ~/vertica-6.1.2-0.x86_64.rpm -u dbadmin -T

Installation will take more time as it installing on couple of nodes and doing system and networks tests. Once it done you can create new DB and check that it’s working on all the nodes

[root@vertica01 ~]# /opt/vertica/bin/adminTools

Common problems

Installation bug – installation fails with Error: invalid literal for int() with base 10: '8%'    

This is bug in Centos/Red Hat installation confirmed by Vertica, current solution is manual fix. Open /opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/SSH.py on line 1982 change

df /tmp | tail -1 | awk '{print $4}'

to

df -P /tmp | tail -1 | awk '{print $4}'

Network tests fails without error:

Installation will not succeed if the SSH fails between the nodes. If you never SSH from vertica01 to vertica02 the system will ask to add it's fingerprints to know_hosts file, but installation script can't do it by it's own. There are two workarounds for this:

• Make manual     SSH between all the nodes for the first time
• Change or add StrictHostKeyChecking no to /etc/ssh/ssh_config on all the nodes, this will cause SSH not to check server fingerprints with known_hosts. You can read more about here: http://linux.die.net/man/5/ssh_config

Provided by: ForthScale systems, scalable infrastructure experts