mounting AWS (Amazon Web Services) EFS on Linux Ubuntu 18.04

Amazon Elastic File System (Amazon EFS) is a scalable file storage for EC2 and services that run on EC2 (for example Kubernetes clusters). The device is accessible on Linux via the NFS protocol and can be used my multiple instances and pods at the same time.
For more information on EFS visit AWS documentation.


Step one: Gather information
In our case ti is pretty straightforward. Ubuntu instance in the same VPC as the EFS and a DNS name of the file system we want to access. The format uses following convention:

http://file-system-id.efs.aws-region.amazonaws.com

And the exact URL is available on AWS console AWS home under filesystem's DNS name or via cli

Step two: Install the NFS Client for Linux

$ sudo apt-get update
$ sudo apt install nfs-kernel-server

Step three: Mount the file system on EC2 instance.
Create (if you don't have already) a mount point for the EFS

$ sudo mkdir -p /mnt/efs-mount-point

Mount the EFS share on the instance

$ sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport mount-target-DNS:/   /mnt/efs-mount-point

Now we have a mounted Amazon EFS file system on Ubuntu EC2 instance.
Keep in mind that command mounted doesn't persist across reboots. if you want it to be permanently accessible you have to add it to the fstab.


Common error:

efs mount.nfs: Connection timed out

This error can occur because either the Amazon EC2, mount target security groups or file system access are not configured properly.

For more troubleshooting tips you can visit:
https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html

How to Install Terraform 0.12 on Ubuntu 18.04

To this day (8/8/2019) Terraform is not packaged in an official apt repository. There is an option to install it with Snap but be careful it will probably be an older version. When i checked it was v0.11.11
If you do want to install it with snap, run:

$  snap install terraform

To install the latest version follow this procedure.

You might want to update your system just in case:

$ sudo apt-get update

Now since you are getting a Terraform binary from official Hashicorp site, you will need both wget and unzip packages unless already installed:

$ sudo apt-get install wget unzip

Last step would be to download an unzip Terraform package (you can find latest here).

$ wget https://releases.hashicorp.com/terraform/0.12.6/terraform_0.12.6_linux_amd64.zip
$ sudo unzip ./terraform_0.12.6_linux_amd64.zip -d /usr/local/bin/

check that it is installed:

$ terraform -v

you are all done.

Provided by:Forthscale systems, cloud experts

How To Install DenyHosts on Ubuntu

What is DenyHosts?

This is a python-script for the protection of  ssh server, like fail2ban, but consumes 

far less memory. The script parses the log file and finds failed login attempts and 

prohibits login from these IP addresses.

Installation steps.

Installation process is simple and contains some steps:

1. Retrieve new lists of packages

sudo apt-get update

2. Installation DenyHosts:

sudo apt-get install denyhosts

If the process is successfully completed, you will see the information of the screen:

After installation you can add your IP address in whitelist:

sudo nano /etc/hosts.allow

and insert following string:

sshd: 12.34.45.678

Where “12.34.45.678” – your IP address

And restart denyhost:

sudo service denyhosts restart

Configuration denyhosts.

After the installation process, the DenyHosts is ready to work, but some 

configuration options are noteworthy. All configuration values are in the file 

/etc/denyhosts.conf

1. PURGE_DENY: removed HOSTS_DENY entries that are older than this 

time.

Default value is empty and all listed ip-addresses remain in the file 

/etc/hosts.deny forever.

2. BLOCK_SERVICE: the service name that should be blocked in 

HOSTS_DENY

Default value = sshd, By default, the service protects only ssh server, but it 

can be customized to other services.

3. ADMIN_EMAIL: if you would like to receive report from denyhosts.

Default value = root@localhost, but you can change this value to your email.

After changing the settings, you should restart the DenyHosts again:

sudo service denyhosts restart

The result of  DenyHosts you can see in the file:  /etc/hosts.deny

Provided by:Forthscale systems, cloud experts

Backup to AWS S3 with duply on Ubuntu

1. Install duply and dependences (as root).
sudo apt-get install duplicity duply python-boto

2. Create backup profile (as root).  
sudo duply bck_profile create

3. Edit profile /root/.duply/bck_profile/conf like as:
GPG_PW='___YOUR GENERATED PASSWORD____'
TARGET='s3://s3-endpoint-name.amazonaws.com/bucket-name/directory/for/backup/'
TARGET_USER='AWS_KEY_ID'
TARGET_PASS='AWS_SECRET_KEY'
SOURCE='/'
MAX_AGE=6M
#MAX_FULL_BACKUPS=1
#VERBOSITY=5
TEMP_DIR=/content
#DUPL_PARAMS="$DUPL_PARAMS --time-separator _ "
#DUPL_PARAMS="$DUPL_PARAMS --short-filenames "
MAX_FULLBKP_AGE=1W
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
VOLSIZE=200
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

4. Edit /root/.duply/bck_profile/excludes to include the list of non wanted directories such as:
- /dev
- /home/*/.cache
- /home/*/.ccache
- /lost+found
- /media
- /mnt
- /proc
- /root/.cache
- /root/.ccache
- /run
- /selinux
- /sys
- /tmp
- /var/cache/*/*
- /var/log
- /var/run
- /var/tmp

5. Tun a full backup test:
sudo duply bck_profile backup

6. Configure crone for daily backup:
sudo crontab -e 
and  add:
0 2 * * * duply bck_profile backup

7. Some useful commands to operate your backup:
To get current list of present backups.
sudo duply status
To restore all files from last backup to /target_dir
sudo duply restore /target_dir
To restore directory etc from last backup to /target_dir
sudo duply fetch etc /target_dir

Note:
 All backups early than MAX_FULLBKP_AGE=1W will be incremental, every MAX_FULLBKP_AGE=1W will create full backup.

Provided by:Forthscale systems, cloud experts

PPTP server on AWS Ubuntu instance

Simple VPN server configuration for easy VPN access to AWS using built in Windows VPN client.

First to install pptp server package.

sudo apt-get install pptpd

Now to configuration:

edit /etc/pptpd.conf

option /etc/ppp/pptpd-options

logwtmp

localip 172.20.0.1 

remoteip 172.20.0.10-50

edit  /etc/ppp/pptpd-options

refuse-pap

refuse-chap

refuse-mschap

require-mschap-v2

require-mppe-128

ms-dns 8.8.8.8

ms-dns 8.8.4.4

proxyarp

nodefaultroute

lock

nobsdcomp

mtu 1420

mru 1420

edit  /etc/ppp/chap-secrets 

# client        server  secret                  IP addresses

client1      pptpd   secret1      *

client2      pptpd   secret2      *

Add to /etc/rc.local

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

iptables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

iptables -t mangle -A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Run:

/etc/rc.local

service pptpd restart

Done.

Provided by:Forthscale systems, cloud experts

fixing missing apt-add-repository command

You might need to use apt-add-repository script and will get a missing command error. For example:
sudo: apt-add-repository: command not found

in order to fix that you nred to install  python-software-properties package

# apt-get install python-software-properties

That's it.

Provided by:Forthscale systems, cloud experts

Making a service autostart on Ubuntu, Debian, Oralce Linux, CentOs and Red Hat (Fedora and RHEL)

For RedHat based distributions (CentOS, Oracle Linux, Fedora) execute (as root):

chkconfig servicename on


For Debian based distributions (Ubuntu) execute (as root):

update-rc.d servicename defaults

Provided by: Forthscale systems, Linux support team

Setting up Amazon AWS EC2 ftp server with Linux and VSFTP:

Install vsftp (example for Ubuntu / Debian)

apt-get -y install vsftpd

Edit configuration file (in our example with local authentication and no guest user)

vi /etc/vsftpd.conf

write_enable=YES
anonymous_enable=NO
local_umask=022
local_enable=YES

#to add passive ftp:
pasv_enable=YES
pasv_max_port=12100
pasv_min_port=12000
port_enable=YES
pasv_address="your external instance ip or address"

and open inbound port range 20-21 and 12000-12100 in your security groups

Provided by: ForthScale systems, scalable infrastructure experts

Debian / Ubuntu: purge removed packages with apt

Removing packages with aptitude or apt-get keeps some configuration and temp files on disk.
to purge them (to get rid of these configuration files) execute:

dpkg -l |awk ‘/^rc/ {print $2}’ |xargs sudo dpkg --purge

Powered by 123ContactForm | Report abuse

Provided by: ForthScale systems, scalable infrastructure experts

fixing E: Archive directory /var/cache/apt/archives/

During apt-get install of a package you might receive message :
E: Archive directory /var/cache/apt/archives/

to fix it you need to create a directory and repair deb packages database:

mkdir -p /var/cache/apt/archives/partial
apt-get autoclean

you should get response output similar to:
Reading package lists... Done
Building dependency tree
Reading state information... Done

then feel free to install you package with apt-get or aptitude


Powered by 123ContactForm | Report abuse

List open ports and listening services in Linux

To list open network ports (sockets) and the processes that run them on any Linux distribution use netstat. You might need to install it via yum / apt-get.
Syntax:
netstat -lnptu

Powered by 123ContactForm | Report abuse

Provided by: SiQ systems, Cloud experts

Mounting S3 bucket as a file system on Linux (Ubuntu)

In order to mount S3 bucket on Linux you need to install ssh file system (FUSE) libraries first.

Install: aptitude install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev comerr-dev libfuse2 libidn11-dev libkadm55 libkrb5-dev libldap2-dev libselinux1-dev libsepol1-dev pkg-config fuse-utils sshfs

Follow the instantiations in S3fs WIKI:

https://code.google.com/p/s3fs/wiki/InstallationNotes

It was using http download, not SVN.

Create a mount point for the new file system (ex. mkdir /mnt/s3)

Mount the file system:

ex.: s3fs tmpname -o use_cache=/tmp -o allow_other /mnt/s3

this mount comes with option of using a tmp as a cache for S3 content, you can clean that cache as you feel fit.

You can also add your bucket to /etc/fstab

s3fs#mybucket /mnt/s3 fuse allow_other,url=https://s3.amazonaws.com 0 0

Update for mounting S3 bucket on CentOS 5.5 and other old distros.
CentOS 5.5 comes with old FUSE version 2.7.4 as latest in it`s repository.
You need to manually compile version 2.8.4 befor you can use S3fs.

Remove old FUSE
yum remove fuse fuse* fuse-devel

Install needed libraries:
yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap

Download new FUSE from the Sourceforge:
wget "https://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz?r=&ts=1299709935&use_mirror=cdnetworks-us-1"

Untar and compile:

tar -xzvf fuse-2.8.4.tar.gz
cd fuse-2.8.4
./configure --prefix=/usr
make; make install


Configure FUSE in the system:
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
ldconfig
modprobe fuse

Confirm that 2.8.4 is the version of FUSE displayed :
pkg-config --modversion fuse

Proceed with regular S3FS installation.

Have questions? Just contact us right away and we will be happy to assist

Running manual NTP (network time protocol) In Ubuntu Linux

If your Ubuntu is out of time sync you can run a manual comand to syncronize the clock. command is:
ntpdate
Running ntpdate with no options / switches will couse it to try to sync with default servers. If none are set you will get an error similar to:
ntpdate[15110]: no servers can be used, exiting
then run it with servers address.
for example:
ntpdate europe.pool.ntp.org north-america.pool.ntp.org
you might need to run sudo if you are not system root:
sudo ntpdate europe.pool.ntp.org north-america.pool.ntp.org
P.S. Server pool for Israel is:
il.pool.ntp.org

Provided by:SiQ systems, Cloud experts

MySQL and debian-sys-maint account

If for some reason your debian-sys-maint account is unset in MySQL, during MySQL start you will have connect to server error. You might need to set this account in the MySQL server itself.
cat /etc/mysql/debian.cnf to get the password value from the file.
Then connect to the DB
mysql -u root -p
and execute:
GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY 'YOUR_PASSWORD' WITH GRANT OPTION