mounting AWS (Amazon Web Services) EFS on Linux Ubuntu 18.04

Amazon Elastic File System (Amazon EFS) is a scalable file storage for EC2 and services that run on EC2 (for example Kubernetes clusters). The device is accessible on Linux via the NFS protocol and can be used my multiple instances and pods at the same time.
For more information on EFS visit AWS documentation.


Step one: Gather information
In our case ti is pretty straightforward. Ubuntu instance in the same VPC as the EFS and a DNS name of the file system we want to access. The format uses following convention:

http://file-system-id.efs.aws-region.amazonaws.com

And the exact URL is available on AWS console AWS home under filesystem's DNS name or via cli

Step two: Install the NFS Client for Linux

$ sudo apt-get update
$ sudo apt install nfs-kernel-server

Step three: Mount the file system on EC2 instance.
Create (if you don't have already) a mount point for the EFS

$ sudo mkdir -p /mnt/efs-mount-point

Mount the EFS share on the instance

$ sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport mount-target-DNS:/   /mnt/efs-mount-point

Now we have a mounted Amazon EFS file system on Ubuntu EC2 instance.
Keep in mind that command mounted doesn't persist across reboots. if you want it to be permanently accessible you have to add it to the fstab.


Common error:

efs mount.nfs: Connection timed out

This error can occur because either the Amazon EC2, mount target security groups or file system access are not configured properly.

For more troubleshooting tips you can visit:
https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html

Getting AWS EC2 instance id (instanceid) from within the ec2 instance

In general you can get a lot of instance metadata by accessing API on
http://169.254.169.254/latest/meta-data/
That includes instance id.

On generic Linux system, you can get the ID either using curl:
curl http://169.254.169.254/latest/meta-data/instance-id
or wget:
wget -q -O - http://169.254.169.254/latest/meta-data/instance-id

If you instance is based on Amazon Linux or have cloud-utils installed you can also run:
ec2-metadata -i
for instance id.

more documentation on metadata is a available here:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Provided by:Forthscale systems, cloud experts

Backup to AWS S3 with duply on Ubuntu

1. Install duply and dependences (as root).
sudo apt-get install duplicity duply python-boto

2. Create backup profile (as root).  
sudo duply bck_profile create

3. Edit profile /root/.duply/bck_profile/conf like as:
GPG_PW='___YOUR GENERATED PASSWORD____'
TARGET='s3://s3-endpoint-name.amazonaws.com/bucket-name/directory/for/backup/'
TARGET_USER='AWS_KEY_ID'
TARGET_PASS='AWS_SECRET_KEY'
SOURCE='/'
MAX_AGE=6M
#MAX_FULL_BACKUPS=1
#VERBOSITY=5
TEMP_DIR=/content
#DUPL_PARAMS="$DUPL_PARAMS --time-separator _ "
#DUPL_PARAMS="$DUPL_PARAMS --short-filenames "
MAX_FULLBKP_AGE=1W
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
VOLSIZE=200
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

4. Edit /root/.duply/bck_profile/excludes to include the list of non wanted directories such as:
- /dev
- /home/*/.cache
- /home/*/.ccache
- /lost+found
- /media
- /mnt
- /proc
- /root/.cache
- /root/.ccache
- /run
- /selinux
- /sys
- /tmp
- /var/cache/*/*
- /var/log
- /var/run
- /var/tmp

5. Tun a full backup test:
sudo duply bck_profile backup

6. Configure crone for daily backup:
sudo crontab -e 
and  add:
0 2 * * * duply bck_profile backup

7. Some useful commands to operate your backup:
To get current list of present backups.
sudo duply status
To restore all files from last backup to /target_dir
sudo duply restore /target_dir
To restore directory etc from last backup to /target_dir
sudo duply fetch etc /target_dir

Note:
 All backups early than MAX_FULLBKP_AGE=1W will be incremental, every MAX_FULLBKP_AGE=1W will create full backup.

Provided by:Forthscale systems, cloud experts

PPTP server on AWS Ubuntu instance

Simple VPN server configuration for easy VPN access to AWS using built in Windows VPN client.

First to install pptp server package.

sudo apt-get install pptpd

Now to configuration:

edit /etc/pptpd.conf

option /etc/ppp/pptpd-options

logwtmp

localip 172.20.0.1 

remoteip 172.20.0.10-50

edit  /etc/ppp/pptpd-options

refuse-pap

refuse-chap

refuse-mschap

require-mschap-v2

require-mppe-128

ms-dns 8.8.8.8

ms-dns 8.8.4.4

proxyarp

nodefaultroute

lock

nobsdcomp

mtu 1420

mru 1420

edit  /etc/ppp/chap-secrets 

# client        server  secret                  IP addresses

client1      pptpd   secret1      *

client2      pptpd   secret2      *

Add to /etc/rc.local

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

iptables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

iptables -t mangle -A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Run:

/etc/rc.local

service pptpd restart

Done.

Provided by:Forthscale systems, cloud experts

Setting up Amazon AWS EC2 ftp server with Linux and VSFTP:

Install vsftp (example for Ubuntu / Debian)

apt-get -y install vsftpd

Edit configuration file (in our example with local authentication and no guest user)

vi /etc/vsftpd.conf

write_enable=YES
anonymous_enable=NO
local_umask=022
local_enable=YES

#to add passive ftp:
pasv_enable=YES
pasv_max_port=12100
pasv_min_port=12000
port_enable=YES
pasv_address="your external instance ip or address"

and open inbound port range 20-21 and 12000-12100 in your security groups

Provided by: ForthScale systems, scalable infrastructure experts

Fixing SSH Daemon - Authentication refused: bad ownership or modes for directory

Then you are unable to connect to your ssh server with your PEM (or PPK) key with connection refused message check the /var/log/auth.log file.
If you see something similar to:
sshd: Authentication refused: bad ownership or modes for directory /Your/Home/Path
You have a home directory permission problem.
change to that user if not already one and execute:
chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys 
It will restore your permissions and you should be able to connect.

 Provided by: SiQ systems, Cloud experts

Amazon AWS EC2 storage types

You can connect different storage types to Amazon EC2 instances, two of them provided naively by Amazon platform and the rest are either provided by external sources or tweaks.  In any EC2 instance (except micro) you have instance storage included in a package. You can also use elastic block storage (EBS) and have an option connecting different 3rd party storage over the network.

 

Instance storage is a fast non-persistent storage provided by Amazon. It means it will revert to it`s original state after any system shutdown, erasing any changes you have applied to the file-system. It is very useful for running “dumb” servers that do not store data locally or as an additional storage for temporary files.

 

Elastic block storage (EBS) is a persistent storage provided by Amazon. All and any data stored on it is available after instance shutdown and can be manipulated with on device level. For example you can detach an EBS volume from one instance and attach it to another. However EBS can not be attached to more then once instance at the same time.

 

Using S3 as a file system. S3 is a storage infrastructure provided by Amazon as a service, it is not a part of EC2 (Elastic cloud) but can be used to store and retrieve any amount of data from anywhere at any time. Because S3 infrastructure is fully managed and scaled by Amazon it is very useful for large scale web projects, backup media and large volume data transfers. Using S3 as a file system is done via FUSE in Linux or as a mapped network drive in MS Windows. We are providing a tutorial on Linux implementation of S3 as a file system .

 

There are also few companies out there providing iSCSI storage arrays for AWS, one of them is Zadara storage, company providing Virtual Private Storage Arrays and currently in beta stage. iSCSI as any other network attached storage systems will incorporate persistence and availability of EBS at much faster speed.

Taken from SiQ systems cloud glossary.

Setting Filezilla Server on Amazon EC2 instance with passive ftp

If you want to set Filezilla  ftp server to handle passive connections on your AWS EC2 instances, you should do the following.

Select non used TCP ports, for example 9024-9048 range

Configure firewalls:

In your AWS EC2 security group, allow the incoming connections on chosen ports:

tcp port 20
tcp port 21
tcp port 9024-9048

If using Windows firewall  on you instance, allow connections on same ports.

Now configure Filezilla to use specific port range on Passive connections:

Open Filezilla management console.

Got to: Edit > Settings > Passive Mode Settings

'External Server IP Address for passive mode transfers'

If you use AWS Elastic IP, enter it in "Use the following IP",
if not - use Filezilla provided web service with "Retrieve external IP address from" option.

Check 'Don't use external IP for local connections'

Check 'Use custom port range'

Enter chosen values (in our example) 9024 - 9048 for custom port range.

Powered by 123ContactForm | Report abuse

Provided by:SiQ systems, Cloud experts

allowing public access to amazon s3 bucket

You might need to give public access to bucket contents, for example if you use the bucket to load images for your site. Simple GUI option supplied by Amazon only gives access for a specific files but if you need a whole bucket, you have to use policy.

To edit policy, in GUI go to

'Bucket => Properties => Add Bucket Policy'

And add for example (just change yourbucketname with the name of your bucket:

{
    "Version": "2011-10-24",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::yourbucketname/*"
        }
    ]
}

save it and policy is applied.

Provided by:SiQ systems, Cloud experts