Setup, problem and solution design.
Purpose of log aggregation is to develop single point of access for servers data (in our case nginx web servers).
We have a lot of web servers writing off huge amount of log and no real way to understand what is going on there. Initial solution was to have each systems write a local log file with a Munin agent with custom Perl parser transferring data to Munin server there it was displayed as an RRDtool graph. It worked, however servers themselves generated a lot of logged data making it impossible to parse close to real time forcing us to drop out significant amount of data.
After making a small internet research and due to budget constrains we decided to go with open source tools only. Those applications however still had to be high volume, high load, scalable and big data supporting.
We have decided to setup a dedicated loghost and ship all the data to it parsing it on spot to a needed results. Another thing our proposed solution took into consideration was future log indexing for both technical and BI search ability /readability.
Proposed solution consisted of:
Logstash - Tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching) . Elasticsearch - Distributed, RESTful, Search Engine Graylog 2 - Software to run analytics, alerting, monitoring and powerful searches over your whole log.
As a bonus, Logstash gave us possibility to export events to a monitoring system or support shift management.
login to mysql as a root: mysql -uroot -p
mysql> show processlist;
will show you list of processes running in MySQL
and using \G delimiter will present processes in more readable format.
mysql> show processlist \G
Sometimes you need to pass system variables to applications running in tomcat environment. those could b for example configuration files. Trying to pass them to start-up script as variables such as
will set it in the system shell and will not pass it to the tomcat on it`s start since it runs as a different user.
Will return: NULL value.
You need to edit relevant tomcat.conf (tomcat6.conf for example in /etc)
to the end of file
You can check that variable s set with System.getEnv("Your.var") in you java code.
Both tools are great to enable external GUI remote access on Linux.
To install NX Free server: Note. Installation of NX Server requires installation of all three packages: client, node and server due to tools and libraries dependencies. Download the packages from Nomachine: wget http://184.108.40.206/download/3.5.0/Linux/nxclient-3.5.0-7.x86_64.rpm wget http://220.127.116.11/download/3.5.0/Linux/nxnode-3.5.0-9.x86_64.rpm wget http://18.104.22.168/download/3.5.0/Linux/FE/nxserver-3.5.0-11.x86_64.rpm
Then you are unable to connect to your ssh server with your PEM (or PPK) key
with connection refused message check the /var/log/auth.log file.
If you see something similar to: sshd: Authentication refused: bad ownership or modes for directory /Your/Home/Path
You have a home directory permission problem.
change to that user if not already one and execute: chmod go-w ~/ chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys
It will restore your permissions and you should be able to connect.
You can connect different storage types to Amazon EC2 instances, two of them provided naively by Amazon platform and the rest are either provided by external sources or tweaks. In any EC2 instance (except micro) you have instance storage included in a package. You can also use elastic block storage (EBS) and have an option connecting different 3rd party storage over the network.
Instance storage is a fast non-persistent storage provided by Amazon. It means it will revert to it`s original state after any system shutdown, erasing any changes you have applied to the file-system. It is very useful for running “dumb” servers that do not store data locally or as an additional storage for temporary files.
Elastic block storage (EBS) is a persistent storage provided by Amazon. All and any data stored on it is available after instance shutdown and can be manipulated with on device level. For example you can detach an EBS volume from one instance and attach it to another. However EBS can not be attached to more then once instance at the same time.
Using S3 as a file system. S3 is a storage infrastructure provided by Amazon as a service, it is not a part of EC2 (Elastic cloud) but can be used to store and retrieve any amount of data from anywhere at any time. Because S3 infrastructure is fully managed and scaled by Amazon it is very useful for large scale web projects, backup media and large volume data transfers. Using S3 as a file system is done via FUSE in Linux or as a mapped network drive in MS Windows. We are providing a tutorial on Linux implementation of S3 as a file system .
There are also few companies out there providing iSCSI storage arrays for AWS, one of them is Zadara storage, company providing Virtual Private Storage Arrays and currently in beta stage. iSCSI as any other network attached storage systems will incorporate persistence and availability of EBS at much faster speed.
Sometimes you need to preconfigure your servers for dns resolution and you are not sure what local DNS addresses will be or just to use backup dns servers for resolution. Here is small list of globally accessible DNS servers: