Then you are unable to connect to your ssh server with your PEM (or PPK) key with connection refused message check the /var/log/auth.log file.
If you see something similar to:
sshd: Authentication refused: bad ownership or modes for directory /Your/Home/Path
You have a home directory permission problem.
change to that user if not already one and execute:
chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
It will restore your permissions and you should be able to connect.
SiQ systems, Cloud experts