Useful SSH fingerprint check

http://blog.binarymist.net/2013/02/16/establishing-your-ssh-servers-key-fingerprint/

To save you going there, here is the info;

Establishing your SSH Server’s Key Fingerprint

When you connect to a remote host via SSH that you haven’t established a trust relationship with before,
you’re going to be told that the authenticity of the host your attempting to connect to can’t be established.

me@mybox ~ $ ssh me@10.1.1.40

The authenticity of host '10.1.1.40 (10.1.1.40)' can't be established.

RSA key fingerprint is 23:d9:43:34:9c:b3:23:da:94:cb:39:f8:6a:95:c6:bc.

Are you sure you want to continue connecting (yes/no)? y

Please type 'yes' or 'no':

Do you type yes to continue without actually knowing that it is the host you think it is? Well, if you do, you should be more careful. The fingerprint that’s being put in front of you could be a Man In The Middle (MITM). You can query the target (from “it’s” shell of course) for the fingerprint of it’s key easily. On Debian you’ll find the keys in /etc/ssh/

On

ls /etc/ssh/

you should get a listing that reveals the private and public keys. Run the following command on the appropriate key to reveal it’s fingerprint. For example if SSH is using rsa:

ssh-keygen -lf ssh_host_rsa_key.pub

For example if SSH is using dsa:

ssh-keygen -lf ssh_host_dsa_key.pub

If you try the command on either the private or publick key you’ll be given the public key’s fingerprint, which is exactly what you need for verifying the authenticity from the client side.

Do not connect remotely and then run the above command, as the machine you’re connected to is still untrusted. The command could be dishing you up any string replacement if it’s an attackers machine. You need to run the command on the physical box or get someone you trust (your network admin) to do this and hand you the fingerprint.

Now when you try to establish your SSH connection for the first time, you can check that the remote host is actually the host you think it is by comparing the output of one of the previous commands with what SSH on your client is telling you the remote hosts fingerprint is. If it’s different it’s time to start tracking down the origin of the host masquerading as the address your trying to hook up with.

Now, when you get the following message when attempting to SSH to your server, due to something or somebody changing the hosts key fingerprint:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that a host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

23:d9:43:34:9c:b3:23:da:94:cb:39:f8:6a:95:c6:bc.

Please contact your system administrator.

Add correct host key in /home/me/.ssh/known_hosts to get rid of this message.

Offending RSA key in /home/me/.ssh/known_hosts:6

remove with: ssh-keygen -f "/home/me/.ssh/known_hosts" -R 10.1.1.40

RSA host key for 10.1.1.40 has changed and you have requested strict checking.

Host key verification failed.

The same applies. Check that the fingerprint is indeed the intended target hosts key fingerprint. If it is, run the specified command.

Application Support Analyst Generic Steps

For those new to or about to start their carrier as an application support person, here are some guidelines for investigating an application issue.

User Side Checks

1. Make sure they have the correct connections for the application, hostname, IP address, port number, etc.
2. Check that the users environment to ensure any application environment variables are set correctly.
3. Check the routing tables on the users system. Make sure their default gateway is correct.
4. Ensure that their firewall (especially Windows) isn't blocking the application connectivity. Desktop applications may sometimes be blocked.

Network Checks

1. Can you access the application?
   Use a web browser or the application (you may want to check network traffic with something like wireshark or fiddler)
2. Are you able to access the application from different locations? E.g Web browser or from a different host using something like ncat or Telnet.

• telnet hostOrIP appPortNo
• ncat hostOrIP appPortNo

1. Are you able to connect to another service running on that host, e.g. ping, or an Email service?
2. Check that the load balancers are able to connect to the application and that the load balancers are configured correctly.
3. Check with nmap to identify what ports are available on the host.  You will need a more intrusive view rather than just the basic check as it may well miss your applications port.  Or you can tell nmap which port to check.

The use of ncat would allow you to see if you can connect to the specific ports for an application response. Also these steps should allow you to see if connection is possible, or whether a firewall or switch is causing issues, so you should try from different networked hosts.
However, connection to an application port may not be enough to confirm it is working, so you should be able to send it a message to check that it responds since this will tell us if the process has hung.

Checks on the application host

1. Log on to the host and check that the process is still running. You'd be surprised how often its not if the connection is down.
   ps -ef
2. Check the log files for any application warnings or errors. If you have a Java stack trace ask a developer to get involved and perhaps send a copy of the stack trace to them if it's not obvious what the issue might be.
   egrep -ir 'ERR|WARN' /var/log/*
3. Check that the firewall is allowing the application port through. This will vary depending on the operating system. You can also use nmap here on the host to see what ports are available or if the application port is open.  For Red Hat systems and those using iptables then iptables -L should help.
4. Check that the host can connect to any load balancers that it may be attached to, especially for return information.
5. Check that the network interfaces are responding, especially if there is more than one.
6. Check the routing tables of this host just in case there is one route in and another out. Sometimes routing loops or mis-configuration of the network makes traffic take a different route back causing the users application to refuse or not see the return data.
7. Ensure that the host can connect to the database server port. There could be a connectivity issue here. Again, similar checks to connecting to this host from a remote location should be used to ensure connectivity to the database is available.  The key thing here is that in the production environment you will only have a limited set of hosts to use to check that connectivity to the DB server is there as they may be on different VLANs.

Final Steps

1. Check the support documentation for similar symptoms
2. Never be afraid to ask someone for help. Someone may have already seen this issue before.
3. Gather as much information as possible before speaking to the next level support making sure you have;

• Network connectivity information from all sides (user, application server, other hosts in different networks, load balancers and firewalls)
• Log information if you are able to access the logs
• Users configuration information (gateway, variables, firewall)

Disclaimer

This is a starting point, and I'm sure there are other techniques and pieces missing, but in most cases the generics here will provide a good basis for acquiring the relevant information to diagnose issues with an application. Other methods and ideas to build this to a useful learning platform for people coming in to the application support arena would be welcome.