Monday, February 11, 2013

Application Support Analyst Generic Steps

For those new to or about to start their carrier as an application support person, here are some guidelines for investigating an application issue.

User Side Checks


  1. Make sure they have the correct connections for the application, hostname, IP address, port number, etc.
  2. Check that the users environment to ensure any application environment variables are set correctly.
  3. Check the routing tables on the users system. Make sure their default gateway is correct.
  4. Ensure that their firewall (especially Windows) isn't blocking the application connectivity. Desktop applications may sometimes be blocked.

Network Checks


  1. Can you access the application?
    Use a web browser or the application (you may want to check network traffic with something like wireshark or fiddler)
  2. Are you able to access the application from different locations? E.g Web browser or from a different host using something like ncat or Telnet.

  • telnet hostOrIP appPortNo
  • ncat hostOrIP appPortNo

  1. Are you able to connect to another service running on that host, e.g. ping, or an Email service?
  2. Check that the load balancers are able to connect to the application and that the load balancers are configured correctly.
  3. Check with nmap to identify what ports are available on the host.  You will need a more intrusive view rather than just the basic check as it may well miss your applications port.  Or you can tell nmap which port to check.
The use of ncat would allow you to see if you can connect to the specific ports for an application response. Also these steps should allow you to see if connection is possible, or whether a firewall or switch is causing issues, so you should try from different networked hosts.
However, connection to an application port may not be enough to confirm it is working, so you should be able to send it a message to check that it responds since this will tell us if the process has hung.

Checks on the application host


  1. Log on to the host and check that the process is still running. You'd be surprised how often its not if the connection is down.
    ps -ef
  2. Check the log files for any application warnings or errors. If you have a Java stack trace ask a developer to get involved and perhaps send a copy of the stack trace to them if it's not obvious what the issue might be.
    egrep -ir 'ERR|WARN' /var/log/*
  3. Check that the firewall is allowing the application port through. This will vary depending on the operating system. You can also use nmap here on the host to see what ports are available or if the application port is open.  For Red Hat systems and those using iptables then iptables -L should help.
  4. Check that the host can connect to any load balancers that it may be attached to, especially for return information.
  5. Check that the network interfaces are responding, especially if there is more than one.
  6. Check the routing tables of this host just in case there is one route in and another out. Sometimes routing loops or mis-configuration of the network makes traffic take a different route back causing the users application to refuse or not see the return data.
  7. Ensure that the host can connect to the database server port. There could be a connectivity issue here. Again, similar checks to connecting to this host from a remote location should be used to ensure connectivity to the database is available.  The key thing here is that in the production environment you will only have a limited set of hosts to use to check that connectivity to the DB server is there as they may be on different VLANs.

Final Steps


  1. Check the support documentation for similar symptoms
  2. Never be afraid to ask someone for help. Someone may have already seen this issue before.
  3. Gather as much information as possible before speaking to the next level support making sure you have;

  • Network connectivity information from all sides (user, application server, other hosts in different networks, load balancers and firewalls)
  • Log information if you are able to access the logs
  • Users configuration information (gateway, variables, firewall)

Disclaimer

This is a starting point, and I'm sure there are other techniques and pieces missing, but in most cases the generics here will provide a good basis for acquiring the relevant information to diagnose issues with an application. Other methods and ideas to build this to a useful learning platform for people coming in to the application support arena would be welcome.
Posted by at