Application Support Analyst Generic Steps

For those new to or about to start their carrier as an application support person, here are some guidelines for investigating an application issue.

User Side Checks

1. Make sure they have the correct connections for the application, hostname, IP address, port number, etc.
2. Check that the users environment to ensure any application environment variables are set correctly.
3. Check the routing tables on the users system. Make sure their default gateway is correct.
4. Ensure that their firewall (especially Windows) isn't blocking the application connectivity. Desktop applications may sometimes be blocked.

Network Checks

1. Can you access the application?
   Use a web browser or the application (you may want to check network traffic with something like wireshark or fiddler)
2. Are you able to access the application from different locations? E.g Web browser or from a different host using something like ncat or Telnet.

• telnet hostOrIP appPortNo
• ncat hostOrIP appPortNo

1. Are you able to connect to another service running on that host, e.g. ping, or an Email service?
2. Check that the load balancers are able to connect to the application and that the load balancers are configured correctly.
3. Check with nmap to identify what ports are available on the host.  You will need a more intrusive view rather than just the basic check as it may well miss your applications port.  Or you can tell nmap which port to check.

The use of ncat would allow you to see if you can connect to the specific ports for an application response. Also these steps should allow you to see if connection is possible, or whether a firewall or switch is causing issues, so you should try from different networked hosts.
However, connection to an application port may not be enough to confirm it is working, so you should be able to send it a message to check that it responds since this will tell us if the process has hung.

Checks on the application host

1. Log on to the host and check that the process is still running. You'd be surprised how often its not if the connection is down.
   ps -ef
2. Check the log files for any application warnings or errors. If you have a Java stack trace ask a developer to get involved and perhaps send a copy of the stack trace to them if it's not obvious what the issue might be.
   egrep -ir 'ERR|WARN' /var/log/*
3. Check that the firewall is allowing the application port through. This will vary depending on the operating system. You can also use nmap here on the host to see what ports are available or if the application port is open.  For Red Hat systems and those using iptables then iptables -L should help.
4. Check that the host can connect to any load balancers that it may be attached to, especially for return information.
5. Check that the network interfaces are responding, especially if there is more than one.
6. Check the routing tables of this host just in case there is one route in and another out. Sometimes routing loops or mis-configuration of the network makes traffic take a different route back causing the users application to refuse or not see the return data.
7. Ensure that the host can connect to the database server port. There could be a connectivity issue here. Again, similar checks to connecting to this host from a remote location should be used to ensure connectivity to the database is available.  The key thing here is that in the production environment you will only have a limited set of hosts to use to check that connectivity to the DB server is there as they may be on different VLANs.

Final Steps

1. Check the support documentation for similar symptoms
2. Never be afraid to ask someone for help. Someone may have already seen this issue before.
3. Gather as much information as possible before speaking to the next level support making sure you have;

• Network connectivity information from all sides (user, application server, other hosts in different networks, load balancers and firewalls)
• Log information if you are able to access the logs
• Users configuration information (gateway, variables, firewall)

Disclaimer

This is a starting point, and I'm sure there are other techniques and pieces missing, but in most cases the generics here will provide a good basis for acquiring the relevant information to diagnose issues with an application. Other methods and ideas to build this to a useful learning platform for people coming in to the application support arena would be welcome.

Blackberry outage and iPhone release: Coincidence?

Is it just me, or coincidence that on both releases of the iPhone last year and this year that Blackberry's internet connectivity just happened to be trashed prior to or on the day of the iPhone release?
Is there some fowl play going on somewhere?

• 10th October 2011 - Blackberry suffer a network outage
• 14th October 2011 - iPhone 4s goes on Sale
• 18th September 2012 - Blackberry suffer a network outage
• 21st September 2012 - iPhone 5 goes on sale

Now if that's not a marketing ploy then I don't know what is. If I was Blackberry I'd be looking at who the cause of the outage is rather than what, and I'd find out when Apple plan to release the next iPhone and if it occurs again would draw up a law suit if I was them as this is too close for comfort.

Next time Apple release a new iPhone I will be watching the Blackberry network with anticipation and if another outage occurs (since the rest of the year the network is fine), then it can only be assumed that Apple are causing it to make Blackberry customers think that they should get an iPhone (even though the tech in the iPhone is at least a year old by the time Apple release them).

BASH issues that break the rules

I always thought BASH was too complex for it's own good. Unlike the commercial KornShell, BASH has things that break the rules of a good shell, such as forking a process for a built-in command, or caching commands instead of tracking them as aliases.

As I find some other annoyances with the Bourne Again Shell I'll list them in here and the reasons.

1. Caching commands.
   Instead of tracking aliases to the real command BASH decides to use a different command (built-in) to deal with tracking commands that you have already run.  In the KornShell you would normally run alias -t to see what commands have been tracked when you install a new command and the old version is still running.  However, with BASH you have to use the hash built-in command to identify what commands are being tracked.  The -l or -t option will display a list of the commands tracked, whilst the -r allows you to remove the tracked command(s).
2. Piping a while loop in BASH forces the built-in into a sub-shell, which means that any variables that you create inside the while loop body will be lost after the execution of the loop ends.  In KSH the while loop remains part of the shell process that is running, so you have access to the variables created in the loop once the loop execution ends.  Try the following code in both shells, and notice the major issue that the BASH generates which breaks all the rules on shell built-ins.
   
   cat /etc/passwd | while read line
   do
       if echo $line | grep steve >/dev/null 2>&1
       then
           newvar=$line
       fi
   done
   echo "This will only show in KSH: $newvar" 

 

Play On Linux - Microsoft software in Wine the easy way

Just come across this tool for Linux, and testing it out with Office 2007 at the moment.

http://www.playonlinux.com/en/download.html

Install the repo, then install the playonlinux package.

Once installed run playonlinux, follow the first time run steps.

Once installed then you can install applications and have it perform the wine configuration for you.

vlc-2.0.3-1.fc17.x86_64 main libvlc error

Having recently upgraded from F14 to F17, I found 1 small issues with my update on VLC.  F17 was so easy to install and on my HP DV3 pretty much worked, apart from the regular kernel line hack acpi_osi= to make the display come up.

VLC on the other hand decided that on my laptop didn't want to work for my account, but did on another user account.

When I ran it from the command line I received the following error;



VLC media player 2.0.3 Twoflower (revision 2.0.2-93-g77aa89e)
[0xa51bd8] main interface error: no suitable interface module
[0x97f108] main libvlc error: interface "globalhotkeys,none" initialization failed
[0x97f108] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Segmentation fault (core dumped)

On further investigation into my environment I comment out my LD_LIBRARY_PATH that was currently set in my .bash_profile, logout and back in and hey presto VLC started to work again.

The trick here was the Segmentation fault at the end and the "no suitable interface module"

Installing Sybase 12.5 ASE on CentOS 5.2 and 5.5

INSTALLING LINUX

1. Install a minimal version of CentOS 6 You will require 3 partitions 
2. To aid set up ensure that SELinux is disabled and iptables is off 
3. Set up kernel parameters in /etc/sysctl.conf;
            kernel.exec-shield=0
            kernel.randomize_va_space=0
            kernel.shmmax=67108864 # Only if tis number is bigger than the default
4. Update kernel parameters;
            sysctl -p /etc/sysctl.conf
5. Ensure that you have 3 partitions, or seperate disks
6. export LD_POINTER_GUARD=0
7. Install the following rpms;
           yum install compat-libstdc++-296
   For this you might want to install the rpmfusion repository
           Do this by downloading the rpm and the epel rpm
           Install using rpm -i rpmfusion epel

INSTALLING SYBASE

1. Extract your Sybase tgz file
2. Run the setup command
           ./setup -is:javaconsole -console

                         Use the -options-record

                         Where filename will record your options, note that this option can only be used in GUI mode.

Follow the on screen instructions, which includes agreeing to the T&Cs, deciding where you want to install Sybase, create the directory, typical/full or custom.

If installing client only then use custom and select the following;

        Select everything except Sybase Server

Creating the first database

1. Now we need to create your first database

        cd /opt/sybase

        You will find example resource files in ASE-/init/sample_resource_files

        Select the one appropriate to your install

        (This example will be for the adaptive_server.rs)

2. The resource file

        Copy the srvbuild.adaptive_server.rs to /opt/sybase/ase.rs

        Edit the file as follows;

                sybinit.release_directory: /opt/sybase   (or where you installed it)

                sqlsrv.server_name: STEVE       (SYBASE logical name of the database. This will be the name in the interfaces file, server instance)

                sqlsrv.network_hostname_list: localhost (the real hostname of your server)

                sqlsrv.network_port_list: 4000  (The port number you wish to run this instance on)

                sqlsrv.master_device_physical: /dev/sdb1        (physical partition or filename for master database)

                sqlsrv.sybsystemprocs_device_physical_name: /dev/sdb2

                sqlsrv.sybsystemdb_device_physical_name: /dev/sdb3      (remember this applies to all databases created within this instance)

                sqlsrv.default_backup_server:

3. Update root's enviroment using the SYBASE.sh script

        . ./SYBASE.sh

4. srvbuildres -r ase.rs

5. If the above command results in "Server 'STEVE' was successfully created'

        find . -name RUN*

        Should return /opt/sybase/ASE-12_5/install/RUN_STEVE

6. Test your connection

        isql -Usa -S STEVE

   You should be prompted for a password, just press ENTER

   You should eventually get the   1>   prompt

   Type quit and press enter

Creating the start up script so that SYBASE starts on reboot

1. vi /etc/init.d/sybase

2. Add the following to the file;

#!/bin/bash

# chkconfig: 35 95 95

# description: Start sybase

case $1 in

        'start')

                unset LANG

                export LD_POINTER_GUARD=0

                . /opt/sybase/SYBASE.sh

                startserver -f /opt/sybase/ASE-12_5/install/RUN_STEVE

                ;;

        'stop')

                pkill dataserver

                ;;

esac

3. Save the script and then do;

        chmod +x /etc/init.d/sybase

4. Add the script to the start up and shutdown sequence

        chkconfig --add sybase

5. Test the script;

        - Shutdown the server

HP DV3-4300sa Fedora 14 Monitor Setup

Having finally finding some time to sit with my new HP Laptop I have managed to get the screen to work during boot up, and the Wifi connection to work. (Now just have to see about the intermittent fault on the fixed ethernet port).

Getting the graphics working
0. Install the ati-driver-installer-11-8-x86.x86_64.run driver direct from the support.amd.com web site - http://support.amd.com/us/Pages/AMDSupportHub.aspx
- Category: Notebook Graphics
- Product Line: Radeon HD Series
- Product Model: Radeon HD 6xxxM Series
- Operating System: Linux x86_64

1. Remove from /etc/grub.conf kernel line
- rhgb

2. Add to /etc/grub.conf kernel line
- video=VGA-1:e vga=0x318

3. Added /etc/X11/xinit/xinitrc.d/45custom_xrandr-settings

#!/bin/bash
xrandr --addmode VGA1 1366x768
xrandr --output LVDS1 --mode 1366x768 --fb 1366x768
xrandr --output VGA1 --mode 1366x768 --fb 1366x768

4. chmod +x /etc/X11/xinit/xinitrc.d/45custom_xrandr-settings

5. /etc/X11/xorg.conf

Section "ServerLayout"
Identifier "aticonfig Layout"
Screen 0 "aticonfig-Screen[0]-0" 0 0
Screen 1 "LCDPanel"
EndSection

Section "Module"
EndSection

Section "Monitor"
Identifier "aticonfig-Monitor[0]-0"
Option "VendorName" "ATI Proprietary Driver"
Option "ModelName" "Generic Autodetecting Monitor"
Option "Enable" "true"
Option "DPMS" "true"
ModeLine "1366x768" 69.30 1366 1398 1420 1474 768 770 774 784
EndSection

Section "Monitor"
Identifier "LCDPanel"
Option "Enable" "true"
Option "DPMS" "true"
ModeLine "1366x768" 69.30 1366 1398 1420 1474 768 770 774 784
EndSection

Section "Device"
Identifier "aticonfig-Device[0]-0"
Driver "fglrx"
Option "DesktopSetup" "clone"
Option "Centermode" "off"
Option "monitor-VGA" "aticonfig-Monitor[0]-0"
Option "monitor-LVDS" "LCDPanel"
BusID "PCI:1:0:0"
EndSection

Section "Device"
Identifier "LCDPanel"
Driver "intel"
Option "DesktopSetup" "clone"
Option "monitor-LVDS" "LCDPanel"
Option "monitor-VGA" "aticonfig-Monitor[0]-0"
BusID "PCI:0:2:0"
EndSection

Section "Screen"
Identifier "aticonfig-Screen[0]-0"
Device "aticonfig-Device[0]-0"
Monitor "aticonfig-Monitor[0]-0"
DefaultDepth 24
SubSection "Display"
#Viewport 0 0
Depth 24
#Modes "1366x768" "1024x768"
EndSubSection
EndSection

Section "Screen"
Identifier "LCDPanel"
Device "LCDPanel"
Monitor "LCDPanel"
DefaultDepth 24
SubSection "Display"
#Viewport 0 0
Depth 24
#Modes "1366x768" "1024x768"
EndSubSection
EndSection

Making Wifi Work
1. Edit the file /etc/rc.local and add the following lines to it

modprobe lib80211
insmod /lib/modules/`uname -r`/kernel/drivers/net/wireless/wl.ko
modprobe b43