Troubleshooting: The human side of the technical process

I have a friend who is working on a new hardware deployment for a large bank.  He ran into a couple hiccups and called me for my opinion.  After reviewing his configuration and not seeing any obvious problems, I suggested he confirm that the hardware is working correctly.  He did the usual stuff such as swapping out cables, but still no love.  At that point, I suggested he open a support ticket with the vendor.

Probably like you, I've gone through the process to build what should be a simple configuration and discover that it doesn't work.  Then, I spend way too much time and energy troubleshooting it, only to find that the reason it didn't work was a hardware problem.  Other times, it's been an obscure command that has thrown me off.  When things don't work as expected, I do basic troubleshooting:

• Check the physical layer stuff (cables, connectors, check link lights)
• I audit my config for typos and human-error kinds of issues
• I do a quick Internet search to see if I'm dealing with a common issue
• If it involves a WAN connection, make sure the problem doesn't lie with your service provider (sometimes easier said than done)
• If I'm working as part of a team or if I have a friend I can lean on, I'll have someone else review my configs (kind of like a proofreader).

If none of those procedures solves the problem, I quit wasting time and bring in the specialists.

How much time should you spend?

The appropriate amount of time to spend troubleshooting varies based on the amount of time available, the size of the configuration(s), your comfort level with the technology in question, and the urgency of the issue.  It could be anywhere from 15 minutes to a few hours.  It's a judgement call.

Here's what I do:

• If I have to troubleshoot it for more than about 60 minues, I walk away from it for a while and go do something else.  Then I come back to it with a fresh set of eyes.
• If, after about 30 minutes, I still can't figure it out, I open a support ticket.  (These times are not hard-coded.  A lot of this depends on what else is going on at the time.)
• If it's not a work-related system, I'll spend as long as necessary researching it until I solve the problem or learn that it's some known-issue and can't be solved.
• If it's an issue with the home network and my wife can't get online, I become obsessed with the problem until it's solved.  (Some priorities are quite clear!)

Generalist or specialist?

I'm a generalist.  Perhaps you are, too.  I work with lots of different devices and software.  Although I have good depth of knowledge on the devices and software I work with most, I don't know as much about, say Cisco VPNs, as a Cisco specialist.  I've built enough working configs that setting up basic things is fairly routine.  I'm also comfortable setting up many unusual configurations.  When a client throws me a curveball, however, and I can't resolve the issue in a fairly short amount of time, I open a support case. 

Here's the way I look at it:

• My job is to deliver a working system.
• I have a broad enough and deep enough body of knowledge to understand how all the pieces should work together.
• I also have a broad enough and deep enough body of knowledge and experience to know how to do routine configs, plus many special configs.
• I can usually get things set up quickly.

When I can't get things figured out in a reasonable amount of time, I bring in a specialist to either help me identify a hardware problem or show me what I was missing in the configuration.

It's fun to troubleshoot and experiment, but on production systems we rarely have the luxury of leisurely exploring and experimenting.  It's not about the learning process in that situation, it's about getting the job done quickly and dependably.  I need to leave my substantial ego at the door and just get the job done.

Troubleshooting websites

Among the websites I like for troubleshooting are:

• The vendor's forums
• serverfault.com
• experts-exchange.com

While conducting training workshops, I've had students ask why they should purchase a support contract for their devices.  They reason that they'll learn what they need to know in my class or they'll be able to figure it out on their own after class.  I'd love to believe that is true, but the reality is, whether you take a two-day workshop, a five-day workshop, or a semester-long class, many of the systems we work with as IT pros have way too much depth and breadth for us to expect to know every aspect of their configuration.

What is your approach to troubleshooting?  What websites do you like when you need help?

Don R. Crawley is President/Chief Technologist at soundtraining.net, the Seattle IT training firm. A geek and nerdy kind of guy since sometime back in the 60s, today he pontificates at Computerworld as well as on his own blog, writes books for IT people, and speaks to IT groups on customer service, communication, and technology.