This is the next installment in the Archetypal Analytics Cycle ™ series. In the first post I listed the questions I ask any time I start a new analysis. In this post, I discuss the first question, namely, ‘what is the problem to be solved?’
This is not as straightforward as it might first appear. Too often, the objective is obscured by a myriad of disconnects and interpretive layers that devolve from the attempt to solve by constructing from the bottom up. In other words, the problem domain or question to be answered is not yet clearly articulated and instead, the typical (at least in IT) response is to begin asking for ‘report requirements.’ As I will cover in a later post, visual analytics is not reporting. Its objective is fundamentally different.
Let me give you an example. I was working on a client site one time. I was conducting a JAD (joint ‘application’ development) session with about 15 people. I asked them to tell me what problem they were trying to solve. They spent about 90 minutes chasing around in circles as they tried to explain what the question is they were trying to answer.
First, we spent 30 minutes trying to map out operational state changes (i.e., how long did it stay in stage A, B, C) for every transaction (10s of thousands each day). I asked the simple question “why?” Crickets in the room. Then I was proffered the explanation we need to know if it stays in a certain state past the amount of time it should. Aah, I said, so you are looking to see if there is an exception that needs intervention? “Yes,” was the response. So, I said ok, what are your business rules to define those exceptions. Another 60 minutes later and I called another time out.
“Are you really concerned about every stage of every transaction and whether or not it had an exception at a single stage, or are you trying to figure out which transactions failed the overall SLA of 48 hours from end to end? And then, the follow on “And of those transactions that failed your overall processing SLA, figure out if there are trends that point to underlying issues at certain stages?” Silence for about a minute then an exclamation from one of the business analysts “that’s exactly the question our VP asked us to answer!” When I inquired why we didn’t start with that question, gently of course, it was because they had a row level view of the data, not a business perspective. The lead business analyst told me after that session that it was one of the most valuable lessons he had ever learned–to ask the ‘right’ question.
You can infer from this the importance of understanding and framing the actual question or problem to be addressed. Due to poor framing of the actual objective, the discussion had devolved into the typical bottom-up ‘create something’ mentality. Framed a different way, in private plane flying, we call it ‘getthereitis.’ Put another way, most IT folks and business analysts are taught to a work breakdown approach… breaking the problem down into discrete steps and then trying to pull it all together at the end. In contrast, most scientists develop a hypothesis(s) or research question(s) first, then figure out how they are going to test it. In the above example, I ended up with research questions:
- Is there really a problem with exceptions to the agreed on SLA?
- What, if any, is the scale of this problem (i.e. is there a 0.5% exception rate or a 50% exception rate)?
- Is the ROI to ‘fix’ the exception rate there? Framed differently, are you chasing a diminishing rate of return where it costs you more to fix than accept the exception rate?
- Are there patterns to the exceptions, i.e. do certain stages exhibit above average problems in meeting SLAs?
- Finally, given there are patterns in the stages, can we build a pattern or algorithm to determine whether or not we can predict whether a given transaction might bust the SLA?
My final takeaway I will leave you with is this. Deming famously said “you can’t manage what you can’t measure.” However, he had an unfortunately less famous corollary to that, which is, “be careful what you measure, because that’s what you’re going to get.”
Until next time…