Technical Debt

Technical debt is a great term, originally coined by Ward Cunningham, to convey the reality of future problems brought about by making decisions with an eye to short-term gains instead of long-term correctness. This is not a new concept, but before Ward, it had never been applied to software development. (Martin Fowler reports that the term was used in Ward's report at the OOPSLA conference in 1992.) Let's say that again incase you missed it.

Technical Debt: future problems brought about by making decisions with an eye to short-term gains instead of long-term correctness.

This exact concept was used years ago in the advertisements for Fram motor oil filters. Every car owner knows that they need to replace the oil and filter in their car on a regular basis or they will eventually experience engine problems. The advertisements in question stated that using inferior oil filters (naturally, anything that wasn't sold under the Fram brand name) would eventually cause the same problems. At the end of the commercial the mechanic looks at the camera and invites you to "pay me now or pay me later". This is how you avoid mechanical debt; take small amounts of appropriate action now, or take massive (and expensive) reparative actions later.

The financial world has had this concept from the beginning of money (or at least the lending thereof). Debt is a very real thing for many people and it's something that gets dramatically worse the more you fail to address it. The definition of worse can vary of course. For some "worse" means having a credit card declined, or a car repossessed, or a house repossessed, or a business declared insolvent. And then there are some who operate outside of the realm of the legal, who will be more than happy to break your kneecaps when your debt exceeds the agreed repayment amounts.

One aspect that most forms of debt share is the personal pain (sometimes literal pain if you borrow from stereotypical Italian gentlemen named Vinny) from failing to address that debt. When you ignore it long enough it has a way of getting your attention, often to the exclusion of everything else. While this is generally quite unpleasant, it does effectively concentrate the mind on efforts to bear down on the debt and start making it go away.

A big problem with technical debt is that the personal pain is often not applied to the one making the decisions and who, by all rights, should be experiencing it. I'm talking about Information Technology managers here. While not a few bad decisions are made by programmers, the clear and massive majority of them are made by managers. The problems and pain of the technical debt is then felt by the programmers. The irony in the situation is that those same programmers have likely warned against exactly the situation that they now find themselves placed in.

I have seen technical debt accrued in almost every company that I have worked for. There seems to be a slavish adherence to the concept of "first mover advantage". This would be lovely if the concept worked, but the general public seems to be learning to place a premium on correct over fast. Sadly, the memo hasn't made it to Information Technology management yet. Consider Microsoft's Vista operating system. I understand from reading technology blogs that Vista was released because they were fed up of the computer press asking when it was going to be ready (Really, what's the rush? Doesn't everyone take five or six years to release a new operating system version?). Microsoft picked a date and shipped it "no matter what". The result was a stack of bugs tenuously piled up into the shape of an operating system.

I'm sure some are reading this and saying words to the effect of "but you have to spend money to make money", meaning you have to accrue some debt to make money. I call "nonsense" on that. The only way to get out of debt is to spend less than you earn. It also happens to be the only reliable way to not get into debt in the first place.

Technical debt is completely avoidable. You can run your project in a debt free fashion. I've done it and thereby feel that I have the right to refuse to hear that it's impossible. The trick is knowing what's right, sticking to your guns and doing it. Lather, rinse, repeat; as the shampoo instructions say.

WHEN IT'S DONE

A good example of running a technical project debt free may be had by watching almost any open-source project run by the Apache Software Foundation. The Apache folks have a reputation for high quality work. Part of this comes from the natural tendency of good programmers to want to work on Apache projects. Another part of the quality equation comes from the peer review that naturally happens in an open-source project where everyone can see all of the code and is free to examine and comment on it. The biggest part of the reason for the quality of the Apache projects comes from their standard answer to the most common question received on their mailing lists.

The most common question that I remember seeing when I frequented the Apache Struts mailing list was "when will it be done?" A few months after any release I remember seeing this question being asked about the next version or point release. As regular as clockwork this question would come up time and time again. The impressive thing was that the answer was always the same. The answer was always "when it's done!"

Even a superficial consideration of the question leads to the conclusion that it's the only answer that can realistically be given. If a project team sets out to perform a certain unit of work, then they either perform it or they don't. It's a binary issue and that unit of work is delivered or it isn't. While I understand that life brings surprises and plans can change, the work is still either done or it isn't. Sometimes those life surprises are bigger than expected and the plans have to change and the unit of work is modified and that will affect the estimated completion date (remember, an estimate is only a wild guess in a suit) because the work is still done when it's finished. Even changing scope does not eliminate the binary nature of the matter.

All Apache projects are done when they're done. Period. Apache projects carry a heavy expectation of excellence. I know that even back in the 1.0 days of the Struts web framework, I was able to rely upon it totally for the system that I was the tech lead over. In fact we even went to production with a beta version of Struts 1.1 as the quality was so high that I couldn't see any reason not to use it while I waited for the final 1.1 release.

The Apache projects know what the right thing is and they stick to their guns. They know that they pick a unit of work to perform. They work on it until it is finished to their satisfaction and only then do they release it to the waiting world.

Sadly, the concept of "it's done when it's done" is foreign to the modern Information Technology manager. Modern Information Technology projects are all driven by deadlines; arbitrary deadlines at that. I'm working on just such a project now, fixing a compliance issue with our widget sales. We're already out of compliance, but the management thinking was that it should be fixed by the end of the year so that we'd be compliant next year. There was no examination of whether that was doable by a single developer still getting used to the system in question and for which there are no tests so who knows if anything gets broken? Management says proceed because it's more important to be seen to be fixing the issue even if it's a hurried fix that might in turn need to be fixed.

I don't think I could begin to count the number of timeframe driven decisions that I have witnessed or have heard details of from reliable sources that I trust. These decisions have a long history of turning out badly, but because the pain is felt by the programmers, the managers find them most agreeable.

This is root cause of technical debt. Information Technology management making bad decisions based on a desire to get things done "at the speed of business" and then not feeling any of the follow-up pain of those decisions. And the punditry wonder why there are falling numbers of programmers. I know that I've advised my little geeklets to not even think of working in Corporate America and especially not Information Technology. Even just last week I was chatting with co-workers about Corporate Escape Routes; just how do we tunnel out of the cubes that are our prison cells?

So, how does a deadline driven approach to project management, the way that it's implemented in Corporate America today, cause technical debt?

Let me start by saying that deadlines aren't all bad, especially if they are determined by a careful examination of the amount of work that needs to be done and the availability of skilled programmers to do it. This is not how it's done in Corporate America today, so we'll skip straight to discussing "pick any two".

PICK ANY TWO

Information Technology is still a relatively new discipline, but it has been around long enough to have had classic project management principles applied to it. These principles tell us that there are three aspects to a project and that two can be tightly controlled at any time with the third varying depending upon the decisions you make on the two you choose to fix.

The three aspects of an Information Technology project are Scope, Time and Quality. Some people use the term Cost instead of Time, but Cost tends to vary in direct proportion to Time, so with the time obsession of Corporate America, it seems more appropriate to call that aspect Time. The interplay of these aspects are seen by way of tradeoffs. The more rigidly you fix any two of the aspects, the more the third one is left to vary.

A good example of an engineering tradeoff would be NASA. These folks put people into space and bring them back again alive and well. NASA fix Quality at it's maximum and Scope doesn't change because otherwise there's no point in the mission. This leaves Time/Cost as the variable. Of course, being a government agency, it seems like Time/Cost is the last thing they worry about anyway!

Now, in most Information Technology projects, and when I say most I mean every one that I've been on and just about everyone I've heard of from my contacts, the two fixed aspects are Time and Scope. As a motivational speaker that I heard many years ago said, "Every project starts out with a deadline and a name."

Even the Scope is not usually that well defined. Hence the large number of incidents of scope creep. It's not really scope creep ... it's more that the project was started before they knew what they wanted. This is the usual behavior of Information Technology management. They aren't sure what they want, but they're deadly certain about when they want it. To get a change in the planned project completion date usually requires a presidential declaration, delivered by pink pigs flying in formation with white unicorns.

This is why it's so dangerous to offer estimates to managers, because once they've heard a date inside the timeframe they wanted to hear, they stick to it like the proverbial limpet. I know that I've offered estimates to managers and have had the number halved right in front of me. Usually they say something to the effect that they had to do that as their manager wouldn't accept an estimate that large.

So with the Time aspect being cast in concrete and the Scope staying still at best and increasing under normal conditions, the only other aspect left that can vary is Quality. And the universal observation is that Quality will always very downwards. This fits with the law of conservation of energy. If Time is fixed and Scope tends to increase, then the only direction that Quality can go is down.

This concept does not seem to fit with the worldview of the average Information Technology manager. Funnily enough, everyone else seems to get it. I particularly like the way that the U.S. Navy SEALS express it: "Fast is slow. Slow is fast." When everything is melting down around you and you need something done right, then slow down, deliberately slow down and concentrate on doing the action slowly and correctly the way you would have done it in training and let the muscle memory take over. It will be done right (for whatever your definition of "it" is).

Information Technology managers are the reverse of the SEALS. When a problem occurs they switch into panic mode. Everyone is expected to stay late. Senior managers tend to start hovering outside the cubicles of those performing the fixes. Hourly status meetings get called and are conducted in stand-up fashion outside the fixer's cube.

BON APPETITE

I'm going to wrap up with an example currently being worked on by others while I relax at my local Starbucks and enjoy a cup of coffee. The core pricing module for our widgets is broken for the introduction of a new widget. It is blowing up when pricing is requested for that widget. I'm in the process of coming up to speed on this module myself and so I know that there are zero unit tests in the code base. I have written tests for all of the code in the area that I'm working on, but these have not been promoted into the main code base yet. The lack of unit tests means that there are no areas that can be considered as bug free. So the bug could be anywhere. (Where there are good unit tests, there can be no bugs!)

It would seem that the problem involves a NullPointerError, so the chances are good that some domain object is being incorrectly initialized. Unfortunately no one knows which one it is because none of our domain objects have any kind of guard conditions for their input values. Any of our domain objects can be in any condition. There are no guarantees that they are ever in a valid state.

The decisions to have no tests and no guard conditions to force data integrity are the result of previous bad decisions motivated by the desire to get stuff done quickly. These decisions have consequences, we know these as Technical Debt, and those consequences have grown large teeth and claws and have developed a taste for untested code. In this core pricing module the untested code stretches as far as the eye can see.

Bon Appetite Mr. Consequence!

The Fastest Guys on the Block

A classic of the Internet. I was reminded of this one the other day. Read and enjoy! :-)

There were a lot of things we couldn't do in an SR-71, but we were the fastest guys on the block and loved reminding our fellow aviators of this fact. People often asked us if, because of this fact, it was fun to fly the jet. Fun would not be the first word I would use to describe flying this plane. Intense, maybe. Even cerebral. But there was one day in our Sled experience when we would have to say that it was pure fun to be the fastest guys out there, at least for a moment.

It occurred when Walt and I were flying our final training sortie. We needed 100 hours in the jet to complete our training and attain Mission Ready status. Somewhere over Colorado we had passed the century mark. We had made the turn in Arizona and the jet was performing flawlessly. My gauges were wired in the front seat and we were starting to feel pretty good about ourselves, not only because we would soon be flying real missions but because we had gained a great deal of confidence in the plane in the past ten months. Ripping across the barren deserts 80,000 feet below us, I could already see the coast of California from the Arizona border. I was, finally, after many humbling months of simulators and study, ahead of the jet.

I was beginning to feel a bit sorry for Walter in the back seat. There he was, with no really good view of the incredible sights before us, tasked with monitoring four different radios. This was good practice for him for when we began flying real missions, when a priority transmission from headquarters could be vital. It had been difficult, too, for me to relinquish control of the radios, as during my entire flying career I had controlled my own transmissions. But it was part of the division of duties in this plane and I had adjusted to it. I still insisted on talking on the radio while we were on the ground, however. Walt was so good at many things, but he couldn't match my expertise at sounding smooth on the radios, a skill that had been honed sharply with years in fighter squadrons where the slightest radio miscue was grounds for beheading. He understood that and allowed me that luxury. Just to get a sense of what Walt had to contend with, I pulled the radio toggle switches and monitored the frequencies along with him. The predominant radio chatter was from Los Angeles Center, far below us, controlling daily traffic in their sector. While they had us on their scope (albeit briefly), we were in uncontrolled airspace and normally would not talk to them unless we needed to descend into their airspace.

We listened as the shaky voice of a lone Cessna pilot asked Center for a readout of his ground speed.

Center replied: "November Charlie 175, I'm showing you at ninety knots on the ground."

Now the thing to understand about Center controllers, was that whether they were talking to a rookie pilot in a Cessna, or to Air Force One, they always spoke in the exact same, calm, deep, professional, tone that made one feel important. I referred to it as the "HoustonCenterVoice." I have always felt that after years of seeing documentaries on this country's space program and listening to the calm and distinct voice of the HoustonCenterControllers, that all other controllers since then wanted to sound like that... and that they basically did. And it didn't matter what sector of the country we would be flying in, it always seemed like the same guy was talking. Over the years that tone of voice had become somewhat of a comforting sound to pilots everywhere. Conversely, over the years, pilots always wanted to ensure that, when transmitting, they sounded like Chuck Yeager, or at least like John Wayne. Better to die than sound bad on the radios.

Just moments after the Cessna's inquiry, a Twin Beech piped up on frequency, in a rather superior tone, asking for his ground speed.

"Ah, Twin Beach: I have you at one hundred and twenty-five knots of ground speed."

Boy, I thought, the Beechcraft really must think he is dazzling his Cessna brethren.

Then out of the blue, a Navy F-18 pilot out of NAS Lemoore came up on frequency. You knew right away it was a Navy jock because he sounded very cool on the radios.

"Center, Dusty 52 ground speed check."

Before Center could reply, I'm thinking to myself, hey, Dusty 52 has a ground speed indicator in that million dollar cockpit, so why is he asking Center for a readout? Then I got it -- ol' Dusty here is making sure that every bug smasher from Mount Whitney to the Mojave knows what true speed is. He's the fastest dude in the valley today, and he just wants everyone to know how much fun he is having in his new Hornet.

And the reply, always with that same, calm, voice, with more distinct alliteration than emotion:

"Dusty 52, Center, we have you at 620 on the ground."

And I thought to myself, is this a ripe situation, or what? As my hand instinctively reached for the mic button, I had to remind myself that Walt was in control of the radios. Still, I thought, it must be done -- in mere seconds we'll be out of the sector and the opportunity will be lost. That Hornet must die, and die now.

I thought about all of our Sim training and how important it was that we developed well as a crew and knew that to jump in on the radios now would destroy the integrity of all that we had worked toward becoming. I was torn. Somewhere, 13 miles above Arizona, there was a pilot screaming inside his space helmet.

Then, I heard it. The click of the mic button from the back seat. That was the very moment that I knew Walter and I had become a crew. Very professionally, and with no emotion, Walter spoke:

"Los Angeles Center, Aspen 20, can you give us a ground speed check?"

There was no hesitation, and the reply came as if was an everyday request:

"Aspen 20, I show you at one thousand eight hundred and forty-two knots, across the ground."

I think it was the forty-two knots that I liked the best, so accurate and proud was Center to deliver that information without hesitation, and you just knew he was smiling. But the precise point at which I knew that Walt and I were going to be really good friends for a long time was when he keyed the mic once again to say, in his most fighter-pilot-like voice:

"Ah, Center, much thanks. We're showing closer to nineteen hundred on the money."

For a moment Walter was a god. And we finally heard a little crack in the armor of the HoustonCentervoice, when L.A. came back with,

"Roger that Aspen, Your equipment is probably more accurate than ours. You boys have a good one."

It all had lasted for just moments, but in that short, memorable sprint across the southwest, the Navy had been flamed, all mortal airplanes on freq were forced to bow before the King of Speed, and more importantly, Walter and I had crossed the threshold of being a crew. A fine day's work.

We never heard another transmission on that frequency all the way to the coast. For just one day, it truly was fun being the fastest guys out there.

Cute kitten story

LawDog seems cured of his writers block. And with a cat story to rank among the classics. Now I'll have to dust off some of my cat stories. Having previously been blessed by seven (sometimes eight) cats in a previous chapter of my life, I have a tale or two of my own.

Best. Church. Picnic. Ever.

Stuffed. Roll me home stuffed. Far too much food. And it was all good. Mmmmmmmm.

We rented a shelter at one of the local municipal parks and grilled too much dead animal and eat it with too many carbs for desert while drinking too many sugary carbonated beverages. And the guys talked about guns. Whats not to like?

Fair Warning

I feel obligated to issue fair warning to all pieces of paper with concentric circle patterns drawn on them. My order from Cheaper Than Dirt has arrived and much target shooting will be scheduled real soon now.

I love CTD's Wolf 7.62x39 ammo for my SKS. I've tried the fancy stuff, but the Wolf just shoots tighter groups and feels better. And it's cheaper! Try some, you'll like it.

Now, that's political humor!

Several good political cartoons over at Irons In The Fire. My favorite one is the one about John Bolton as president.

A tip of the hat to The Smallest Minority.

Employee Reviews

As a pastor and the editor of our district's newsletter, I have always been used to having to report on what's happening and how my team is performing.

Being the editor of the district newsletter this responsibility includes reporting before the full district board and the district superintendent each fall during the district planning session.

While it might sound intimidating reporting to the Bishop, it's really not a huge deal as I stay in contact with my representatives throughout the year and make sure that I know what we all did, why we did it and what we are planning to do in the year to come.

It's not much different at church board meetings. I meet with the church board roughly every quarter and we discuss the state of the church, finances and direction. I consider it part of my job to know what is going on in every part of the church, who is doing it, why they're doing it and how it's going.

Then at work I find that the process for end of year reviews is totally messed up. This is not just a slam on my current employer. I have found this to be the case almost everywhere that I have worked for going on twenty years. The whole approach that Corporate America takes is fundamentally broken.

Part of the reason for this is that Corporate America is full of managers instead of leaders. That's a rant for another day, so I'll move on to the rest of the reason. The big part of why end of year reviews are so broken is that managers don't know what people are doing because they get themselves so overloaded with anything other than interacting with their direct reports.

The funny thing about that, is that many Human Resource departments even have rules in place to limit the number of direct reports that a manager has, to ensure that they are not overloaded to the extent that they can't track their direct reports. At my current employer, I believe that the maximum number of direct reports is in the region of a dozen.

Everything is in place for the manager today to know what their people are doing. Yet they don't. I just talked to my manager yesterday for the first time in months about what I'm currently doing and that was only because I bumped into him in the bathroom. Now, my manager's a nice guy and I have nothing personal against him, but I'm pretty sure that's not how the Human Resources department thinks that information flows in the corporation.

As I've spoken of before, the size of the congregation that I pastor is in the low thirties. I can tell you how they're all doing. I see these folks once or twice a week for a few hours at a time, yet I know more about my congregation than most managers know about their direct reports who come to work five days a week for a minimum of eight hours a day. Does this strike anyone as odd? I think it's disgraceful.

I know which ones of the congregation are undergoing medical treatment and have visited a number of them (or their spouses) at the local hospitals. I know which ones are struggling with issues and what the issues are. I know which ones are trustworthy when it comes time to ask for tasks to be done. I know which ones with kids are in charge and which ones let themselves be walked all over by their kids. I know which of our men wear the pants at home and which ones have abdicated the family leadership to their wives. I know who works in which industries and how their jobs are going. I know who is happy at work and who isn't. I know who is praying regularly and who is tithing regularly. I don't do this to pry. As far as I am concerned, this is just part of the job. You cannot lead someone or minister to them if you don't know them.

But it gets better. Almost everywhere I have worked, the end of year review process involves the direct report filling out a self-assessment review in which they write what they've been doing and rate themselves on how they've done with their projects. The big giveaway is that when the manager hands out the end of year review, that it bears an uncanny resemblance to the self-assessment that the direct report handed in.

Did I say "uncanny resemblance"? How about when you get your evaluation and the manager has copied and pasted your text into their document changing "I" to "You". Perhaps I'm alone on this, but I find that uninspiring and unimpressive.

May I kindly suggest to any manager who can't give an on-the-spot report of how their direct reports are doing that you are obviously not managing. I don't know what you are doing (and it's likely that your staff don't know either), but you are not managing. Please, either start managing, or ask for a different job title and have your direct reports transfered to someone who is prepared to put in the effort to know what they're doing!

Why project managers get no respect

I think that Scott Berkun comes close to understanding a fundamental truth of modern project management in his blog post about Why project managers get no respect. There's more to it than he suspects, but he gets high marks for realizing that, on average, the problem lies with the PMs rather than the programmers.

I have a good rant simmering on the mental back burner about project managers, but it'll take several more weeks (at least) until it's ready for posting. Stay tuned and watch this ASCII character 32.

Happy Birthday USA and a busy week

Wow, what a busy week, but in a good way.

We started with district camp. As the editor of the district newsletter I attended for Purpose Institute graduation ceremony (it's our district bible school and our first four year students were graduating.) Geeklet number one received the Holy Ghost at the children's church service on the Monday evening.

Then Friday was the fourth of July, or as we like to refer to it in the good ol' US of A, Independence Day. This is the birthday of this fair country and for a 232 year old, I'd say it was looking pretty good. Happy Birthday Uncle Sam! :-) To celebrate this event, I headed out to the range with a friend of mine and he even brought his daughter for her first time ever shooting. As said daughter is a slim 15 year old, I had mercy upon her and just took the .22 rifle. Much fun was had by all and we add another one to the Nation of Riflemen. Pictures were taken, but I'd better get permission from my friends before posting them.

Saturday, with our recently received stimulus check recently deposited, we headed off to Sis. Geek's favorite furniture store and made quite a contribution to the economy. We're doing our part for the economy, Mr. President!