Incremental & Iterative Development Risk Mitigation

Some thoughts on where incremental & iterative development (IID) methods mitigate risks not provided by a waterfall approach:

• Risk-driven iterative development forces tackling the most complex and riskiest problems first.
• IID requires testing early on (even before coding), with tests being improved over iterations.
• Early visibility for clients into the interim and target product and therefore better feedback mechanisms to refine the functionality/capability. This is forced by continuous and formalized input and presence of clients during development.
• Performance risk profile of the application is improved by delivering tested software frequently.

Other thing I have noticed recently (not us!) is people building up a door stops' worth of functional specifications, and then looking to do agile development against that. Amatuers.

2-D Risk Maps

In James Lam's Enterprise Risk Management book, there are some general approaches to Operational Risk Management as well as brief discussions on some of the specific techniques on how to assess and control risks. One particular approach is that of developing 2-D Risk Maps whereby a general risk assessment is treated with the application of relative risk rankings (with respect to probability and severity). Additionally, he discusses risk indicators and performance triggers that get factored into the "dashboard"

I can think of a number of examples of the indicators and triggers (e.g. 99.97/8/9% uptime of a production application or piece of hardware, etc). But, I would love to see some specific examples of these Risk Maps for an bank's application environment as I think this could be a rather tough thing to create given the diversity and complexity of the enterprise. Presumably, the severity ranking would include the criticality of the application in general, impact of certain functions/information not being available, and the knock-on (chain) effects. As for probability, I would imagine that it is more of a combination of some emperical data (ie knowledge of existing problematic systems) and a some finger-waving guestimates.

Corporate Taxonomy

Not everyone sees or thinks of things in the same way, and this has become alarmingly obvious when a group of colleagues and others discuss the Agile umbrella of IT methodologies. This is a fundamental risk we need to deal with in order to demonstrate our corporate culture, professional excellence, consistent project approach, and platform for marketing to clients.

In this case, Agile Development, not only are we talking about a set of methodologies under an overarching set of principles, each method has its respective theoretical pros and cons as well as what the professional and individual experiences of our staff have been, with any one or more approaches. No doubt there will be divergent opinions, but that is why we need a core position that we can then at least tweak on a project basis. In our business, experience is everything.

One avenue to aid this problem, might be to have a corporate taxonomy for our company that we all agree on. This would include the definition of terms, how techniques/process relate to one another (or not), and any crossovers. The natural extension would then to be to draw out the "best practices" or some working model that we believe would form the core approach for our company, given the type of projects we typcially undertake in Capital Markets.

Based on other feedback, it sounds like we need a hybrid model that also factors in real client needs as well as the best blended approach that might have a chance of *actually* working on client sites. Either way, we need to have a common language and "corporate view"that we can all adhere to before, during and after client projects. Perhaps a corporate Wiki is what we need.

Fixed Price: A Naive Approach

It would seem that most of the larger financial institutions are adopting one of the following models for outsourcing IT project work:

• Pure Staff Supplementation - get a body as cheap as you can to work under the client PM as part of the team
• Fixed Price (Cost) Project Work - (theoretically) fixed cost, set scope, set time.
• Offshore - the staff supplementation approach on steroids with some multi-location overhead thrown in for good measure (yes, I am a cynic)

Some are still permitting T&M project work as well. From a banks' perspective, I understand that minimizing / locking in / capping costs is important. However, how many of these company's business,IT or vendor management understand the constraints and risks imposed on the vendor by some of their preferred approaches ?

The three axis of classic systems development management are of course scope, time, budget, with the assumption that if you lock 2 of these, you set the 3rd. Often the time is set by business needs and their wants/needs without due consideration for the poor IT guys actually doing the work. Budgets are set, pretty much, and so the price constraints are pushed to the vendor. But do they flex on the scope.. I think not. From a vendor's perspective, unless you are building a space shuttle, the scope simply is never bolted to the floor -- and even the best specification can create ambiguities, or misrepresent what is going to be built .

All of these constraints that a vendor faces can be alleviated through a T&M approach, or by a more agile approach to project execution. (I am not pitching Agile here, although we all know that iterative/RAD/Agile approaches can certainly be the best way of building out functionality to maximize user satisfaction - where appropriate).

The real risk here is quality of delivery and the adversarial relationship that can occur when clients expect vendors to deliver a set scope in a fixed timeframe and/or budget. The legalese are a nightmare. The relationships can sour and spoil reputations, and the quality of the product can be terrible through corners being cut, functionality being descoped (not always to plan), and customer service can go through the floor.. all of which costs more money directly or indirectly.

Mature relationship, project and risk management are essential (always, no matter the project), but real life is not always that cookie cutter.

Perception is everything

True story. We have been working for a fixed income trading desk that officially (from a management's perspective) uses a homegrown processing system for its positions, P&L, and back office processing. This system can provide intra-day positions and P&L upon request, but the traders have never (meaning over years) believed the numbers this system produces and so have not used it. Worst still, they **guess** their positions and P&L during and at the end of the day, and can often be $0-1mm out in their guesses.

We have been working on a number of projects for this business where are are doing a mix of RAD and tactical/prototypes for some new risk mgt tools for the traders. One of these prototypes pulls together a bunch of intra-day data from various sources, including the above mentioned system, does some aggregation and provides a rich and flexible front-end for position and P&L tracking (the server and client side is .NET, btw).

These traders think this tool is the best thing since sliced bread and think it is much better and more accurate than the old system they are/were supposed to use. How ridiculous is this ? Can you imagine a multi-million dollar a day business being run by people guessing their P&L, let alone thinking that the new sytem is better than the one they despise ...!!

PS. Not a bad argument for Agile development though, where users get what they want ..

An Insider's View on Agile

This is from someone inside our client base who has also tried to sell & manage Agile projects....

The issue with selling (and managing) Agile projects is maintaining the balance between rapid delivery and controls. Traditional projects that businesses are used to following expect a roadmap, milestone dates, dependency dates and anticipated ends with expected functionality. With Agile methodologies, requirements are developed as you go and the end state is less predictable. Keys to success include tightly managing expectations and setting guidelines for each iteration. Selling these types of programs are difficult because they don't fit within the traditional framework - if you can walk through a project lifecycle and build the predictability by setting tight guidelines for estimations, time spent on an iteration, level of detail spent on each component at any given time, building a working end-to-end system albeit "rough" functionality in the first few iterations I think you can demonstrate the benefits of the methodology and mitigate the risks.

Also, large companies and large programs will worry about the forecasting of key dependencies on integration points with other systems. You will need to accout for that as well.

In the end you will create a hybrid traditional/Agile methodology to enable you to gain the speed and agility of rapid iterations with the controls, planning and forethough of integration.

Situational Leadership

colleague of mine sent this link about styles of management required to run agile projects effectively .. interested to get some opinions on this ..

http://www.mountaingoatsoftware.com/articles/SituationalLeadership.pdf

Selling Agile or Agile Selling ?

I suspect this will be the first of many entries here, so here goes with the can of worms: How do you sell an agile approach to delivering projects, factoring in the needs/issues/barriers of the folks in the client: PMs, CTOs, vendor mgt, legal, CFOs ?

We can create the nice glossies and presentations talking about this, that and the other benefit of agile methodologies (whichever one or combination you prefer), but there is more to it than this.

My take on it is more to do with the classic 'Solution Selling' approach: block & tackle question model whereby there are three main lines of questioning : diagnose, explore impact and visualize capabilities, each with a subset series of questions: open, control, confirm... (btw - this Solution Selling is probably another blog entry for the future). The point is that you need to understand the business problem and pain (or latent pain), and then apply your judgement (and agility) as to how you can solve their problem quickly and effectively. I think this is a pragmatic and experiential sell - you need to know your stuff, but be able to iteratively work out the right approach for the problem - sound familiar ?

Others have blogged about the legal side of things, which I think requires a shifting of the mindset on the side of the clients' legal/vendor mgt. It almost like you have to disprove the status quo and show that they business NEVER gets it right first time, and that quick meaningful deliverables (production quality) is actually what the business wants. Some further thoughts from the 'Agile Champion' are here.

As I said, this was just to open up the can of worms. Any first hand experience of others in selling agile approaches to non-buyers would be appreciated ...

Attrition from Client projects

Attrition happens -- I have no problem with it and we need to deal with it as a course of doing business. Whether it is for more money, better opportunity, change in career direction, family demands, whatever, it is all cool with me.

What really gets my goat up, is when people agree on a start date with a new employer, but fail to agree on the last date with their current employer. Two weeks notice is standard issue in the US, 4 weeks in the UK, but for goodness sake please assume that that's what your employer and client expect from you, not whatever you think you can get away with.

What is intruiging to me is when I have to tell a client one of our people has quit, and the notice is less than the contractual obligation (and remember we as a vendor have our own obligations to the client), the leaving party actually thinks they have done the right thing by everyone and will say "..it is a small world and I am sure we will work together again sometime .. I really like you guys ..". Yeah, right. They have in one fell swoop just nicely torched a bridge or four, and will never be employed again by us or our client. Grow up, please.

From a risk mgt perspective, there are some obvious negatives to this and only so much you can do (other than having a bench!). On the positive, I have found there is opportunity to relate to the clients' heartache and to possibly re-jig a few things around to our mutual benefit.

Agile: Frequency = Currency

Was talking to our Chief Architect (see netflings ) this morning about his experience with what well executed agile projects can do for you and your client IT manager. Showing the business that your quick hits add value and iterating to what they actually want, buys you a lot of credibility and therefore currency in dealings with them later on .. either to get more busines from them, or to use your credits for when you do mess things up (which you will at some point!).

The interesting thing about our CA's project is that is was done with a small team of .NET developers working along side the traders of a hedge fund who are big into credit derivatives. Instead of buying an off the shelf product like JRisk, this team delivered with a high frequency and thus very low risk (once they got over the initial hump of getting their feet under the table)..

Scoping Problems

Another rats' nest of a topic ..

Banks are horrible at scoping their needs, so why don't they wake up and do something about it ? Too easy a question to ask, I guess. Part of it is management's incompetence, some of it is crap BAs, indecisive business users, the list goes on.

This week though was another good example where we spent 6 weeks working on an RFQ to define a scope of work (in general terms of course), and then get our approach and shop in order. We spent a lot of time on the governance and risk mgt side of things ensuring we got the client dependencies & responsibilities outlined (first cut until we actually start the project, of course). One of those dependencies was getting the main set of use cases done and any other requirements before we could do our thing. Literally, on the first day of the project our project lead tells me "the guy doing the use cases is working on something else next week, so won't be done until then". First friggin' day. Then next day "we [the client] have some delays, we want your team to help out on another project..." Time to bolt down the doors and stick to our plan, I say.

I know this is a common thing in scope creep, etc, but these guys are just simply awful at it. My whole thing in doing this blog is because of the need for risk mgt everywhere and this is one of the prime culprits. Any waterfall based methodology is simply flawed (as many would atest to), but it is our duty as consultants to bring them around to a new way of thinking in terms of either sticking to a plan (meaning both some upward management and introducing them to an agile plan that can adapt to changes in not only scope but timing of tasks), or saying "NO", or "games a bogey, let's be a staff supplementation company" .. first two sound good to me..

Project Measurement and Control

How horrible are people at managing projects ? Create a plan, let it gather some dust while you do something different , then bin it. This is one of the biggest problems today in IT.

One thing I want to do is to derive some risk management best practices for the types of projects we tend to do in Capital Markets, based on some of the experience out in the marketplace for agile development and other alternate approaches. The other area I want to evolve is the role and experience of our project leads in these projects -- are they domain experts, converted PMs, upgraded BAs, maturing techies ..... ? Stay tuned on this one.

From a vendor's perspective, I have found that a formalized status meeting of both sides is key -- bi-weekly being a good balance between not intruding on people's time and staying on top of things. The focus is not so much on measurement and control (M&C), but on risk and issue management, as well as general performance feedback. A couple of main reasons, I see, for this: a) it is the client PM's job to do the M&C as well as our own team lead (in terms of his team's tasks), b) I have to trust my guys and not to micro-manage them (not my style anyway), and c) it is the big ticket items, politics and barriers that will screw the project, and not so much the minor implementation or functional details.

That said, too many times project-based consultants get treated like the staff-supplementation folks (and that's not just our people that feel this), resulting in the client PMs (micro) managing the tasks of our folks. This is an uphill battle, but a fight we need to stick to in order to ensure we don't get sucked down a rathole and away from the project in hand.

AD & Production environments at banks

One thing I have heard a lot of people talking about in the banks are the issues around shared production environments (yes, yes, dev environments as well, but I am talking about the stuff that loses businesses money in real-time). Recent gems include:

• real-time calculation engines, batch and process based applications, and web applications running in the same memory space on a single VM.
• distinct applications running on their own JVM, but communicating across a common messaging hub whereby a huge amount of Java objects are serialized/deserialized, taking up to 90% of the CPU cycles, with 10% being left for actual business functionality

How does this happen ? Is it bad architects ? Are there historical / evolutionary reasons ? Is it bad implementations of the design patterns (if there are any) ? A lot of "bubble diagrams" and no prototyping ? A bit of everything, I suspect. I understand when tactical systems become less-tactical and more entranched, bad things do happen. But on new builds ? Seriously, when these guys are putting together new component-based architectures (in one client now), how does this happen consistently? There is a fundamental problem with the way these organizations build even the most basic of production applications. Call it incompetance, lack of quality control, poor engineering talent, bad management, or whatever, it IS incredible...

Work In Production ....

I have developed a somewhat sick interest for how people screw things up. I have an (aerospace) engineering background, been in the IT development & management space, done the strategic marketing for technology companies, amongst a few other things, and have seen a lot of hackers, incompetants, prima donnas, and complete BS-ers. Along the way I have seen so many things that could have been avoided through relatively simple risk management and better governance over the decision making and development processes of projects (of course all with perfect 20/20 hindsight). All sides of running and servicing a business is risky, and always will be, but this blog is my humble views and random ramblings on what is going on at the moment in the corporate world. Think of it as the out-takes of work in production ....