Toronto’s IT Fiasco: Repeating the Past Again

The people who run Toronto’s municipal IT infrastructure have managed to ground themselves so securely between rocks and hard places that they’re about to charge the taxpayer another hundred million or so to bail them out. It really is too bad, but Toronto’s not exactly beloved by the rest of the country, and I’m having a hard time remembering to feel guilty about laughing at the predicament the city has created for itself.

A year or two ago, Joe Barr wrote a series on the relationship between Microsoft and his home city of Austin, Texas. As Joe pointed out in “Why Austin TX is considering a Microsoft enterprise license,” the relationship started out a little one-sided, with the city unable to influence licensing terms. Later, as he also reported, the city looked at Linux as an alternative and is now well on its way to piloting it as a generic desktop solution — thus giving itself lots of bargaining clout for the next go-round with Microsoft.

Austin seems to be doing the right thing — assessing what it needs, publicly debating the options and moving to break Microsoft’s monopoly pricing. Unfortunately, that’s not what’s happening in Toronto, where an almost farcical series of revelations about past PC purchasing behavior is cloaking a deep determination to change nothing.

Toronto is simultaneously Canada’s largest city and its most insular. Culturally, it wants to be New York, but its view of the national news includes the Toronto traffic report, and its business community is roundly despised in the rest of the country for combining the worst of the arrogance and greed attributed to the New York financial community with a service ethic straight out of Dilbert. To an Albertan like me, therefore, the public embarrassment produced by the city’s present computer-leasing problem is more amusing than pitiable — despite my knowledge that the people who will pay for this mess aren’t the people who created it.

The Usual IT Magic

What seems to have happened is that, way back in 1997 and 1998, the city decided to lease about 9,000 new Microsoft PCs, and various companies lined up to provide them. The eventual winner, a company called MFP Financial Services, offered a purely financial deal around leasing the PCs and their software, thereby freeing the bureaucrats from having to deal with that pesky competitive-bidding process in terms of actual configuration, deployment and licensing.

Between one thing and another, the usual IT magic happened: An agreement to spend CDN$42.3 million quietly became a $102 million contract. This wouldn’t have mattered to anyone of importance, except that the leasing company’s practices came under public scrutiny when a councillor in some hick town pioneered the use of a similar escalation in an unrelated MFP Financial transaction as a means of bringing down a political opponent.

The resulting revelations led a disgruntled competitor on the PC deal in Toronto to have absolutely no contact with either a reporter or a couple of potential whistle blowers. Nevertheless, one of the latter — obviously from out of town — obligingly threw herself on the sacrificial altar by talking to the former and ultimately precipitated both the present legal inquiry into “all aspects of leasing contracts for computers and related software between the City of Toronto and MFP Financial Services” and a parallel inquiry by the same judge into some related matters.

A Simple Waste of Money

By the time the lawyers are done, the inquiries likely will have gone on for two years and cost $15 million. But here’s how the editorial board at one Toronto newspaper (“City served well by MFP hearings,” Toronto Star, Nov. 29, 2003) described the results to date:

The inquiry showed that the very people whose job was to protect taxpayers’ money — who were to ensure that rules were being followed — were wined, dined and treated to travel by corporate interests. Some were flown to a hockey game in Philadelphia. One was whisked to a golf tournament in Augusta, Ga.

Contrary to this editorial, the most interesting revelation from the inquiry so far hasn’t been that top bureaucrats curried favors from vendors, or even that Toronto managed to pay an average of nearly $8,200 per PC. No, the most interesting finding is that MFP was wasting its money providing those favors.

I’m not kidding. According to their own testimony, none of the executives involved, whether in IT, in Finance or within the city council executive organization, exerted any influence on the contract decision. Under oath, they almost unanimously opted to be frank: “Sure,” they said on the stand, “MFP took me out for a $3,000 lunch in Florida, but frankly I thought they just liked me. I mean, speaking quite frankly now, everyone knows I’m far too senior to have any influence over the people who report to me.”

Catch 22s and Termination Clauses

They say the best defense is a good offense, and certainly this reversal of responsibility was creatively offensive, but the sad truth is that there probably wasn’t anything unusual about any of it. Leasing and outsourcing agreements always cost the taxpayers more than purchases because additional overhead needs to be paid, while deals made under cover of the agreement escape the competitive process. This inquiry, unfortunately, is into the leasing process, not the cost or control consequences of the decision to lease.

The Catch-22 providing the abject lesson here comes from the fact that this lease had the usual paste-in termination clauses calling for return of the leased gear in as-new condition and a nominal six-year life. Now, in reality, leases like this are usually legal shams because both sides expect to replace everything somewhere around month 30 as the budget gets rolled over, appropriate payments are made and new gear is brought in. That’s why a competitor went on record at the time saying that a putative $40 million deal was worth about $150 million; he was counting on automatic rollovers as new Microsoft software made the gear obsolete every two years or so.

In this case, however, years of political controversy have now given way to years of legal inquiry, and absolutely nobody is about to take responsibility for any kind of action while the heat is on. As a result, the city has been adhering to the terms of the lease — which means first that it is still trying to use some of the gear, and secondly that it will eventually have to return an estimated 12,500 of the things in good working order.

Frozen in the Headlights

Unfortunately, this isn’t actually possible. Even if all of it could be found, “as-new” condition requires the installation of fully supported software, and most of what those machines came with is no longer supported. Normally, of course, this is wink-wink, nudge-nudge stuff because new money greases out obstacles, but, in this case, there are lots of legal fisticuffs going on, and MFP might be expected to demand every ounce of its pound of flesh.

Worse, the headlight freeze Toronto’s IT management finds itself in has prevented timely replacement. As a result, they won’t have anything in place when the lease runs out next March and so will find themselves renting six-year-old PCs — at 1999 prices — for an average of at least seven more months.

All of this was nicely captured in the headline “City trapped in $100-million computer dilemma” in the Toronto Globe and Mail (Dec. 27, 2003). The story, by James Rusk, included the following summary:

The city’s problem is that 14,000 of its 17,500 computers are of 1999 vintage or older, which means that the operating system they run on is Microsoft Windows NT, a program which the software company will no longer support after some time next year, and they do not have the capacity to run on the next generation of software, Windows XP, which the city plans to switch to.

That this denouement is pending has been obvious for some time, but what the story in the Toronto Globe and Mail made clear is that they don’t read Joe Barr’s work (now on newsforge.com) in Toronto. If they did, they’d buy those things out and switch them to a Linux desktop. That way, they’d save millions in monthly rentals and at least give their users the ability to access the employee Web site and read executive memoranda instead of having to pass around printed documents.

But they don’t. And unlike Austin, they are from Toronto — meaning they have nothing to learn from Joe Barr or anyone else. Instead, they’re off to repeat the experience, this time asking council to approve 14,000 new Windows XP machines and a staff of 30 to install 800 of them per month while, I’m told, quietly structuring a big vendor “financial proposal” to do the job in three months.