RBS meltdown - Edinburgh, not Mumbai

All is revealed: in his letter to the Treasury Select Committee, Stephen Hester admits that the batch payments screw-up happened in the Edinburgh office:

The initial reviews we have carried out indicate that the problem was created when maintenance on systems, which are managed and operated by our team in Edinburgh, caused an error in our batch scheduler. This error caused the automated batch processing to fail on the night of Tuesday 19 June.

India is in the clear. All the frantic briefing by the RBS UK IT managers has been in vain - the finger points squarely at Edinburgh.

Credit to Hester, he's declining his bonus this year before anyone raises the issue. You can bet he's not pleased though. I'd be pretty annoyed if some clowns in the IT department had just cost me £2.4 million. I'd bet he's going to send a troubleshooter to that office, equipped with a sharp axe and a very high quota for necks. In a sense, this is possibly the best outcome for RBS because at least Hester is motivated to find out whose actual fault this was:

I will ensure a full and detailed investigation into the causes of the problem, overseen by independent experts and reporting to the Board Risk Committee and then the improvements we will make subsequently.
...
We will publish relevant findings from this investigation in due course.

If I were a mid-level or senior manager in IT at RBS Edinburgh, I'd be doing some CV polishing right about now. The published findings should make very interesting reading and be a near-required citation for papers in risk engineering and business-critical systems conferences.

I'm not sure about the Mail's characterisation of this letter as "grovelling". It seems pretty solid and workman-like to me, high on content and low on spin. In fact, I'd say it confirms Hester as a pretty good head of RBS and the Government should look at slipping him some additional compensation in a year or two once the fuss has died down. Heaven knows I don't envy him his job.

Update: Dominic Connor at The Reg has top tips for surviving the oncoming storm, aimed at RBS IT grunts. Helpfully, though, he addresses management as well:

If you're a senior exec at RBS
You're only reading this because someone printed it out for you and other media are quoting us. That's because your IT staff trust random tech journalists that they’ve never met with the truth more than they trust you.

Genius. Read The Whole Thing.

Thank God for the TSA's vigilance

The TSA - fearlessly guarding America's frontiers and shanking elderly starship captains.

Winner of "best comment": rezzyboy

Was George Takei anywhere in the vicinity????

RIM - all over bar the shouting

I'm going to go out on a limb here and hazard a guess that RIM is dead, along with Blackberry. Delaying Blackberry OS 10 until 2013, thereby missing out on the Christmas rush? They must be seriously slipping on internal delivery dates to make the delay announcement 6 months before.

RIM have taken a near-dominant position in business email and handed it out for free to anyone that wanted it (by the look of things, for now, Apple and iOS). They have forced their loyal retail consumers to go elsewhere for anything approaching an up-to-date smartphone; I've played with a Gingerbread full-keyboard device from Verizon and it's quite the match for anything RIM currently sells, never mind what Ice Cream Sandwich / Jelly Bean bring to the party. What is going to RIM announce in 6 months time that will trump what Apple and Google are shipping now?

For RIM, it's all over bar the shouting. A steady decline in market share until they are bought at an insulting discount for what few marketable patents they have. If I were a good engineer in RIM, I'd be reaching out to my buddies in Apple and Google right about now.

How to fill out a customer satisfaction survey

Courtesy of the inimitable James Lileks, trying to help Panera improve their customers' experiences:

Accuracy of your order. When you ask for coffee you get an empty cup and directions to the urn. The way the lady held out the cup so I could take it was highly accurate and professional.
Availability of the menu item you wanted. Well, I wanted calorie-free pizza, and they don’t have it, but that's my problem.

My personal rule of thumb is that a company from whom I've bought something moderately expensive gets three minutes of my time for their online survey. If five minutes have passed and I'm only 30% of the way through, it's 50-50 as to whether I blaze through the rest of the questions with deliberately misleading multi-choice answers and subtle abuse in the text sections, or I forward a ranting email to their head of customer service pointing out that they're not paying me for my time helping them, and if they want any useful information they can bloody well spend some time making the survey as short as possible. The bastards.

Damn those rich foreign students!

Oh no! University entrance standards are slipping! British universities are admitting foreign students at a lower standard than domestic students!

And exactly why are the universities overseeing this deplorable slide in standards?

Universities were accused of profiteering by rejecting tens of thousands of British teenagers, currently sitting A-levels, so they can fill places with more profitable foreign students.

Ah, the money imperative. Quelle surprise. But what's this?

Universities say that even the new £9,000-a-year tuition fees for British and European Union students do not cover their costs, and they need to turn to foreigners who are charged 50 per cent more.

It seems that stopping this slide in standards is going to result in either more taxpayers' money going to the universities to support domestic students (yeah, that's going to happen) or fees rising for said domestic students. Oopsie.

I'd bet this is one subject that any self-preserving politician won't touch with a bargepole. Any MP who grandstands about the issue clearly has no intention or prospect of serving in a government in the next decade.

The Baltic Sea USO - what's going on?

That's Unidentified Seabed Object, just to be clear. So I've been following the earlier Daily Mail articles about this strange rock formation on the bed of the Baltic, thinking "yeah, yeah, Face on Mars territory." Except that now it seems that there's an additional wrinkle:

Professional diver Stefan Hogerborn, part of the Ocean X team which is exploring the anomaly, said some of the team's cameras and the team's satellite phone would refuse to work when directly above the object, and would only work once they had sailed away.
He is quoted as saying: 'Anything electric out there - and the satellite phone as well - stopped working when we were above the object.

Now, let me say that my immediate working hypothesis is that Stefan is enjoying screwing around with credulous journalists. However, just supposing that he's reporting the actual facts, what's going on?

I think the most likely explanation involves the Soviet Navy; there's a long history of Cold War submarine activity with seabed-crawling Soviet mini-submarines, and the Swedish Navy depth-charging them. Maybe, just maybe this artefact is the result of some mid-1980s activity that buried Soviet (nuclear?) hardware under rocks and sand, broadcasting a significant RF signal for some as yet unknown purpose. The Russians are unlikely to admit any involvement unless actually confronted with the retrieved device.

Or maybe aliens crashed their spaceship into the sea, and this expedition will retrieve it. I've seen the second X-Files movie, and know that this never turns out well. Time to go short of Sweden?

Cargo cult journalism

If you want to see a demonstration of cargo cult journalism, look no further than Vanessa Allen in the Daily Mail writing about the RBS / Natwest payments system failure. For context, compare and contrast it with The Register's exclusive article on it from earlier today (which The Mail references).

Compare Anna Leach in The Reg:

The main batch scheduling software used by RBS is CA-7, said one source, a former RBS employee who left the company recently.
...
A second former employee, familiar with RBS's use of the CA-7 tool, confirmed that the majority of batch processing at RBS was performed using the software and fingered a failed update as the cause of the problem:
...
Staff who oversee batch scheduling for RBS are based in India:
At least some staff overseeing CA-7 batch scheduling software used by RBS are based in India as this job advert for a CA-7 admin in Hyderabad, dating from February this year shows:

Fairly well sourced and clear. Now let's compare Ms. Leach's output with Ms. Allen's:

The flawed computer programme that led to chaos for millions of RBS customers was being supervised by an IT support team in India, it was revealed last night.
Last February, RBS advertised for a series of key jobs, paying between £9,000 and £11,000 a year, in the Indian city of Hyderabad. That is way below what an equivalent worker would be paid in Britain.

The Mail article can be summarised as "OMG, dirty foreigners are stealing our jobs and wrecking our banking system." The Reg is far more nuanced: "this is the system involved in payments, it's reasonable to suspect its involvement in last week's problems, at least some of the support staff are in India and RBS seemed to know back in February that it needed more staff there."

You can just bet that the CA-7 staff in RBS will be doing their utmost to blame Hyderabad for the failed update. You can equally be sure that RBS management will do and say anything to avoid the implication that offshoring support to India caused the problem. Frankly, however, I'm inclined to take RBS management's side. ("How do those words taste coming out of your mouth?" "Like vinegar.") If any CA-7 work were retained in the UK, it would have been the development side. Contrary to popular opinion in the software world, it's the developer's responsibility to ensure adequate testing of their changes. They may rope in the testing and release team to help them, but they are the ones who made the changes and so they are the only ones who can realistically determine the necessary scope of pre-release testing.

Thank goodness for the TSA...

The TSA were diligently screening all passengers passing through JFK Terminal 7 on Saturday morning, spotting any metallic items with their carefully calibrated magnetometers. Until someone finally realised why not a single passenger had caused the machine to bleep that morning:

The chaos at Terminal 7 was caused by screener Alija Abdul Majed, who had manned Lane No. 1 during the morning shift with no idea his metal detector had no juice, sources said.
...
Higher-ups at the Transportation Security Administration finally discovered the security boondoggle at 9:44 a.m. — leaving the Port Authority with no choice but to call for a complete evacuation of the international terminal that is home to British Airways, Cathay Pacific, United Airlines and others.

Well, that's not really true, is it? If they had any reason to suspect that the disabling of the scanner was deliberate, they should have immediately arrested the responsible screener and grilled him. As it was, this was a pure cover-our-backsides action, security theatre at its best. No-one really expected to find a suspect item on any of the passengers re-screened -- and anyone who had successfully taken a suspect item through security would have had plenty of time to hide it before re-screening and pick it up afterwards.

Congratulations, TSA and Port Authority. You took a security snafu and used it to make it clear that you really don't intend for these procedures to contribute to actual passenger safety, merely as a continued source of government funds for your activities no matter how useless and annoying.

This must be some meaning of the word "glitch" of which I was not previously aware

Let's recap what we know about the NatWest/RBS banking outage: looking at The Register's assessment on Friday we see:

The screw-up has been pinned down to a flaw with payment-processing software, and primarily means that bank balances don't register inbound payments.

This problem hit on the 18th of June (a Monday), and so presumably was the result of a Sunday code roll-out. Since this hit so many customers, it's clearly not an intermittent problem; I'd be fascinated to learn who in the RBS/NatWest IT management OK'd a general deployment of this code version. I'd also be interested to learn why it couldn't be immediately rolled back once the problem was apparent - perhaps there was a data schema change involved, so they did the schema upgrade over the weekend downtime and then enabled it on Monday, but for some reason didn't have a verified reversion plan in place. Oopsie.

I particularly liked:

Customers were further infuriated after NatWest charged them for ringing its emergency helpline by initially providing an 0845 number instead of a free 0800 number, although the bank later said it would reimburse the cost of the calls.

where NatWest compounded an already massive technical failure with an entirely avoidable PR failure. Good job guys! I hope someone in PR is going to get a P45 for that particular decision.

It's easy to be wise after the fact, but releasing new software versions to a small set of customers to identify problems just like this is not exactly an unknown approach in software engineering. Nor is having a verified rollback procedure to revert the change when disaster has become apparent. Since it took until the weekend to deploy a fix, my bet is that it's not actually possible to deploy a software upgrade when RBS's systems are in operation, and that the broken payments system is tied to enough other currently-functional systems that downing it for replacement would have made things even worse.

I'm really looking forward to a technical post-mortem here, though I suspect the RBS lawyers and senior IT management will do their damndest to prevent it becoming public due to "propietary technology concerns" or similar weasel words. If I were writing it, though, it would go something like this:

1. We were told to deploy (new feature X) by June 4th;
2. Development and testing took longer than anticipated, and on June 13th we were instructed to deploy it by June 18th or face the consequences;
3. Our software passed all the relevant tests (which were inadequate, because good testing is the first thing that a development team skimps on);
4. We had a rollback plan (but did not verify it, because we never use them);
5. We deployed the software in our standard release plan (upgrade the data schema over the weekend, verify the data integrity, then push the upgrade on Sunday night);
6. Early on Monday morning our pagers and phones starting ringing off the hook;
7. We attempted a rollback and discovered (unanticipated issue Y) which meant the rollback would not work;
8. We entered a holding pattern and put together a bugfix that could not be deployed until Saturday since downing the system would have made things even worse;
9. There followed five days of customer support hell;
10. We downed the systems on Saturday, applied the bugfix and tested extensively, with everything now appearing fine;
11. We opened for business on Monday and everything worked just fine.

Follow-up actions for IT management:

1. Find a suitable scapegoat middle-manager and fire them;
2. Announce "lessons will be learned";
3. Appoint a new Director of Business IT Risk who happens to be a mate of one of the RBS board;
4. Splurge some money on whizzy IT release software and associated consultancy that doesn't make a blind bit of difference;
5. Move their own personal bank account to a UK bank with some grasp of how to run financial IT.

I'm not entirely sure that getting the Government, or even the FSA, involved is a good idea though. There is going to be plenty of hauling-over-the-coals going on in the RBS IT department without regulator intervention -- and, let's face it, designing and testing business-critical software systems is not exactly the FSA's line of expertise. "There's no problem so bad that government intervention can't make worse", and all that.

It also occurs to me that the best way for RBS to prevent a reoccurrence of this is to pay Good Money to find and recruit a small number of really good engineers from Google, Amazon, Microsoft etc. (all famed for their uptime and reliability - yes, even Microsoft) and give them free rein to fix software, processes and people. Give them a large, meaty bonus conditional on measurable reliability improvements and comprising a significant proportion of RBS shares. Perhaps this is one case where public and FSA wouldn't object to large "banker bonuses".

Wonderful aphasia in the DM

An otherwise unremarkable article about a family in negative equity being slightly shafted by the HS2 route is given colour by the journalist's poor English and/or the sub-editor's poor proof-reading:

Some homes on the train line route face being the subject of a Compulsory Purchase Order by the network - which pays an aggravated valuation of each house.

I rather suspect you mean "aggregated" there, Tammy Hughes..

Still, let's look at the story now we're there. So is the rail line forcing them out of a modest semi?

But they discovered the planned HS2 network high speed rail link between London and Birmingham will pass through their 13 acres of land - just 130-metres from their home.

Oopsie. This is in Warwickshire, and the house is a sprawling 7-bedroom affair. The house was mortgaged for 600K at the height of the property bubble in 2007, but now is only (optimistically) worth 400K. The father of the family died, which is tragic, and yet didn't have a life insurance policy that would pay off any significant amount of the mortgage.

You have to feel sorry for Vikie (sic.) the mother of the family, as it's clearly a hard situation to be in. Nevertheless, if you were intending to spend all your money and borrow up to your limit to get a house of that value, wouldn't it have been a good idea to set aside £50/month for a life insurance policy that would have paid off most of the house?

The real meat is tucked away at the tail of the article:

A spokeswoman for HS2 said: 'We have advised the Shanks' family that their building has not been identified for demolition or as being at risk of demolition in our assessments to date.
'According to the land ownership information we have obtained from Land Registry, the line does appear to pass through some of the land associated with the Shanks’ property.
'Until further design and planning work has been completed we are not in a position to confirm the extent of the land that would need to be acquired.

Ok, so what's the fuss?

'We have advised the family that if it did become necessary to buy land which they own, they would be fully compensated for it.'

Hmm. This smells like the family trying to press-blackmail HS2 into giving them a lot more money than their house is technically worth...