Despair over UK Government IT

[silverback]

Don't know if international loungers have seen today's UK news, but 16000 positive COVID-19 tests - and thus, access to their close associates, have been lost. This is, according to the media, because Public Health England has been a) using an Excel spreadsheet to record test results and b) have been using 'outdated' software.
This seems to hinge on older versions of Excel being limited to 65000 rows whereas 'modern' versions can record up to one million rows.
The central spreadsheet was created by receiving spreadsheets from outlying units carrying out tests, and it seems that while importing these, no-one noticed that 16000 rows had been chopped off.
Is it only the UK government that has such problems with IT or do other governments have IT foul ups? Why aren't they using Office 365 with automatic updates? Why are they using Excel and not a database?
As I said in the title, I despair.
Silverback

[StuartR]

Being very careful not to make political statements here...

I think the underlying issue was even worse than using Excel as a database. It looks to me as though they were dumping Excel files into a folder instead of creating an API. This meant that there was no acknowledgement that each record had been received, and it was easy to lose thousands of records without noticing.

[GeoffW]

The solution they were using suggests that the initial solution was a quick bandaid for a small quantity of cases, something which was needed very rapidly. The rapid increase in cases meant that the bandaid solution kept being used well past its use by date, probably without people even thinking about it.

I don't know if it was the version of Excel which was the problem, rather the version of the spreadsheet saved. CSV files were sent to a central location, and the results saved into an XLS file rather than the later post 2007 XLSX format.

[stuck]

...without people even thinking about it.

People rarely do. Hanlon's razor, "never attribute to malice that which is adequately explained by stupidity".

Ken

[Argus]

Is it only the UK government that has such problems with IT or do other governments have IT foul ups?

Look no further ... Exhibit A.
(Not IT per se, as in mismanaged software/data, rather a very reckless decision about subcontracting, sensitive data, not following advice ...)

[jonwallace]

Disclaimer: Before retiral I was a Biomedical Scientist for NHS Scotland. My area of specialty was Microbiology, but since 1998ish I looked after IT matters for the Microbiology Lab. Initially this meant overseeing the implementation of a resource management package, but soon spread into making sure the 3 PCs we had for general use in the "computer room" (a large cupboard, shared with a freeze drier...) ticked over, and supporting those hardy souls who were dipping their toe into the word processing water. Like a lot of folk in this situation, it soon became "if it had a plug on it, it was mine".

When we rolled out the Lab IT System I became responsible for retrieving data for all sorts of purposes, for all sorts of people as well as managing the 50-odd PCs in the lab and making sure that our analysers could talk to the lab system.

My role in this was fairly unique in that right up until I retired in 2019, few diagnostic labs in the UK using Lab systems from our supplier had dedicated IT people. Mostly, the data mining was done by staff in between doing actual lab or management duties.

My experience of data mining in Diagnostic Labs was that it was either done using direct queries on the database (Uniquery latterly, but an earlier supplier used SQL) or since 2016, writing Business Objects queries against a Universe supplied by the Lab System people. The rather crappy data structures produced by these queries were usually imported into excel for further processing, with all the headaches that entails (leading zeros being stripped off our 10 digit patient identifier being a favourite ).

This worked ok with the reasonably small numbers produced by "How many Organism x sensitive to antibiotic y in ward z did we get last year?" style queries, but the occasional one that produced thousands of lines was a nightmare. Then you pass the data to people who only have a vague idea of what they wanted in the first place, and an even vaguer idea of what to do with it when they got it.

So I can see why lots of small Labs, running a variety of competing Lab Systems, might not be able to supply the data in a format that the collators wanted. And if I remember correctly, Business Objects didn't export XLSX files...

[John Gray]

Most government large IT projects fail in some fashion, spectacularly or just in some areas, largely because big systems are frequently not well understood, 'scope creep' occurs during implementation, and unintended consequences appear.

The Register has a [beware, strongly-worded] article on the folly of misusing Excel spreadsheets, including a link to a very long list of Excel-based disasters and unhappinesses.

[ChrisGreaves]

...including a link to a very long list of Excel-based disasters and unhappinesses.

John, you know how I hate to be pedantic, but your is misplaced here, if only because the linked article contains the unforgettable phrase:-
and was also purposely implemented in Excel, for the purpose of bug compatibility.[66]

(I'm sorry; I haven't yet worked out how to apply the [strikethrough] to the smiley [rotfl]. )
Cheers
Chris

[silverback]

. . .including a link to a very long list of Excel-based disasters and unhappinesses.

And the link provided by Argus.

Oh
Silverback

[silverback]

Thank goodness. The powers that be have taken notice of Eileen's Loungers and changed their practices.

Even Public Health England seems unable to kick the habit. According to reports, it has temporarily solved the problem by splitting up its big, broken Excel spreadsheets into batches of smaller Excel spreadsheets.

Daily Telegraph, Business News, 7th October 2020

oh, for sake.
Silverback

[LisaGreen]

In following some of the links given, the wiki one for excel has introduced me to a new term.

"bug compatability".

I'm Gobsmakked!

Lisa

[StuartR]

bug compatibility has been very important over the years. I'm pretty sure that Excel introduced some of these when it first came out to support spreadsheets that were developed in Lotus123

[ChrisGreaves]

In following some of the links given, the wiki one for excel has introduced me to a new term. "bug compatibility". I'm Gobsmakked! Lisa

Whew!
Hi Lisa. For a while there I was flummoxed; I thought that by Gobsnacked you meant a mouth stuffed full of zucchini, which would be a relief.

Yesterday I was banned from Swyers for a month when they caught me unloading zucchini from my orange satchel into their vegetable bins. There's another avenue of disposal closed down .
That leaves Foodland as my only hope.

Bonavista_20201008_103935.JPG

The one good thing about zucchini is that it is bug-incompatible, as you can see from the 7+-pound monster that made its way into my backside porch yesterday. The bugs and slugs leave zucchini alone, which ought to serve as a warning to the rest of us.
Today I pulled everything except the cabbage plants out of the raised beds. "That's It!" I would have heard myself say had I said it.
The bugs have eaten so much of the cabbage leaves that there was not enough leaf left to make food for the plants.

I possess, as you know, a conscientiously evil mind and so I have left the cabbage plant skeletons out there. My devious plan is to starve the bugs to death.

Cheers
Chris

[ChrisGreaves]

... but 16000 positive COVID-19 tests - and thus, access to their close associates, have been lost.

Untitled.png

You can relax Brian!
Crud implementation is everywhere.

Above is a screen snapshot of part of a web page designed to help a certain university (whose name I shall withhold so as not to embarrass The University Of Western Australia) update my details. It seems that they want to EMAIL the Alumni magazine to me instead of posting it, but they don't have my email address.
So I filled in the form and arrived at a 3-button option that allowed me to receive the publication by postal mail (Yes! Pumps fist in air), so I clicked ON the send-by-post option and made sure that the <send-by-emai> and <don't send> options were clicked OFF.

Not sure at this stage why they want my email address, but allow me to opt out of email and keep to glossy copies that pad out my bookshelves.

But what really caught my eye was the calendar.
We are talking here about graduates, so say about 20 or more years old. Why initialize a calendar to the current month (October 2020 as I type this) when a more appropriate date might be useful?!!???
I think of October 2000 as a date that will be close to the left-hand end of a curve of distribution of ages.
I think of a Median date which ought to minimize the clicking backward and forward in time.
I think of a weighted median which ought to minimize a predicted total number of clicks over the life of the web page

Anything at all as long as it eliminates some new-born baby in <your favorite town here> from making use of a web page for University Graduates.
I mean, what does that say about the level of education of those graduates.
Ooops!

Cheers
Chris

[HansV]

Yesterday I was banned from Swyers for a month when they caught me unloading zucchini from my orange satchel into their vegetable bins.

Wrecking the economy, eh?

[ChrisGreaves]

Yesterday I was banned from Swyers for a month when they caught me unloading zucchini from my orange satchel into their vegetable bins.

Wrecking the economy, eh?

I should have known it. Last week they nailed me for uploading zucchini into the cucumber section; I did not know that shoppers here were so critical.
There again, perhaps the shoppers were just upset that I'd found a novel way to get rid of mes marrows that they had missed.

Some folks just don't have the courgette of their convictions.
I am the other way around.
Cheers
Chris

[GeoffW]

The big zucchinis are known as hippo zucchinis. Unusually, not elephant zucchinis.

Hence the shoppers were hippo critical.

[ChrisGreaves]

The big zucchinis are known as hippo zucchinis. Unusually, not elephant zucchinis.
Hence the shoppers were hippo critical.

I thought that they were called hip-oh! zucchinis because the only way people can carry them out to their cars is to hoist them onto their hips.
I just got rid of another Big One. Not the 7-pounder, though.
Actually, If I'd been on the ball I would have used four at a time and refused to let vehicles leave my property without their organic wheel chocks.
Cheers
Chris

[John Gray]

Local newspaper website headline: "Chris Greaves has marrowing experience!"

[BobH]

In an attempt that I expect to be in vain, I'm returning to the subject of IT.

Chris and I date our nascence to the Triassic Period of IT, the period when paper cards with holes punched into them were used to feed the beast and programmers were any warm body that would agree to be called one.

I wrote some bad code in my early days. I learned it was bad because I was not forcing out every extraneous byte of either program instructions or data so that our 4k (yes I said 4K) of RAM (which we - in our ignorance - called the 'memory unit) system could digest and complete what we wished to accomplish. IOW, I learnt to twist bits until they byte.

As the years passed and I progressed from operator to programmer to analyst and system designer to manager of those who did those things to the CIO responsible for all those transgressions plus the evaluation, purchase and implementation of hardware components and packaged software, I was constantly fighting the battle of 'scope creep.' IBM had their motto: THINK. Mine was not as terse but I had a card that I insisted that system designers and programmers give to every user. It read: JUST BECAUSE WE CAN DOESN'T MEAN WE SHOULD.

The introduction of what I choose to call the 'small platform' instead, of the PC, came before I retired. I was amazed, and had to fight on 3 fronts: 1) dealing with those folks who, with no training and no concept of IT management - much less data structures and management - were given software and told to use it; 2) those who similarly were given such tools and who thought they could replace IT because they knew everything when they knew how to boot one and became marginally proficient with certain software; and 3) all of those who were told that the IT department would cover their hind sides and to just call them. Soon after that line was crossed, IT management required more 'help desk' organization and staffing than running a large mainframe and complex mainframe applications.

Thank you for this post. Thank you especially because I learned of Hanlon's razor. Oh how I wish I had known it long ago for certainly it would have been useful in explaining things to the BOD and equity holders.

Thank you!