Precious failures

Failure is not something people want to talk about. Yet, contemporary approaches to development, including IT development, stress that failure is inevitable  – the more we attempt to innovate, the more we see failures – and useful. The wisdom is to prepare for failure and learn from it. As the leader in public sector digitalisation, Estonia has surely failed many times. Kaire Kasearu, Head of IT Financing; and Ott Velsberg, Government Chief Data Officer, both at the Ministry of Economic Affairs and Communication, share some of their experiences from years of work in Estonian public sector development projects.

“It is true that publicly, people often feel uncomfortable sharing their failures. Privately, however, we discuss them thoroughly and try to find the best solutions to minimise risks in the future. Recently, we have even come up with platforms where sharing failures have become more common. Because learning process is a normal part of the development and helps to understand what goes wrong and how we could do it a better way,” Ms. Kasearu states.

Too rigid development

“I think the biggest problems we have had come from procuring too big and rigid development processes,” Ms. Kasearu states. “For example, with the iconic failure of SKAIS 2 (the information system upgrade for Social Security Office) we started to develop a huge project without dividing it into smaller, manageable parts. When something changed – either legislation or a developer bails out, it was almost impossible to make relevant changes in the development process quickly enough.”

Her realisation is backed by similar findings by the State Audit Office’s report from 2019, which pointed to legislative amendments made during the development process which notably slowed down the development work or gave rise to extra costs to developments of SKAIS 2 and 3 other major public sector systems.

Based on these lessons, principles of iterative development have been increasingly implemented in the Estonian public sector according to Mr. Velsberg. This entails focusing on the problem rather than the solution in order to prevent early-stage lock-ins, and encourage creativity.

“This is especially useful regarding data-science projects,” Mr.

Velsberg points out. “For example, when we were developing traffic accident prevention solution we realised that we might benefit from using data about large-scale events and gatherings. No-one had even thought about it in the beginning.”

No overview of available data

Another early-stage problem that can lead to significant failures comes from the lack of an overview of what data is available in the public sector. Mr. Velsberg describes how hundreds of databases are regularly filled with terabytes of information which may or may not fuel a particular solution. Hence, the final solution can differ drastically based on available data. This, however, requires both knowledge of and the inclusion of experts.

“Availability of data and expert input are connected to each other,” Mr. Velsberg explains. “Although some data may be available, experts’ role is crucial in validating the use-case and where necessary annotating the data.”

Mr. Velsberg shares his experience of the recent development of a conversation robot. He describes how running a test project validated that the implementation can save a lot of time and money. Because data quality can drastically change the outcome, test developments ensure that the final product is actually useful both for the organization and for the end-user.

 Not enough competition

„Because most of the development in Estonian public sector is procured, a tangible problem is consortium lock-in,“ Ms. Kasearu points to a third bottleneck. “It is inevitable that there are delays and setbacks during large-scale developments. The problem is that it is difficult to find others to pass on the work when necessary.”

The size of the country surely has an effect. Thorough background checks clear only a handful of companies to compete for government projects. In this environment, even small changes in the business environment have an effect. For example, with the recent sales of IceFire, an Estonian company that has developed many public systems, the global giant checkout.com, competition declined sharply.

Although one cannot simply change the size of the country, changes can be made to the administrative system. Ms. Kasearu describes principles of in-consortium competition that are currently updated in the public procurement contracting processes. “This is something we are currently working on along with our legal team,” Ms. Kasearu explains.

Similarly, other innovative approaches bring up the necessity for change in legislation. For example, as public sector organisations increasingly use hackathons as the development mechanism, procurement laws that did not allow for developing a solution and developing wider implementation by the same company needed to be changed. In this regard, the small size of Estonia comes in handy, as legislation changes can be made quickly.

Underfinanced upkeep

Both Ms. Kasearu and State Audit Office report stress the importance of maintenance to IT systems. “In a sense, this is a chronic failure – the lack of attention to the upkeep of software systems,” Ms. Kasearu laughs. “Politicians are keen to promote a specific solution and argue for securing investments for it but often downplay or simply forget altogether the necessity to keep the system running after it has been implemented.”

According to the State Audit Office Report the planning phase of four major audited projects did not include an analysis of future maintenance of developed software, the provision of support for the agencies’ core processes, or an estimate of the annual maintenance and improvement costs. This resulted in getting stuck to what Ms. Kasearu titles as “financial maintenance hole”.

“I think this is a lesson we had to learn the hard way,” Ms. Kasearu sighs. “We just had to keep these systems running. Now, no development is planned without budgeting for maintenance, improvement, and support.”

User unfriendly

As for the final lesson learned from failure Ms. Kasearu and Mr. Velsberg talk about these instances when projects are developed and run technically smoothly, but the result is suboptimal for the end-user. For example, a widely publicised – but luckily easily repaired – issue came up during the Covid-19 crisis when the digital health care system was clogged by thousands of users trying to book vaccination times, blocking the system for the rest of the users. Technically, the solution was simple – to divert some traffic elsewhere. It only required being receptive to complaints and reacting quickly.

Also the State Audit Office’s report point to the general shortcoming of not gathering feedback from end-users. For 6 of the 9 audited projects, it was hard to determine whether the developed software met the user’s needs or not because regular user satisfaction or usability surveys were not conducted after the completion of development work. Luckily, this issue is among the easiest to fix as end-user feedback is systematically woven into the upkeep and development processes.

Conclusion: Share your failures!

Both Ms. Kasearu and Mr. Velsberg are unanimous about the main lesson about failures: share them! Private discussions are a start, but public ones are even better. They talk about two Estonian platforms where failures are frequently discussed: a panel of digital architecture where about 50 engineers produce feedback on developments; and a working group for AI and data science, where 3-4 organisations share their experience monthly to an audience of 200 people. They both also agree that it is difficult to estimate the savings these panels have produced – but they are big!

✈️  Can’t travel but want to hear the e-Estonia story or implement e-services in your country or company? Take a look at our services and get in touch – we’ve got you covered!