How can you tell whether a digitalisation project has actually made a difference?
Going live is a milestone, not proof. Whether a digitalised process has genuinely changed anything only becomes visible afterwards – and only if someone looks systematically.
A few months ago, a case-handling process in a public authority is switched over to a new online portal. The project closure is celebrated, the report to management is positive, and the number of digitally submitted applications rises visibly. Six months later, the same team in the same department is still dealing with the same follow-up queries, processing times have barely changed, and some staff print out the digitally submitted applications to keep working with them as before. On paper – in both the literal and the figurative sense – the project is a success. In terms of impact, it is not.
I encounter this pattern regularly in my advisory work, and the reason is simple: most digitalisation projects define success by the rollout, not by the impact that follows it. Once the system is introduced, the first applications go through, and the interface works, the project is considered complete. What happens afterwards is rarely observed with the same care as what was planned beforehand.
Going live is not proof of impact
The reason for this lies in the logic of the project itself. A digitalisation project has a project plan, a deadline and a budget – and all of that ends on the day of the launch. The organisation that carried the project usually disbands afterwards or turns to the next initiative. It is precisely at this point that the phase begins in which it becomes clear whether anything has actually worked.
A digitalised process can look successful for two very different reasons. Either because it has genuinely become faster, less error-prone or more resource-efficient. Or because it merely reproduces the old way of working in a new form – the application now runs digitally through the same five areas of responsibility as before, only nobody has to touch a piece of paper any more. Viewed from outside, and measured by indicators such as “number of digital applications” or “portal usage”, both cases look identical. Only a look at the actual processing time, the number of units involved and the error rate distinguishes one case from the other.
Four criteria that say more than the launch itself
Anyone who wants to know whether a digitalisation project has had an effect cannot avoid four questions, all of which can only be answered after go-live:
Has the throughput time actually shortened – measured from receipt to completion, not just up to digital capture? An application that is now captured digitally within seconds but then sits in a queue for just as long as before has gained nothing at the decisive point. Capture is rarely the bottleneck – processing is.
Has the number of units involved decreased, or has it simply been reproduced digitally? If a case still passes through five areas of responsibility, only electronically instead of on paper, nothing has changed at the actual root cause of delay. A genuine sign of impact is a falling number of handover points, not their acceleration.
Has the actual workload for staff changed – or has a media break been added instead? A reliable warning sign is when digitally submitted documents are printed out so they can be processed “as usual”. This suggests that the digitalisation ends at the surface and the underlying work process has remained unchanged.
Are there fewer follow-up queries and less rework than before, or does the new system create new sources of error? Digitised forms with unclear field labels or missing plausibility checks often generate more follow-up queries than the analogue equivalent, because errors are now technically possible that would previously have been spotted immediately at the counter.
What this means in practice
For public authorities that want to turn this observation into something concrete, four consequences follow. First: a digitalisation project needs an impact review date set several months after go-live – not as a formality, but with the same attention as the project closure itself. Second: the indicator that is collected must put processing time, not usage figures, at the centre. A high number of digital applications says nothing about whether the processing behind them has actually become faster.
Third: anyone who takes impact review seriously asks not only the project leads but the case handlers themselves – that is where it first becomes apparent whether a media break has occurred or whether something has genuinely become simpler. Fourth: an organisation that consistently checks its digitalisation projects for impact rather than for implementation gradually learns which initiatives genuinely create impact and which mainly shift effort elsewhere. That knowledge is worth more than any single rollout figure, because it can be carried over to the next project.
A short checklist for practice
Anyone who does not want to set up this impact review as a project in its own right can get a long way with a few recurring questions. How long does a case take today from receipt to completion, compared with the figure before the changeover – not estimated, but read off from the available case data? How many units are still involved in a typical case, and has that number actually fallen compared with before? Are digital documents printed out, scanned or manually transferred into another system at any point in the process? And has the number of follow-up queries handled by case workers risen or fallen since the new process went live? Anyone who can answer these four questions six months after implementation knows more about the project’s real success than any statistic on the new portal’s usage figures.
This kind of review costs little – a conversation with case handlers, a look at the available case data, half an hour of analysis. But it delivers something no project closure report can: an honest answer as to whether anything has genuinely improved for the people who process the case and for the citizens who submit it.
The success of a digitalisation project is not decided on the day it goes live, but by what can be measured months later. A public authority that takes this distinction seriously does not need an elaborate new bureaucracy for impact review – it needs the willingness to ask, once again after the fact, the same questions it asked before the project began.