Causes Behind Major Disruption to System at ICA Checkpoints on 31 March 2023

The following question stood in the name of Dr Lim Wee Kiak –

5 To ask the Minister for Home Affairs in light of the immigration clearance disruption due to a computer glitch at ICA checkpoints on 31 March 2023 (a) whether there is a backup system to automatically take over the operations under such circumstances; (b) what is the standard procedure for any work carried out on major IT systems to anticipate any glitches; (c) why is upgrading work not done during off-peak hours; and (d) how many travellers have missed their flights and whether they are compensated.

6 Mr Ang Wei Neng asked the Minister for Home Affairs (a) what are the main causes behind the major disruption to the immigration system at the ICA checkpoints on 31 March 2023; (b) how many people are affected by the disruption; and (c) what lessons can be learned from the disruption to prevent a future occurrence.

7 Mr Murali Pillai asked the Minister for Home Affairs (a) what is the estimated number of people who were affected by the five-hour system disruption at the land and air checkpoints on 31 March 2023 arising from a scheduled system upgrade that caused a system overload; (b) why was the problem hitherto unanticipated; (c) what is the Ministry's assessment of the contingency plan activated to deal with the disruption; and (d) what steps will be taken to strengthen the plan.

8 Mr Alex Yam asked the Minister for Home Affairs (a) whether the Ministry is satisfied with how ICA handled the recent service outage on 31 March 2023; and (b) how future outages can be prevented.

9 Mr Dennis Tan Lip Fong asked the Minister for Home Affairs (a) what was the cause of the system breakdown which led to the long delays in clearance across all immigration checkpoints in Singapore on 31 March 2023; (b) how many people were affected by the delay; (c) whether the ICA has worked out a solution which will prevent similar breakdowns in the future or, if not, at least to ensure that the system will be restored in a much shorter time.

Mr Ang Wei Neng (West Coast): Question No 5, Sir.

The Minister of State for Home Affairs (Assoc Prof Dr Muhammad Faishal Ibrahim) (for the Minister for Home Affairs): Mr Speaker, may I have your permission to take together Questions Nos 5 to 9 in today's Order Paper?

Mr Speaker: Yes, please.

Assoc Prof Dr Muhammad Faishal Ibrahim: Sir, my response will also cover the matters raised in the questions by Mr Desmond Choo, which are scheduled for a subsequent Sitting. I invite the Member to seek clarifications today, if need be. If the Member feels that his questions are addressed today, it may not be necessary for him to proceed with them during the next Sitting.

On 31 March 2023, at about 10.40 am, the Immigration and Checkpoints Authority (ICA) was conducting a pre-scheduled trial needed for an upgrade of the Multi-Modal Biometrics System (MMBS). The MMBS facilitates automated immigration clearance using travellers' biometrics.

Unfortunately, the trial caused a severe system overload. Automated clearance lanes at all the departure halls in Changi Airport, and certain automated lanes at Woodlands and Tuas Checkpoints were impacted. The rest of the automated lanes, including those at the sea checkpoints, remained operational.

Prior to the incident, ICA had been upgrading its systems progressively and cautiously, with 10 out of 12 systems enhanced. Only MMBS and one other system had not yet been upgraded. For MMBS, the system upgrade would involve replicating the large biometric database to an upgraded system. This has to be done continuously over a few days and the MMBS has to remain operational during this period.

Our standard procedure for any system upgrade is to conduct tests in the User Acceptance Test (UAT) environment before deployment. The UAT tests were successful. Subsequent trials in the production environment were also stable.

Next, we had planned to conduct further controlled trials during different times of the day, to ascertain that the system upgrade would not disrupt operations during the wee hours, off-peak hours and peak hours, before proceeding with the actual upgrade.

The wee hours trial was conducted on 15 March 2023, from 1.30 am to 3.30 am. The trial was successful.

The trial on 31 March 2023, when the incident happened, was the off-peak hours trial, to take place from 10.00 am to 2.00 pm. The vendors were on standby onsite and the plan was to recover the system within 30 minutes if the trial did not go well.

The trial caused the storage systems to overload at about 10.40 am and the process was aborted immediately. However, the extent of the overload was much more severe than anticipated and the vendors who were onsite had to work with their global support team to diagnose and reboot the servers. In the end, the recovery process took about four and a half hours, and MMBS recovered around 3.00 pm.

ICA immediately activated its Business Continuity Plan (BCP) once the system went down. Off-duty officers were recalled to help man manual immigration counters and perform crowd control. Across all the checkpoints, the failover process kicked in, and all the manual counters and certain automated lanes switched to back-up systems. Not all the automated lanes have this failover capability, as different models were procured over the years.

At the land checkpoints, travellers experienced delays of up to 30 minutes at the onset of the incident. At Woodlands Checkpoint, there was later another hour of delay for cars as the car arrival zone had to be converted to clear motorbikes manually. About 55,000 travellers passed through the two land checkpoints during the disruption.

Immigration clearance at the Airport's departure halls was significantly impacted. Travellers were re-directed to manual counters for immigration clearance. ICA immediately stepped up the manning of the manual counters through a combination of measures, such as recalling off-duty officers, deploying administrative staff and retaining the officers from the outgoing shift.

Changi Airport Group (CAG) assisted ICA by deploying additional Changi Youth Ambassadors and office staff, along with CAG's Duty Terminal Managers and their Customs, Immigration and Quarantine Team. They helped identify travellers whose flights were departing soon, so that their departure clearance could be prioritised. Public announcements were also made to appeal for such travellers to step forward.

During the disruption, about 30,000 passengers on 113 departure flights and 111 arrival flights, cleared immigration. Despite the measures taken by ICA and CAG, 21 out of the 30,000 cleared passengers missed their flights at Terminal 4. These travellers were subsequently offered to rebook on alternative flights within a week at no extra cost.

In light of the incident, ICA is reviewing the approach to the upgrade. The alternative approach that ICA was initially offered by the vendors was not tenable. It involved upgrading the system directly without replicating the database but may require five days of system downtime.

Even with the implementation of BCPs, degradation in service standards was inevitable, given the volume of travellers, and especially, at the onset of the incident. We apologise to affected travellers for the inconvenience caused and thank them for their understanding. I would also like to thank the ICA officers and CAG staff who had worked tirelessly, without complaint, to deal with the situation.

Going forward, ICA is in the process of implementing a multi-year plan, known as the New Clearance Concept (NCC), to transform immigration clearance across all the checkpoints. NCC will provide more seamless clearance and help ICA, with limited manpower resources, cope with the ever-rising traveller volume. Under NCC, all manual counters and automated lanes will be replaced in phases with the next generation Automated Border Clearance System (ABCS) gates, which will allow travellers to continue using automated clearance, even when MMBS is down.

In the meantime, the resiliency of existing automated lanes that have yet to be replaced by the ABCS gates will also be enhanced in the second half of 2023.

Sir, Singapore is one of the first countries in the world to implement NCC-type capabilities. Some of these capabilities will be implemented in a form and manner not used or tested elsewhere, as we customise them to suit our operational needs. As such, while we will work closely with all stakeholders to further minimise technical disruptions and enhance system resiliency, the reality is, disruptions will still happen now and then. When they do, we will make sure we have robust contingency plans in place and try to recover as fast as possible. I hope for the public and Members' understanding.

Mr Speaker: Mr Alex Yam.

Mr Alex Yam (Marsiling-Yew Tee): Speaker, first, allow me to thank the ICA frontliners on what is, perhaps, is going to be the busiest weekend this year, with an estimated two million to pass through our checkpoints for this Hari Raya weekend.

I have two supplementary questions for the Minister. One, during the period of the glitch, was there any loss or breach of data detected? And as the disruption lasted over close to five hours, were there any instances of mis-clearance at immigration or security concerns during the incident itself?

Assoc Prof Dr Muhammad Faishal Ibrahim: Sir, I thank the Member for the supplementary questions. There was no loss or breach of data during the incident. And in fact, ICA checks showed that there were no unusual activities found. With regard to mis-clearance, there were back-up processes in place, so that itself minimised the risk of mis-clearance. What we did was that we continued to verify the traveller's identity using the back-up system and also, just to be on the safe side, the ICA officers also did face-to-face checks with the passport for identification purposes.

Mr Speaker: Mr Ang Wei Neng.

Mr Ang Wei Neng (West Coast): Speaker, Sir, we also like to thank the CAG staff for helping during the disruptions, to minimise the flight disruptions – only 21 passengers missed their flights, which is very commendable.

I have a couple of supplementary questions. One is that the Minister of State mentioned that during the disruptions at Changi Airport and the land checkpoints, the sea checkpoints were not affected. So, the question is that why did we not do the upgrading at the sea checkpoints first, rather than to do it straightaway at the busiest part of the Changi Airport, as well as the land checkpoints?

And two, during the whole upgrading, did the Ministry of Home Affairs (MHA) check out what is the cause of these disruptions; and for the overloading of the database, if it was the fault of the contractors, are we imposing any liquidated damages?

And thirdly, how could we have some assurance that during this Hari Raya weekend, a similar thing will not happen again?

Assoc Prof Dr Muhammad Faishal Ibrahim: Sir, I thank the Member for the supplementary questions. The travel volume at the sea checkpoint on 31 March 2023 was generally manageable. And, in fact, ICA manual counters across all checkpoints and the automated lanes at the sea checkpoints remained operational. So, essentially, these were not affected. The passenger volume in that sea checkpoint were relatively low and travellers were able to clear via the automated lanes and they were not affected.

As I shared in my answer, some of the models of these lanes were procured at different periods. As such, they responded differently to the whole process.

I thank the Member for the next question, whereby, we needed to do the trial at all the checkpoints at one time, because this MMBS is integral to all checkpoint operations. As such, the trail was done at the backend common infrastructure that supports the MMBS system, which in turn supports all the checkpoints.

So, it was not possible to conduct the trial, checkpoint by checkpoint, or just doing it at the sea checkpoint, or doing it in batches.

Nevertheless, in an effort to test the system load before proceeding with the actual upgrade, our approach was to conduct additional controlled trials during different times of the day.

This is something that we thought we should do so that we can prevent more serious disruptions when the system upgrade has been done. However, as I have shared earlier, we have contingency and mitigation plans, whenever there are such disruptions that happen.

Mr Speaker: Dr Tan Wu Meng. I believe some of your Clementi residents were affected by this?

Dr Tan Wu Meng (Jurong): I thank Mr Speaker and the Minister of State. I can indeed confirm there were Clementi residents who were affected and concerned by what happened. I also want to put on record my thanks for the ICA and CAG officers who did their best under these unexpected and very difficult circumstances. Can I ask the Minister two supplementary questions?

Firstly, given that the MMBS would be considered critical information infrastructure for our checkpoints with ICA, what existing back-up redundancy is there during business-as-usual? That is the first question.

My second question is whether there are best practices throughout Government or internationally, on how we can take lessons from such situations to improve the resilience of our critical information infrastructure, especially at checkpoints where any delay can lead to very extensive backlogs and implications for Singapore's reputation?

Assoc Prof Dr Muhammad Faishal Ibrahim: Sir, I thank the Member for the supplementary question. As I shared earlier, we are actually at the frontier of technological development, with regard to clearance matters, especially border security.

If we look at what ICA has been doing, especially in recent years, it is really to see how we can improve technologically, to provide a more seamless experience and travel experience for the passengers, whichever checkpoints that they go through.

We also have very good contacts with our counterparts overseas. More often than not, they would like to come to learn from us. Nevertheless, we are always open to learn from the other countries too.

So, I assure the Member that this is a process that we have been undertaking and we want to make sure that we provide one of the best services.

But looking into the security aspect of it, as Members know, it is not only about efficiency, but we must make sure that Singapore remains safe and secured.

With regard to the MMBS, currently, we have the back-up system. As we saw during the disruption, we were able to go into manual clearance and we were able to bring out other aspects of the BCP. However, when NCC is rolled out, this would be where, even though there will be disruptions, the system will enable the passenger to still be able to use the biometrics component in the travel experience. So, we are essentially building a system that is more robust and will have more resiliency in the future.

Mr Speaker: Mr Dennis Tan.

Mr Dennis Tan Lip Fong (Hougang): Thank you, Speaker. I have one supplementary question – whether there are any claims faced by ICA arising from the delays, coming from passengers or from any commercial companies?

Assoc Prof Dr Muhammad Faishal Ibrahim: As I have shared earlier, 21 travellers missed their flights, and these affected travellers were offered to rebook on alternative flights within a week at no extra cost. Of the 21 passengers who had missed their flights, 11 passengers rebooked their flights and departed on the same day, four passengers rebooked their flights and departed the next day on 1 April 2023, while another two passengers departed by land and two passengers remained in Singapore. [Please refer to "Clarification by Minister of State for Home Affairs", Official Report, 21 April 2023, Vol 95, Issue 101, Correction By Written Statement section.]