Singapore's Preparedness for Global Technology Outages and Impact of Recent CrowdStrike Incident

The following question stood in the name of Mr Ang Wei Neng –

5 To ask the Minister for Digital Development and Information (a) how many websites or phone applications of Government and Statutory Boards were affected by the recent outage on 19 July 2024 that was caused by a software update; (b) how long did the affected websites or phone applications take to restore it back to normalcy; and (c) what are the lessons learnt from the said outage.

6 Mr Don Wee asked the Minister for Digital Development and Information (a) how many Government agencies were affected by the software update from cybersecurity firm CrowdStrike which caused a major technological outage globally on 19 July 2024; and (b) whether the Singapore Government subscribes to CrowdStrike’s services.

7 Mr Alex Yam asked the Minister for Digital Development and Information (a) what is the Ministry’s assessment of the impact to Singapore due to the tech outage caused by the cybersecurity firm CrowdStrike’s software update on 19 July 2024; (b) whether any Government agencies or critical public services have been affected by the outage; and (c) what is the Ministry’s contingency plans for such global tech outages that may impact Singapore’s socio-economic security.

8 Mr Gerald Giam Yean Song asked the Minister for Digital Development and Information with regard to the recent IT outage caused by CrowdStrike (a) whether the Cybersecurity Agency of Singapore (CSA) has updated its threat and risk assessment protocols to cover supply chain risks of this nature; (b) if so, whether these updated protocols will be implemented across all critical information infrastructure (CII); and (c) what new strategies are being considered to enhance the resilience of CIIs against systemic propagated shocks that are not directly linked to cybersecurity threats.

9 Ms Hany Soh asked the Minister for Digital Development and Information with regard to the global IT outage that occurred on 19 July 2024 (a) whether the Ministry has conducted an overall local impact assessment; and (b) if so, what are the Ministry’s findings and whether any measures have been implemented to prevent such a recurrence.

10 Mr Yip Hon Weng asked the Minister for Digital Development and Information (a) whether the Government will conduct an After-Action Review (AAR) to assess Singapore's preparedness and response, and identify areas for improvement, in light of the recent global technology outage; (b) what key considerations will guide such AAR; and (c) whether the Government will (i) develop a local technology ecosystem or (ii) stipulate guidelines that will implement redundancy systems to insulate our essential services from global disruptions as part of digital resilience.

Mr Alex Yam (Marsiling-Yew Tee): Question No 5, please.

The Minister for Digital Development and Information (Mrs Josephine Teo): Mr Speaker, may I have your permission to answer together Oral Question Nos 5 to 10 on today's Order Paper, Written Question Nos 27 and 28 on today's Order Paper, and Question Nos 60, 64, 65, 66, 68 and 69 on yesterday's Order Paper, filed by Members like Miss Cheryl Chan, Ms Ng Ling Ling and Mr Desmond Choo, as they are all related to the outage of information technology (IT) systems caused by CrowdStrike's faulty software update on 19 July 2024?

Mr Speaker: Please proceed.

Mrs Josephine Teo: Mr Speaker, my response will also cover the matters raised in the Oral Questions by Assoc Prof Razwana Begum^1,2,3, which are scheduled for a subsequent Sitting. With your permission, Sir, I would also like to invite all interested Members to seek clarifications after I have given my reply today. If the questions have been addressed, it may not be necessary to proceed with the Questions for future Sittings.

Mr Speaker: Please proceed.

Mrs Josephine Teo: Sir, on 19 July 2024, a faulty software update by cybersecurity service provider CrowdStrike disrupted major services around the world. Images of the now infamous "blue screen of death" appeared in media news cycles and attracted significant public attention. According to public reports, outages were experienced by users of the Microsoft Windows operating system that adopted CrowdStrike’s Falcon Endpoint Detection and Response (EDR) solution. It is a security solution that requires frequent and timely updates to be effective.

The Members' questions fall broadly into two categories: first, what is the impact of the outage in Singapore, particularly in relation to services provided by the Government; second, what are the lessons learnt, particularly in relation to the resilience of our IT systems.

Fortuitously, Government services and most essential services in Singapore were unaffected by the outages. However, some businesses that use CrowdStrike's Falcon EDR solution were affected. In most cases, the impact was to internal staff. In a minority of the cases, customers were impacted due to service disruptions. Prominent examples of these were the passenger check-in for some airlines at Changi Terminal 4 and gantry operations at some Housing and Development Board carparks.

Customers of affected businesses met with delays and were inconvenienced. However, business continuity plans kicked in. These included switching over to manual processes, such as for flight ticketing and check-in. The Singapore Cyber Emergency Response Team (SingCERT) of the Cyber Security Agency of Singapore (CSA) also quickly issued an advisory to guide affected systems administrators and users on how to manually recover their systems. Most of the affected systems recovered within a day and services returned to normal.

As Members know, IT systems may experience outages and disruptions from time to time. In this particular instance, it is not yet fully understood what caused a relatively routine software update to have created such major disruptions around the world. My Ministry has set up an internal task force to engage relevant partners to gain insights into the incident and assess if further measures should be taken to improve Singapore's resilience when such disruptions occur.

In the meantime, one key lesson can already be reinforced. As we have said on previous occasions, even with best efforts, not all disruptions can be prevented. Systems owners should, therefore, have plans in place to help them recover quickly from unexpected disturbances.

On its part, the Government adopts a risk-based approach to ensure that our critical systems and Essential Services (ES) are resilient. Critical Information Infrastructures (CIIs), ES and Government services are all subject to stringent requirements and have to put in place robust business continuity plans, disaster recovery plans and incident response plans.

The Cybersecurity Act and specific sectoral regulations hold CII and key ES operators accountable for meeting the baseline security and resilience requirements. This includes timely review of risk assessments and audits. For example, Government agencies using third-party software in their information and communication technologies (ICT) systems have to do a thorough risk assessment and put in place the necessary mitigation measures. CSA also established the CII Supply Chain Programme to better manage key vendor supply chain risks.

Businesses must also play their part to improve their resilience when disruptions occur and recognise that it is in their own as well as their customers' interests to do so. When things are running smoothly, businesses may question why they should incur cost or prioritise efforts to assess and improve their resilience measures. Unfortunately, some may not take appropriate action until it is too late.

We therefore encourage businesses to conduct their own risk management and assessment measures and put in place the appropriate business continuity plans to help business continuity in the event of a disruption. SingCERT has recently published an advisory on building digital resiliency, which can be found on CSA's website. As part of the support for enterprises' digitalisation, my Ministry offers other practical resources and financial assistance to encourage robust IT practices. This includes CSA's cybersecurity toolkits and IMDA's SMEs Go Digital Programme.

While these efforts may not specifically address IT outages like the one related to CrowdStrike, they can help businesses prevent incidents and recover more quickly should disruptions occur. I also encourage all businesses to take advantage of the Government's resource support to strengthen their digital resilience.

Mr Speaker: Mr Alex Yam.

Mr Alex Yam (Marsiling-Yew Tee): Mr Speaker, I thank the hon Minister. The Minister mentioned that businesses should have contingencies in place. Could I ask the Minister, with regard to critical infrastructure, for example, businesses and airlines that are operating at our airports, whether we should have compulsory requirements? Because as we saw at our airport during the outage, most airlines were able to cope, whereas some airlines faced longer disruptions. This experience, although it is related to the airlines, does also reflect on Singapore on a whole. As such, will the Ministry consider making it compulsory for some businesses to adopt contingency plans?

Mrs Josephine Teo: Mr Speaker, actually, it is in the businesses' own interest to have contingency plans in place. If they are affected, or if their customers are affected, certainly, the operational impact could be considerable. Certainly, their reputations are also at risk. The Government has to adopt a risk-based approach and that would include being quite careful about the occasions when we impose compulsory requirements.

If we attempt to prescribe the measures that businesses must take and we are not careful as to the occasions that we do so, firstly, it could take agency and the sense of ownership away from the IT systems' owners, because then the thinking could be that, if the Government does not say so, then we do not need to do. That would be to the detriment of all of us.

Secondly, it is also, from a sense of humility, that we decide that this is not a good approach, because there are so many different components that go into a system's resilience. To imagine that we have full understanding of all the different things that could cause major disruptions is, I believe, unwise.

I should also say that, in this particular instance, it was a fairly innocuous software update. No one could have expected the amount of disruption that it caused around the world. So, I would say that we will, in certain instances, require measures to be mandated. But in the vast majority of the cases, it is important to allow the systems owners and, indeed, to require the systems owners to take ownership, to build up their systems' resilience. That is still the approach that we would prefer to take.

Mr Speaker: I see many hands up. I will call Members but let us keep the questions succinct and the answers likewise. Miss Cheryl Chan.

Miss Cheryl Chan Wei Ling (East Coast): Speaker, I have a supplementary question for the Minister. I agree with the Minister that these software systems are relatively complex and, generally, you do not just have a single supplier supplying to the overall system. The question would be, for the Government agencies, as we are going more digital or promoting digitalisation in our society, how can we ensure there is sufficient coordination with third-party suppliers, such that any change that they make on their independent end does not affect our overall system?

Mrs Josephine Teo: Sir, it is an excellent question. If I could seek your indulgence, it deserves a fuller response.

Firstly, the use of third-party software is unavoidable because technological systems are complex. Third-party software can offer a wide range of functionalities to meet the requirements of various organisations, the Government included, and this saves time and resources from having to develop such software from scratch. When using third-party software in their ICT systems, Government agencies are required to undergo a thorough risk assessment and to put in place the mitigating measures. That is already baseline.

To the extent possible, agencies must put in place quality assurance measures to ensure that the software changes that will be inevitable will not introduce errors in critical systems. Such measures include testing software updates in controlled settings prior to going live. IT people are very familiar with this – you test it in a controlled environment before you put it to the overall system and then see what happens.

We also deploy software changes progressively to small groups of users before rolling it out widely. This usually allows us to catch and isolate issues early. But I say "usually" because it does not happen all the time. There are ways in which the system components interact with each other that are not always possible to map out so clearly. In addition, agencies with critical systems are required to review the change management processes of their software providers through regular independent audits. This ensures that software changes can be rolled out smoothly and securely.

In some instances, depending on the service provided, it may be beyond the control of users, including governments. For example, Software as a Service will put the onus on the vendor to ensure that their software remains secure and available for use. This is something that we will have to keep in mind and see what we can do about.

To the Member's specific questions – when we have exercises, for example, are different partners in the supply chain involved? Possibly, if they have a major impact on the system's usability as well as resilience, but there are so many vendors involved and it may not be possible to include all of them in the exercises.

Mr Speaker: Mr Gerald Giam.

Mr Gerald Giam Yean Song (Aljunied): Sir, I have two supplementary questions for the Minister.

Sir, much of the legislation introduced recently covers cybersecurity risks. However, as this major outage caused by CrowdStrike was not a cybersecurity attack but a supply chain failure, specifically, a bug in the software update from the vendor, does the current legislation adequately address the risks posed by supply chain failures in digital infrastructure? And would the impending Digital Infrastructure Act have mitigated the impact of the outage caused by CrowdStrike?

Secondly, this incident also highlighted the risk of a single point of failure having widespread impact on digital infrastructure. A similar issue was observed with the Mobile Guardian mobile device management software used in schools which have affected thousands of students, many of whom discovered to their horror that they lost their study notes just before their weighted assessments. Is the Government looking at encouraging or even mandating operators of CII to review their IT procurement practices and diversify their sources of vendors so that no one software can bring down an entire system?

Mrs Josephine Teo: Mr Speaker, the hon Member has a number of questions rolled into his supplementary. Let me try and deal with supply chain risks more broadly and what we do about them in CII.

In fact, CSA's threat and risk assessment for CII already cover supply chain risks of this nature today. So, it is not something that is new or unknown or we are caught by surprise in that regard. We have put in place measures to tackle the supply chain risks that the CIIs face, holistically.

For example, under the Cybersecurity Code of Practice, CII owners must adopt, to the extent possible, the Defence by Diversity principle. To explain to Members what this means – take any IT system. If the system does not have much diversity in terms of its defences and a single attack vector can immobilise the most key components of the system, then the system does not have Defence by Diversity. If the system wants to achieve Defence by Diversity, what the system owner needs to do is to have a different variety of vendors that the system owner works with, different system architectures to the extent possible, different configurations, different communications pathways and, indeed, different vendors for whether hardware or software.

So, Defence by Diversity is something that we ask CII owners to adopt to the extent possible. That is also why, in many instances, we emphasise how software systems ought to be interoperable. Because if interoperability is not common, then you can imagine that for system owners, they are stuck. Once system owners use system A, they must also use the other related systems for A. If they cannot choose to use B, then they do not have that diversity. So, that is a very important principle. It is already part of the way we operate.

The threat and risk assessments are also reviewed at regular intervals so that they remain up to date. Additionally, CSA has also introduced the CII Supply Chain Programme. I mentioned this earlier. This was in 2022. What does the Supply Chain Programme help, whether it is Government agencies or, indeed, any other system owner, achieve? It gives them a toolkit to help identify and inventorise their vendors.

It is very often the case that when something happens, system owners do not know what hit them, even if it has been reported in the media that this was a particular software failure, because the system owners may not know that their systems contain this particular software. So, a simple fact of inventorising what goes into your system is already no small feat because of the number of vendors involved along the whole supply chain.

So, the toolkit helps the systems owners to also assess and rate their cyber supply chain risk using a standardised vendor management methodology. So, that is another thing that we have put in place. To the Member's point about a single point of failure, we agree and there are ways in which we mitigate against that.

For Government systems, critical functions are required to cater for redundancy. You must be able to have a failover, and this could include both the hardware and software components, networks and databases, as well as even aspects of the physical environment. So, this is something that is already practised and you can always improve them.

The Digital Infrastructure Act that we had talked about will seek to improve resilience that are over and above what needs to be done within the cyber system environment. We will have more details. We are in the process of consulting with the various stakeholders and, in due course, we will be able to say more about that.

Mr Speaker: Ms Hany Soh.

Ms Hany Soh (Marsiling-Yew Tee): I thank the Minister for sharing with us in relation to answering our Parliamentary Questions (PQs). I have one supplementary question. I understand that investigation is still ongoing, but I am curious to find out whether the relevant investigation team has already been in touch with Microsoft to understand what are the preliminary issues and what are the undertakings that they will endeavour to do to prevent such a major technical glitch from happening again.

Mrs Josephine Teo: Sir, the short answer to the Member's question is yes, from the first hour.

Mr Speaker: Mr Yip Hon Weng.

Mr Yip Hon Weng (Yio Chu Kang): Mr Speaker, I thank the Minister for her response. My supplementary questions pertain to public confidence in Government digital services. What measures are in place to ensure that public confidence in Government digital systems is maintained, especially in the wake of outages like the CrowdStrike incident as well as the Ministry of Education's Mobile Guardian cyberattack? Secondly, how does the Ministry plan to communicate resilience efforts to the public to reassure them of the reliability of Government services?

Mrs Josephine Teo: Sir, I think there is no doubt in all of the Members' minds that when an incident of such a nature happens, inevitably there will be questions about digital resilience. There will be questions about why we should continue to engage digitally for the most important transactions in our day-to-day lives and people are naturally concerned about the reliability. So, we fully understand that.

I think there is no shortcut to achieving public confidence. We need to be able to put the systems in place. We need to also demonstrate that when disruptions occur, and they inevitably will occur, we are able to recover very quickly. It is not different from how our residents will feedback to us about their confidence regarding lift systems. For example, the lifts will, from time to time, break down. It happens in every constituency and the residents will very naturally also tell you that, "I can accept that, as long as you are able to recover it within a short time". There is a difference between the lift system being out of service for two weeks, versus two days, versus two hours. And that is also the approach that we must take.

And there is no amount of assurance that you can provide, except by demonstrating that this is indeed possible, which is why our emphasis has to be on the ability to respond to incidents. It also has to be a whole-of-system approach. It cannot be just the Government being able to do this.

As I explained earlier in my reply, in this case, Government services and essential services were largely unaffected. This is not to say that we are secretly happy that nothing happened so badly and that the hit was taken by others. That is not the approach at all.

Citizens do not differentiate how the disruption occurred. It occurred. And whether you are at the car park gantry not able to get out or whether you are at the check-in counter not able to get your boarding pass, you feel just as annoyed or you feel that something has failed you. We understand all of that.

So, it requires all of the actors, all of the stakeholders in the system, to be able to work with a single-minded focus on ensuring resilience of their systems.

Mr Speaker: Ms Jessica Tan, please keep the question short.

Ms Jessica Tan Soon Neo (East Coast): Speaker, I thank the Minister for covering the PQs quite comprehensively. Related to the PQ I had filed, can I just ask, because the Minister also made the important point that you cannot predict all disruptions and disruptions will happen. And because it is so complex – as part of my PQ, I had asked this as well – is there a requirement for organisations, not just the Government, to test their response and resilience plans more regularly? Because you can have the plans in place, but if they are not tested, you do not know whether the systems will fail and if they fail, it is also the response plans because the response plans are both operational as well as digital.

Mrs Josephine Teo: Mr Speaker, the response to the Member's question is very similar to the response to the first set of questions that were posed in supplementary. It is actually in the companies' and systems owners' own interests to assure their stakeholders that they regularly test their systems and their systems have resilience.

Where appropriate, we would, of course, put out what are good practices – and the Supply Chain Programme that we introduced together with the toolkit is one example. Mandatory requirements, if and when they have proven to be essential, foundational to all services, we are not averse to putting them in place. But we are still very mindful that there is such a great diversity in the systems and digital products and services that are being delivered to citizens, that a one-size-fits-all set of requirements may not really do the job but may, in fact, add to resources being diverted to meeting these compliance requirements without achieving the necessary resilience and usability that system owners should be striving for.

Mr Speaker: Last supplementary question, Dr Tan. Keep it short, please.

Dr Tan Wu Meng (Jurong): Mr Speaker, I thank the Minister for her answer. I start by declaring that I have Clementi residents who were affected by the Mobile Guardian outage and that I have also filed a PQ yesterday on Mobile Guardian. But this question is more generally to the Ministry of Digital Development and Information. Can I ask, in the approach to stress testing and vulnerability testing of gov.sg systems, does the Ministry envisage applying the same level of testing to key vendor systems outside gov.sg? This is because even though we can outsource services, outsource authority, but ultimately, as the key stakeholder, we cannot outsource responsibility.

Mrs Josephine Teo: Mr Speaker, I think the Member's question is very specific to gov.sg, nothing to do with CrowdStrike, actually. But briefly, nobody is outsourcing responsibility for gov.sg. It is entirely the Government's responsibility. Gov.sg is a sender ID that is also protected by the SMS sender ID registry that we set up some time ago. The whole reason for requiring all Government communications with citizens, with the public, on SMSes to go through gov.sg is so that we can secure this channel more robustly.

And I can share very briefly that extensive testing was implemented before the roll-out and, indeed, the roll-out was also first to a smaller group and then to a bigger group. So, I hope that addresses the Member's question.