Prevention of Bank Data Centre Failures and Outages

3 Mr Yip Hon Weng asked the Prime Minister regarding the recent service outage by banks caused by data centre failures (a) whether MAS will have more oversight over the banks and mandate that there should be built-in redundancy systems to prevent outages at data centres; and (b) what are MAS’ plans to ensure similar situations do not occur in the future.

Mr Lawrence Wong (for the Prime Minister): Let me start with the causes and impact of the disruption on 14 October 2023. DBS and Citibank experienced system outages in the mid-afternoon of 14 October 2023 which affected their banking and payment services. These outages were caused by a malfunction of the cooling system in the data centre hosting both DBS’ and Citibank’s IT systems. These IT systems support the delivery of retail and corporate banking services. The temperature in the data centre rose above the optimal operating range, causing the banks’ IT systems to shut down.

To restore the impacted services, DBS and Citibank immediately activated their IT disaster recovery and business continuity plans. However, both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective back-up data centres: DBS due to a network misconfiguration and Citibank due to connectivity issues. Services at DBS and Citibank were progressively recovered from 8.21 pm and 7.05 pm respectively on 14 October, but only fully recovered in the early hours of 15 October.

The impact of the service outage was wide. Up to 810,000 attempts to access the digital banking platforms of both banks were estimated to have failed between 2.54 pm on 14 Oct 2023 and 4.47 am on 15 October 2023. Approximately 2.5 million payment and ATM transactions could not be completed. DBS reopened its branches from 5.30 pm to 9.30 pm on 14 October to assist affected customers. Both banks provided updates via social media platforms.

Let me now address the Monetary Authority of Singapore (MAS)’s requirements on banks’ business continuity, IT infrastructure resilience and their outsourced services involving critical IT systems. MAS requires banks to establish IT disaster recovery plans and test them regularly. Banks must conduct disaster recovery exercises with their back-up data centres to validate that critical systems and services can be restored within four hours of an outage. The unscheduled downtime for a critical system affecting a bank’s operations or service to customers must not exceed four hours within any 12-month period.

MAS does not oversee banks’ external service providers, which are typically not financial institutions. This is similar to the approach taken by regulators in major jurisdictions. The onus is on the banks to ensure that the external service providers they appoint to support their operations or service to customers can meet MAS’ requirements on operational resilience. MAS also requires banks to maintain close oversight of external service providers, so that they can deliver services with minimal disruptions.

DBS and Citibank have fallen short of MAS’ requirements to ensure that their critical IT systems are resilient against prolonged disruptions. While both banks conducted annual exercises to test the recovery of their IT systems at the back-up data centres, the specific issues that led to the delays in system recovery on 14 October did not surface during those tests.

I will now elaborate on the accountability and remediation measures taken to uphold the reliability and recoverability of banking services.

First, holding banks accountable. Under the Banking Act, MAS can impose a fine of up to $100,000 on financial institutions found in breach of MAS’ requirements on technology risk management. With the passing of the Financial Services and Markets Act in 2022, which will progressively come into force next year, this fine quantum will be increased to a maximum of $1 million. While the fine quantum is relatively lower compared to those imposed by financial regulators in countries, such as the UK, it is consistent with existing local penalty regimes, such as those under the Telecommunications Act and the Personal Data Protection Act.

Besides fines, MAS uses a range of regulatory tools to address lapses in banks’ risk management. This includes additional capital requirements and suspension of specified businesses or activities. In May 2023, in response to repeated outages, MAS imposed a multiplier of 1.8 times to DBS’ risk-weighted assets for operational risk. This translated to approximately S$1.6 billion in total additional regulatory capital at the time. Holding additional regulatory capital comes with costs for the bank. It increases cost of capital, a key metric that drives business decisions, such as dividends and investments. It is a drag on the return on capital which could, in turn, impact credit ratings and stock price of the bank.

Banks are also accountable to their customers, but matters of compensation are better dealt with between the bank and its customers as it would be highly dependent on individual circumstances. MAS expects banks to have a fair process to deal with this.

Second, remediation. MAS has instructed both DBS and Citibank to conduct thorough investigations into the root causes of the incidents that occurred on 14 October, put in place remediation measures to minimise future outages and strengthen their recoverability in the event of an outage. In addition, they are required to provide to MAS regular system availability reports relating to their critical systems. MAS will also work with the financial industry to incorporate key learnings from these incidents into all banks’ risk management controls, MAS’ future technology risk supervisory approach and the next financial sector business continuity exercise scheduled for 2024.

MAS has adopted a tougher stance against DBS because it experienced five disruptions to its banking services in the last eight months. This is unacceptable. As directed by MAS, DBS convened a Special Board Committee earlier this year to oversee a full review of its IT resilience by an independent external expert. The review has been completed and DBS has set out a technology resiliency roadmap to address the findings and improve system resilience.

To ensure that DBS keeps a sharp focus on restoring the resilience of its digital banking services, MAS has prohibited DBS from making any non-essential IT changes or acquiring any new business ventures for a six-month period. There must not be distractions that take away the needed resources and attention by the bank to strengthen its technology risk management systems and controls. MAS has also barred DBS from reducing the size of its branch and ATM networks in Singapore until MAS is satisfied with the progress of DBS’ remediation.

Another dimension of remediation has to do with data centres, which host the IT systems of not just the banks but also other critical sectors. The Government is studying how best to further strengthen the security and resilience of data centres where lapses could result in a significant impact.

Finally, contingency measures in the face of banking disruptions. No IT system is infallible. Disruptions can occur for a variety of reasons and can happen without warning. When they do occur, MAS expects banks to take prompt steps to reduce inconvenience and costs to customers. This includes being proactive and transparent in updating affected customers on the status of service recovery and alternative services.

While our banking system is generally robust, customers, too, must plan and prepare for contingencies. They can benefit from having alternative payment options and not be over-reliant on one provider for time-sensitive transactions. Indeed, during this recent service disruption, customers who were able to switch to alternative payment providers or use cash as a last resort would have been less affected.

The digitalisation of financial services has brought significant convenience to the public. While some disruption from time to time is unavoidable, we expect financial institutions to build capabilities to safely recover from any disruption within a reasonable time period. Where financial institutions fail to do so, as with this incident, MAS will work with them to thoroughly investigate the incident, apply lessons learnt in our supervisory oversight of the financial industry and take necessary action to further strengthen the resilience of financial service delivery.