The HDFC Saga Shows Chinks In Legacy Banks’ Tech Armour

After months of living with coronavirus curbs, most Indians had started to celebrate their newfound trust in one sturdy digital ecosystem – Banking 2.0. The Indian government launched a #DigitalApnayen campaign on Aug 15 to encourage people, and within a month, the public sector banks (PSBs) onboarded about 1 Cr customers across all digital payment modes. But this is only the tip of the iceberg. According to a Red Seer report, mobile payments users will grow from the current 160 Mn-odd to 800 Mn by 2025, and the digital payments market will grow from INR 2,162 Lakh Cr in FY2019-20 to INR 7,092 Lakh Cr by FY2024-25.

Understandably, private-sector banks have also upped their game. HDFC Bank, the second-largest bank in the country, has been setting up Digital 2.0, a tech-driven, seamless platform for its bouquet of products and services. The initiative started under its former MD and CEO Aditya Puri. Some were surprised, given his reported (personal) aversion to technology. Others were not as Puri had fired people for being slow to adopting technology. But that was in 2015.

Cut to the present, and it will be hard to miss the irony that the bank has been temporarily barred from pursuing its next-gen digital dream due to frequent outages in its digital services. Has the bank got its technology priorities wrong?

The Reserve Bank Of India (RBI) has asked HDFC Bank to stop launching all planned digital services and issuing new credit cards until it fixes the technical glitches, which may take anywhere between three and six months. The latest outage resulted in millions of customers being locked out of their mobile and netbanking accounts due to a power failure at the bank’s primary data centre. The central bank noted that HDFC had seen several major systemic outages across its internet and mobile banking services over the past two years, which need to be fixed before new projects can be implemented. The RBI will lift the restrictions once it is satisfied that the issues have been met with compliance and accountability fixed.

Since 2016, and especially after demonetisation, the Indian government has intensified its focus on putting together a less-cash economy. From mandating companies with a turnover of more than INR 50 Cr to offer payment options via RuPay debit cards and UPI QR code to zero merchant discount rates (MDR) on digital payments, multiple measures are now in place to nudge consumers and businesses towards a digital payment system. As a result, legacy banks are also tying up with new-age fintech players and technology companies to increase the range of their digital products and strengthen their tech backbone for high transaction volumes.

But the repeated service outages faced by banking customers (HDFC Bank is not the only guilty party here) tell a different story. Banks are still getting the scale of transactions wrong, but have they tracked the resource gap yet?

Major Glitches Faced By Banks In 2019-2020

The most recent outage occurred on Nov 21, which was attributed to a failure in HDFC Bank’s primary data centre. Meanwhile, the State Bank of India (SBI), the country’s largest bank, was also scrambling to fix technical issues as its YONO app suffered an outage on Dec 3, prompting users on social media to call for similar restrictions against it. YONO is considered the SBI’s biggest punt in the digital banking world, but its popularity has suffered due to a poor app experience. 

Indian banks alone are not facing these challenges. In November this year, DBS Bank, one of the largest banks of Singapore, suffered a breakdown and saw nearly two days of downtime. In April this year, several U.S. banks suffered similar disruptions after the government issued 80 million stimulus cheques. These outages are not unusual, though, as a large number of digital products will always be linked to legacy banking systems, and a massive user base will be accessing these 24×7, say experts.

Why Do Banking Platforms Crash Despite Investments In Technology?

The biggest concern in the banking sector is to keep the core systems running without fail. In spite of regular upgrades, most of the banks have to support old and obsolete technology across their platforms. But they also need added layers of innovative technology, required for fast-evolving operations. So, the intricacies increase, leading to an extremely complex technical layer. This may not be a problem in itself as all banking platforms pass through such transitions and upgrades. However, unexpected loads on systems may cause them to malfunction.

In brief, banks need a robust technology risk management framework to ensure reliability, resilience and speedy recovery.

HDFC Bank is a case in point. The bank has repeatedly blamed the past glitches on its inability to anticipate transaction volume, but the issue lies elsewhere. Take, for instance, the Nov 21 outage. The bank faced problems when it tried to switch to a backup data centre after its primary data centre went down. As most of the banks have their headquarters in Mumbai, their primary data centres are also located there while the secondary ones are usually based in Bengaluru or Hyderabad. But technology providers note the increasing necessity to have a network of smaller data centres in the same region, to offset unanticipated transaction volumes.

“The relevance and importance of smaller ‘edge’ data centres, which serve client-facing applications, have become crucial. These outages could have been avoided by spreading the applications across two-three data centre sites in a particular city. Moreover, these applications should only be using ‘edge’ data centres of different providers for distributed application operations and delivery. In this way, they will function in sync and points of failure will be reduced as well,” says Nikhil Rathi, founder and CEO of Web Werks India, a Tier 3 data centre service provider.

Edge computing is especially helpful when there is a need for distributed computing, localised data storage and fast response time.

Incidentally, load distribution should be thoroughly tested, ensuring zero downtime for applications which require constant availability even when a particular point of presence – such as a data centre – is impacted. These decisions are subject to how far regulatory frameworks and a bank’s internal policies allow processes to be moved off its own premises.

According to Rohit Sen, cofounder and CEO of NIRA, a fintech lender, high transaction loads can easily lead to system failures. “Maybe, the RBI will have to ask banking systems to undergo some stress testing to gauge their preparedness. This incident could be an early warning, and the central bank should consider this as a systemic risk rather than a particular bank’s problem.”

Load balancing is a critical issue, given that the volume and value of transactions managed by banks have spiked rapidly over the past year. A load test, by definition, measures the performance of a system under an expected load. In contrast, a stress test overloads a system to find the breaking point.

Are Banks Getting Their Talent Mix Right?

Besides spending on technology, there is also the question of who is in charge of running the core banking systems. More often than not, banks are wary of lateral hiring. This scenario is changing, though, as banks are rapidly opening up to fintech partnerships and also looking to source suitable candidates to fill niche positions. SBI has already hired people from tech startups such as AskmeBazaar, GoBazaar, Ola and Swiggy, as well as from KPMG, Infosys, TCS and Deloitte, for the tech team that developed Yono.

It is no secret, however, that IT teams are relegated to little importance when it comes to core banking operations. PSU banks hire their tech teams through probationary officer exams and train them in-house, but the heavy IT planning is hived off to partners and IT service providers, who do not have much at stake in the grand scheme of things. Private-sector banks seek out more specialised skill sets, even from tech and startup ecosystems, but these are rarely leadership roles.

Ashneer Grover, cofounder and CEO of BharatPe, a merchant payments company, says that the absence of chief technology officers (CTOs) as lead playmakers in the banks’ tech future amply indicates how the ecosystem treats the subject. “How many Indian banks have CTOs reporting to the board? It is not yet a defined position in the banking ecosystem and this is where investments are missed.”

According to a 2017 RBI notification, a CTO should be an engineering graduate with 15 years of experience in the banking sector across IT policy and planning, financial networks and applications, financial information systems, cybersecurity technologies and payment technologies.

Speaking on the recent happenings, Grover says that banks are not reinvesting their cost savings from digital transactions to strengthen their core infrastructure, and that is where the gap lies. But it could soon be mandated by the central bank to prevent service outage.

The Way Ahead

What will be the way ahead for the likes of HDFC Bank?

In 2019, a Gartner research report found that India’s banking and securities sector was expected to continue to invest in their digital businesses, with spends estimated to grow to $11 billion in 2020. Experts also suggest that major banks, especially the industry veterans, will always be in a state of flux and may have a few legacy mechanisms in various stages of upgrades. It is done periodically, and all banks, even those with a hefty IT infrastructure budget, have to undergo this process. Any weak link in the chain or any crucial external factor is bound to result in technical glitches.

Currently, banks with heavy investments in analytics make their own estimates and forecasts on transaction growth rates. But even within such forecasts, one can only provision for the most probable situations and not every possible scenario. “These are established banks which have invested in technology and are still facing unprecedented demand. Banks need to invest in early warning systems so that the system architecture grows in sync with the volume of transactions and then ensure that those systems are monitored effectively,” says D.D. Mishra, research director at Gartner.

Senior banking consultants also suggest that the central bank should be involved in this estimation and monitor what the banks are provisioning for their anticipated load. Using predictive analytics, banks can also pre-determine events which are likely to increase digital transaction volumes and dedicate more computing resources to manage the load. This could also be a good opportunity for startups in artificial intelligence and data analytics space to contribute to the legacy banking ecosystem.

Note: We at Inc42 take our ethics very seriously. More information about it can be found here.