“That was absolutely a world record”. PrivatBank COO Mariusz Kaczmarek — on the bank’s migration to the cloud, wartime transformations, and the value of the team

The migration of data centers of Ukraine’s major state bank, PrivatBank, to the cloud became one of the remarkable recent events in the banking and IT communities — given the unprecedented wartime conditions and the operation’s impressive speed itself. 500 people, 45 days, and 50 hours stand behind the heavyweight’s success.

In a recent interview with DOU, Mariusz Kaczmarek, Chief Operating Officer at PrivatBank provided highlights on the migration process and the bank’s overall current stance, and here are the key points.

Migration in wartime was a total record — in Poland it could have taken ages

The banking sector is a regulated market. We need to ensure business continuity in all our operations, which means that the bank’s systems and applications must be duplicated somewhere else from its data centers in case of physical destruction of one of them.

Privat Bank had two physical data centers in Ukraine. Various scenarios of potential Russian full-fledged invasion were discussed, but we never expected that it would be a real war and attack from all directions. That was the worst-case scenario we were having in mind, but we never planned that.

Moreover, before the war started, the regulations were not allowing the banks and other financial institutions to process the data in the cloud outside Ukraine, either in the private cloud or a physical data center abroad. When Ukraine was attacked, the first question was about laws and how to protect the business operations.

With the Privat’s size, infrastructure, number of clients, and services, it’s physically impossible to run the bank on paper. Privat serves 20 plus million clients, owns 65% of ATMs, 70% of POSs, and 50% of payments between the Ukrainian institutions are going through the bank. Obviously, switching off and destroying Privat would generate huge problems for the business and the country’s continuity.

In such an emergency, the National Bank of Ukraine worked quite fast — first, by allowing banks and other financial institutions to store protected backups of physical systems in the data centers in the cloud, and then, on March 8, proceeding to the permission to operate from cloud services and physical data centers if needed.

This regulation will work until the end of martial law and for the next two years beyond. At that moment, we decided with the supervisory board to launch the project of not only having the backup data in the cloud but physically migrating our services from the on-premise data centers into the cloud.

The process was absolutely quick. I’m Polish and I’m operating in the IT and banking markets for many years. In the EU country, it took us ages, not years, ages to agree that we could do the processing of the data in the cloud within the European Union. Under those circumstances, Ukraine was absolutely speedy. That was absolutely a world record from the decision-making perspective as well, and I’m very thankful to the authorities for that.

The process from the inside: picking up systems to be migrated and last-minute decisions

The bank officially says that we have something like 500 different systems, which we have to host on physical and virtual servers. Also, you need production, testing, and development systems testing systems, databases behind all this, and the entire telecommunication infrastructure. We have 15 million online users of Privat24, and 20 million clients using our services simultaneously.

Not every institution would be ready to survive that. And this is actually something I’m proud of being in Privat. The bank is one of the best technical organizations I have met and worked for because it is working 24/7. In any Western organization, I used to work for, we’re not able to deliver the services 24/7.

Out of 500 applications, those which do have a meaning and are important for the bank, — we are talking about the 330. That’s everything the clients are using: the Privat24, card payments, authorization systems, all the servicing functions, contact centers infrastructure, ATM connections, and POS terminals.

We assume that when physical infrastructure would be destroyed, we must be fully operational from the cloud. For the Visa, and Mastercard connections, as well as the SWIFT, we took the decision to rent a little physical infrastructure abroad in the data center. The bank is fully resilient now by operating through the cloud.

If you take the standard approach to the cloud migration without the war, without any threat, the process of proper decision-making, picking up the right cloud solution, getting all the approvals and all the financials would take you six to eight months without even getting to migration itself. In the best-case scenario, by doing it by the book, you won’t manage to finish migration in less than two years. It would be even closer to three years.

You are going through the formal process, and then you are picking up the critical applications to migrate, and then you want to make sure that this application is properly tuned, configured, tested, and switched on. We didn’t have this comfort here. But we did it in 45 days, and out of that, we needed less than 50 hours.

5% of the time during the nights we had to switch off the on-premise solutions and reconfigure them in the cloud. We had two or three days when our migration was not successful — one on Saturday before Easter. Having spent half a day in the cloud, we migrated from the technology we were on to another one. Then we started operating but still saw the problem that it didn’t work. During the night, something like from 06:00 p.m. till 08:00 p.m., we migrated to concluding technology, and that also resulted in a problem.

So we had two migrations in between on Saturday when Ukrainians were impacted by the low level of services, or it was even impossible to pay. But eventually, we were successful, not even having time to test everything properly because we were actually operating under extreme conditions.

We’ll deliver services everywhere where a minimal level of safety is ensured

Infrastructure in the data center abroad is only a part of the success because the work of ATMs and branches must be guaranteed as well. ATMs must be replenished with cash every day or every second day depending on how much money is left. At the beginning of the war, we had 1400 branches and 7080 ATMs.

On the first day of the war, we decided to open 600 branches because we didn’t know what was going to happen. Nowadays, 1100 plus — nearly 1200 branches are opened and operating. Unfortunately, we lost a number of ATMs because they were destroyed by Russians who wanted to take the cash out. But we are keeping 80% of our ATMs operational.

Our crisis management team decides every day on which territories we operate in the normal regime and in which territories do not. Our top priority was to make sure that our employees are not exposed to physical threats. We don’t want to have our employees and other Ukrainians killed by accident or intentionally. So will deliver services everywhere where there is a minimal level of safety. Like soldiers who are fighting to defend the country, our bank employees are protecting the country for it to operate as normally as possible.

The team of dedicated professionals was the core of the bank’s continuity

We have close to 2500 professionals in IT, which is quite an army. Out of this number, 1000 are the people servicing ATMs, and POSs across the entire country;1500 are responsible for the development, testing, maintaining, planning, forecasting, designing, etc. And it’s close to 500 people who were working on the cloud migration.

We did not expect them to work in the normal regime, but the majority of these 500 people were operating day and night, sometimes avoiding sleep because they were so heavily working. We didn’t support this, but that was the internal belief that “I have to do everything possible to secure the money of Ukraine”. The effort of those individuals and the team is impressive, it’s an accomplishment. I’m very proud of the team, who under bombings and threats took the risk of migrating and working long hours to protect the business of Privat.

Of course, the first reaction of IT and other employees were “we have to fight against Russians”. And actually, we had a certain problem with competent and experienced professionals because we said “you do fight against Russians”. Because if you go to the army and you will be fighting physically against them, we will lose the competencies here and somebody has to help us in migration.

We spend a lot of time explaining to our IT guys and other bank employees that what we did during those times is also fighting against the enemy and our role is as important as our soldiers’ protecting us from the enemy. Out of 2500 people, below 100 went to the army. We had a list of particular employees whom we asked to reserve and the support from the state, the President’s office, the government, and the Parliament was very high.

The COVID-19 pandemic experience helped to adjust the wartime workflow

The pandemic forced institutions like ours to switch to the remote working mode whenever you are not needed to work physically servicing the clients in branches or dealing with cash.

We converted the entire back office, IT, contact centers, and a number of other important functions of the bank into the remote model of work. 85% of all my employees were working remotely, and we kept the quality. I think the pandemic helped us a lot because when the war started and people started to relocate, to work remotely due to the physical threat, the bank was operating in the normal regime.

Some of our male employees are abroad, but many of them also returned back to the country. We have a huge female population with kids in different countries, such as Poland, the Czech Republic, Germany, Croatia, and Turkey. And they all continue working.

In general, the ideal model of work for me would be — it’s up to you. But I believe that during five working days you can do it remotely. Two, you should be in the physical interaction. Or you can consolidate and be two weeks off and one week in the office. Because there are certain periods of time that people should meet to agree on the priorities, performance, meet with your friends and colleagues.

As long as we have common objectives, deliver what we would like to deliver, and there is an interaction between the people, I encourage everybody to operate the way that it’s more convenient for them and they are more efficient.

Market dynamics in Ukraine are astonishing, the best developers are here

I love Ukraine, I love Ukrainians and I like the market and the dynamics here. But that’s my personal perception. The problem which I do have as a manager, and this is the biggest surprise for me, is how flexible the market is for the employees. Today you can come to your employer and say “I’m resigning”, two weeks later you’re out, and I can’t do anything with that.

If you go to the West, if you are spending a number of years within the company, you have a three-month notice period. If you are a specialist, I can do the handover and something meaningful. And the impact of your leaving is less significant. That adds other dimensions to the market, that’s why it’s so dynamic and so great to be here.

If you are working for a software company or dedicated to niche players you have different dynamics — you are writing the code, testing it, designing, making the architecture changes, you are selling your products. It’s a different model. Whereas we here use IT to service the clients, sell the products, make them available, and convenient. That’s part of our DNA.

If something happens, whatever the date or the moment of the day, I’m informed about it, and the other priorities do not matter because we have to focus on making sure the bank is up and running. This is the kind of experience you gain working for the huge organizations which must be available for your clients.

Out of 2500 IT people, I have 5 foreigners, including me. Two of them are Polish, one is Romanian. But again, they came here not because they are better. I can tell you that 95% of my Ukrainian employees are much better skilled, and better technology advanced.

I love working with them because they are very creative, and open-minded, they are focused on the results. But what we do miss here is a kind of understanding of the bigger picture. If you are a super expert in one niche product or service, here you have to make sure that you have hundreds of those services and how to manage that.

I had a problem with hiring an IT security guy and a sales guy. Also, I needed to bring some changes to the IT organization and find a senior person with experience in working in different countries and different markets. The candidates from Ukraine we interviewed didn’t have this experience. I believe they will soon have it.

However, let’s make it clear. If I would be importing developers to Ukraine, that mean that the world is upside down. The best developers are here. It’s a shocking story for me that all the payments that you do in Ukraine are delivered within seconds. Western countries do not work that way. This is the knowledge, the ideas of our people, our Ukrainian developers, skilled engineers, and architects who developed such a system.

I was shocked when doing the SWIFT payment back home, clicking yes here with all the internal checks. 2 hours later I had the money in the account in Poland, whereas in Europe it takes up to five days. Ukraine is very advanced technologically we should be proud of the skills we have here.

Privat’s long-lasting tech-savviness as a prerequisite to ultimately successful migration

Typically, if something works well in the banking industry, you don’t change it for a long time. There are examples of a number of European banking institutions which are still using COBOL on mainframes and did not migrate their million-dollar investments from the 70s and 80s, and now they are trying to catch up. Privat, on the contrary, strives to apply up-to-date solutions, systems, and languages to keep up with the times. The majority of systems created by Privat are built with the modern stock and thanks to the knowledge of our people.

Depending on the hour of the day, we are processing between 20 to 60 clients’ transactions per minute. The complexity is obvious, and the bank is huge machinery that must process all the transactions in real-time. If something is delayed for one or 2 seconds you’re already seeing that “Privat doesn’t work”.

There’s a huge infrastructure behind it, which has to be supported by the modern stock both from the infrastructure perspective and from the software perspective. Of course, not all of our people work with the newest stock, but we’re rebuilding a couple of systems created ten years ago because the architecture is really different.

The migration to the cloud was such a success in sense of speed because the guys, the architects, and our team decided that we will be operating on x86, no mid-range, no mainframe, or anything like that. Because this is the peak of thinking of the west. The bigger the powerful machine the better. While it took many players 15-20 years to convert everything into distributed computing, Privat was from scratch built on this model. And it is constantly adapting to reality.

Enhancing the cybersecurity component

PrivatBank had historical problems, as you can recall, back in 2016 or 17, before nationalization, when we lost some cast client data, it was stolen. After nationalization in 2018, our priorities changed and we added a cybersecurity component. Our focus was on making sure that we are protected from the external world: all the firewalls, all the VPNs, all the encrypted communications, all the APIs, all the users, the tracking and monitoring, and creating the access to our systems via the other layers.

We had anti-DDoS commercial packages bought from the companies operating here. But also, luckily enough, we were protected by the cloud entry to our systems for many years.

We were impacted on 15 February by the massive hacker attack; our IP address in the network was destroyed, and we had to fix that. That’s why some ATMs did not work for one and a half hours. And then we had another attack against the cloud, but here we knew that we were able to respond easily enough. So on 15 and 16 February, the Privat was much better prepared for responding to the DDoS attacks than other state institutions. After some configuration changes coming after those two attacks, the bank became very-very resilient.

If you ask our IT guys how much time they spend focusing on cybersecurity? They started to hate me because every second meeting was no business, security privileges, rights, accesses, the network configuration, and adding new components to monitor and respond to the threat of malware, spyware, and whatever else. For instance, there was no impact on the bank from the Log4J issue in January.

We were lucky because of all those techniques and all the systems deployed. Nowadays we’re adding the security SDLC into the thinking. With all this experience grabbed this year, we have a good team and good effort put behind cybersecurity. Nobody is resilient toward the hackers’ activities, so that’s part of life.

Is the PrivatBank a good place to work and self-realization?

The beauty of Privat is that our IT guys together with business are creating new products and services. If you have great ideas and want to do something which the bank doesn’t use or could use, you have a very fast track, you can do it. Would you learn a lot within Privat those days? Absolutely, on every level, because you have many specialists who can teach you how to use the technology for business purposes and those who are managing people.

Many of our developers, focusing on particular competencies or the group of systems, are in close interactions with the business daily. They’re like working in the agile groups, it’s even close to the Scrum so that they are meeting, discussing priorities, and moving forward. And there are a number of people in the back office who have fewer interactions with the business.

But there is a space for everybody — for people who want to create and people who want to do the coding silently. We have the entire spectrum because of the size and the scale of our activities, so you will find the place to be there depending on your profile and your interest.

I like allowing people to do what they can do the best. If in this particular team you are not happy, we should be helping you to move to the other team and try something else. It’s better to do this instead of losing your experience and competence. Privat actually tries to give you the task, help you, give you the source code, explain, and teach you. But we are also giving you a lot of freedom. And that makes your work fascinating and interesting because you are the creator.

We go into many compromises in our lives. Life is too short to do so. You must have the integrity of yourself. And then, when you are happy, your friends, your family are happy, you’re bringing the new value-added to the world. Whenever we think about IT, there are people who like programming and don’t want to interact and they are happy. So why change their life if they are happy? And there are people who talk like me and sometimes they’re also happy. That’s important.

I think we have to have our vision of ourselves and make sure that we have a positive attitude to our life. Then everything is possible.

Все про українське ІТ в телеграмі — підписуйтеся на канал DOU