Lessons Learned From Hurricane Katrina: Preparing Your Institution for a Catastrophic Event
The Federal Financial Institutions Examination Council (FFIEC) member agencies (regulatory agencies)1 and the Conference of State Bank Supervisors are relaying comments made by financial institutions regarding lessons they learned from the effects of Hurricane Katrina. Financial institutions have responded admirably to the unique challenges raised by successive hurricane seasons with significant storms. Major challenges faced by these institutions included the following:
- Communications outages made it difficult to locate missing personnel.
- Access to and reliable transportation into restricted areas were not always available.
- Lack of electrical power or fuel for generators rendered computer systems inoperable.
- Multiple facilities were destroyed outright or sustained significant damage.
- Some branches and ATMs were underwater for weeks.
- Mail service was interrupted for months in some areas.
Business continuity plans generally worked very well in enabling institutions to meet these challenges and to restore operations swiftly. However, the unprecedented magnitude and duration of the effects of Hurricane Katrina caused major disruptions that exceeded the scope of the disaster recovery and business continuity plans of some financial institutions. Many institutions had to adjust plans and improvise responses to successfully address unexpected complications. For example, institutions adapted procedures to facilitate cashing checks for non-customers. Overall, institutions prevailed in very difficult circumstances through advance planning and preparation, and by working together. As a result of these efforts, the financial industry was able to assist customers and communities in their time of greatest need. Certain financial institutions affected by Hurricane Katrina and its aftermath have relayed the following experiences or lessons learned that your institution may find helpful in considering its readiness for responding to a catastrophic event. You may want to consider this information when conducting a review of your institution's disaster recovery and business continuity plans. These lessons learned should not be construed as new regulatory requirements, nor do they supplant or modify the guidance provided by the FFIEC in its Business Continuity Planning (BCP) Booklet.2
Lesson Learned - Some organizations may not have anticipated or prepared for the extensive destruction and prolonged recovery period resulting from Hurricane Katrina.
Are we prepared?
A disaster like Hurricane Katrina, although infrequent, may require financial institutions to implement their disaster recovery plans and to improvise creative solutions to address unforeseen difficulties quickly. You may want to reassess how well your institution is prepared for reasonably foreseeable threats across all levels of the organization, not just from the perspective of recovering your information technology.
How much planning/preparing is enough?
You cannot prevent or anticipate all disasters, so you should prepare and practice for them. Knowing where to go and what critical functions need to be restored can provide confidence to you and your employees when responding to a disaster. Identifying potential threats, assessing their potential impact, assigning priorities, and developing planned responses are the basic principles of sound business continuity planning. Such reviews often categorize threats on a scale from high to low, according to both their probability of occurring and the impact each could have on the institution.
The impact rather than the source of the threat should guide the development of disaster recovery and business continuity plans. For example, a threat that presents a low probability of occurring and a low impact may not warrant further review. However, every threat that could pose a high adverse impact generally warrants further consideration regardless of its probability of occurrence.
You should implement reasonable safeguards to mitigate the range of risks that realistically may confront your institution. Developing, implementing, and regularly testing disaster recovery and business continuity plans to ensure their continued effectiveness for responding to changing business and operational needs takes time, resources, and money. You should consider how to strike a balance between addressing the threats your institution faces with cost-effective measures to mitigate those risks and recognizing areas where it may be either cost-prohibitive or impossible to alleviate your institution's exposure.
Lesson Learned - To be realistic, disaster drills should include all critical functions and areas.
How thorough should disaster drills be?
Disaster drills should be relevant to a specific location (considering infrastructure, population centers, weather, threats of terrorism, natural disasters, etc.) and include worst-case scenarios. You may want to reconsider the frequency and scope of future testing strategies to incorporate more thorough functional and full-scale tests of all support operations, business lines, and geographies.
These periodic tests are most effective when they simulate realistic disasters and require the processing of a sufficient volume of all types of transactions to ensure adequate capacity and capability at all recovery sites. The tests should also consider all critical functions and applications, use only off-site data and supplies, and include some level of improvisation to meet unexpected events.
For example, you may want employees to practice using manual back-up procedures (e.g., debit and credit tickets) to process transactions until electronic systems are restored. Or, a disaster drill could simulate situations that involve the restoration of damaged loan files or documents, and how to protect employees from potentially harmful exposure to contaminated bank records, cash, or contents in safe deposit boxes.
How should we assess disaster drills?
Performance assessments after each disaster test help ensure that each simulation improves the institution's ability to recover from a catastrophic event. After conducting a drill, you should review the results to determine what worked correctly, what went wrong or not as expected, what areas can be improved, and what, if any, adjustments to your plans are needed.
Who should participate in disaster drills?
Your organization's successful recovery can hinge on the efforts of key personnel, and those key personnel may change. As a result, you should promote a "we're in this together" attitude and recognize that all employees can contribute to an institution's disaster recovery and business continuity efforts. Employees at every level of your organization should know their role in the disaster recovery and business continuity plans.
Lesson Learned - Anticipate disruptions in communications services, possibly for extended periods of time.
How can we communicate?
Hurricane Katrina illustrated that a widespread disaster can strand employees without access to working land-line or cellular telephone services. You may want to develop, test, and update a contact list for senior management, employees, customers, vendors, and key government agencies. Maintaining copies of this information at all sites, plus one or more off-site locations, can be very helpful in the event of a disaster.
You also may want to develop alternate ways for locating and communicating with employees and customers. Less-traditional communication methods might include two-way radios, cellular telephones with out-of-state area codes and/or text messaging capability, satellite telephones, or personal data assistant (PDAs). Employees could use these less-traditional communication methods to report their location and obtain current information. In addition, you may want to establish a central point of contact outside the potential disaster area and make pre-established toll free telephone numbers available for employees and customers.
What about the mail?
A widespread disaster can disrupt the U.S. Postal Service for an extended period. During Hurricane Katrina, customers with automatic deposit and bill payment services experienced less difficulty in maintaining their accounts. You may want to encourage or assist your customers in establishing direct deposit account relationships or automatic bill paying services to mitigate disruptions in their finances.
Lesson Learned - Critical staff may not be able to reach their assigned recovery location.
Where is everybody?
Your disaster recovery and business continuity plans should not assume that all key personnel will be available at designated sites to assist in recovery efforts. Evacuation orders, safety and health hazards, or damaged infrastructure (e.g., washed-out roads, collapsed bridges, and downed power lines) may prevent employees from timely reporting to assigned locations, despite their best efforts.
You may want to identify alternative, prioritized gathering place(s) for employees to meet after a disaster. Similarly, you may want to develop multiple, alternate, prioritized contact arrangements for employees to follow if they are unable to reach their assigned location given the likelihood of simultaneous communications disruptions. In addition, you may want to consider what type(s) of credentials employees will need to gain access into a disaster area, as authorities may restrict re-entry.
What alternate transportation methods could be considered?
In the aftermath of Hurricane Katrina, many financial institutions had employees scattered across the region with limited access or means to reach the institutions' facilities. To address this, some institutions arranged alternate transportation methods, e.g., carpools, bus services, and air connections. Some institutions also developed plans to shift and transport employees either from or into affected areas.
Lesson Learned - People are essential to the recovery of operations.
What about my family?
Employees' foremost priority will be the safety and welfare of themselves and their families. You may want to have discussions in advance with employees regarding their personal plans in the event of a disaster. You may also want to tell them what steps will be taken to provide for employees and their families who might need to stay in a disaster area or at a back-up facility.
Is everyone okay?
A widespread disaster can overwhelm medical services. Besides keeping basic first aid supplies stocked and easily accessible, you may want to make preparations for employees who have special needs. Catastrophic events not only cause physical injuries, they also create very stressful situations. Your employees may feel considerable stress after a disaster for an extended time.
What basic necessities will people need?
Damaged infrastructure, disrupted support services, and a prolonged disaster recovery period can make it extremely difficult for employees to obtain basic necessities. Some institutions reported that they have developed short-term and long-term plans for meeting essential human necessities to encourage employees to remain in the area(s) where the institution is operating and so that employees can focus on resuming financial operations. These plans addressed supplies and services such as:
- Food, drinking water, and safe lodging
- Vital supplies such as medicine, clothing, etc.
- Child care, especially if schools are closed
Lesson Learned - Replacement supplies may be difficult to obtain during a protracted recovery period.
How do we obtain more supplies?
A widespread disaster can severely disrupt normal support services and cause a prolonged recovery period. Most institutions' disaster recovery and business continuity plans provide sufficient supplies at the primary operations center and the back-up site to permit several days of operation. However, obtaining replacement supplies as initial stocks are exhausted can be difficult as stores may not be open, and new shipments may be delayed due to transportation delays or damaged infrastructure.
Some institutions reported that they instituted long-term arrangements to replenish basic supplies such as business forms and fuel over an extended period, although this process can encounter unexpected obstacles during an emergency. For example, some institutions contracted to have replacement fuel and other supplies delivered as existing stocks were depleted. However, military personnel, law enforcement officers, or rescue workers had priority in some cases for these supplies, especially fuel. Consequently, you may want to consider this possibility in your planning. With respect to replenishment of routinely used forms, some institutions maintained a master set of routinely used forms at an alternate but easily accessible site.
Employee safety is of paramount importance and should carefully be considered in deciding whether to attempt temporary repairs. However, some institutions found it useful to maintain some basic supplies such as tarps, plywood, tools, etc. to board up broken windows, prevent water leakage from exposed roofs, etc. Demand for these materials will surge and may be in short supply following a widespread disaster.
Lesson Learned - Financial institutions' facilities could be damaged or destroyed, creating a need for alternate facilities.
If our facilities are not safe, what alternate facilities could we use?
Facilities should be safe prior to allowing personnel to re-enter the premises. A professional inspection may be necessary or advisable as some types of structural problems are difficult to detect. An inspection of your sites may determine that the damage to these premises is so severe that it is not safe to resume business operations at those locations.
Your risk assessments and planning should contemplate that your facilities may not be available, and that alternate facilities arrangements may become necessary. Some common substitute accommodations arranged by institutions in the aftermath of Hurricane Katrina included renting undamaged buildings or leasing mobile units. Also, a number of financial institutions entered into "partner institution" or "buddy bank" agreements. These included organizations opening shared facilities and unaffiliated institutions granting affected institutions access to teller stations. Other institutions executed reciprocal agreements where IT systems were shared. Having these types of agreements in place prior to a disaster could significantly improve your institution's ability to resume operations more expeditiously and efficiently after a catastrophic event.
Some disasters can affect a large geographical area. Technological advances in warning systems enabled financial institution managers to activate disaster recovery and business continuity plans 72 to 96 hours prior to Hurricane Katrina making landfall. Before deciding which alternative to pursue, most institutions monitored and/or tracked the predicted path of any adverse conditions; thereby enabling personnel to select a location less likely to be affected by the potential disaster.
What procedures do we follow to establish temporary facilities?
You may want to determine in advance what types of building inspections and permits are required for temporary facilities and to maintain contact information for the governmental authorities that have jurisdiction over these matters. Federal and state bank regulatory agencies expedited or waived many application procedures for establishing a temporary facility after Hurricane Katrina.
Lesson Learned - The location of any back-up site can be critical to successful recovery efforts.
Where should the back-up site be located?
In the aftermath of Hurricane Katrina, data recovery efforts for some financial institutions were hampered by limited access to back-up sites that were in close proximity to the primary location. Institutions with back-up sites reported that they found them most useful when they were located sufficiently far away so as not to be affected by the same infrastructure and other risk elements as the primary operations center.
If you have a back-up site, you may want to reassess its location and the probability that it may be affected by the same risks that threaten your primary locations. In addition, you may want to provide your primary regulator the names, alternate telephone numbers, and addresses of personnel to contact if evacuation and/or disaster recovery plans have been activated.
Do the recovery facilities have sufficient capacity?
The number of institutions affected by Hurricane Katrina created unexpected demands on some servicers' back-up sites. You will want to ensure that your back-up facility has adequate capacity to process transactions in a timely manner.
In assessing this capacity, you may want to consider not only the needs of your customers in an affected area, but also the demands that other affected institutions may place on a given back-up site or servicer. You may want to reassess processing capabilities and joint testing of your recovery plans with your servicer.
How do we assure that we will have electrical power?
Many financial institutions' primary and back-up facilities lost power in the aftermath of Hurricane Katrina because the power transmission grid was not operational. It is not uncommon for all of a financial institution's facilities to be on the same power transmission grid. Therefore, you may want to check with your local power company to determine how it supplies electricity to your primary operations center and your back-up site(s). If the same source supplies electricity to both sites, you may want to consider an alternate location or explore the feasibility of installing an independent power supply at one of the facilities.
What about back-up power sources?
Many institutions affected by Hurricane Katrina used portable generators powered by gasoline or propane as a primary back-up power source. Some institutions pre-wired generators for their most important equipment. Depending on their capacity, these machines usually can provide power for critical operations, but typically should not be used to meet all electrical needs.
You also may want to consider appropriate locations for operating a generator and for storing fuel. Fuel storage containers and generators can leak, and generators may produce deadly carbon monoxide gas and can be subjected to the same damage that the site experiences.
Lesson Learned - Processing transactions may be extremely difficult.
How can we overcome difficulties in processing transactions electronically?
The widespread power and telecommunications outages after Hurricane Katrina hindered electronic transaction processing. Most institutions had multiple types of back-up and timely back up of data, which assisted in recovery of applications and business resumption. In some cases, however, manual processing was required. While this may be a short-term solution, connectivity with the data processing facility is critical in order to restore and sustain routine financial services. If telecommunications cannot be recovered, transaction items must be physically transported to other processing sites.
Lesson Learned - Be prepared to operate in a "cash only" environment.
Why would we need more cash?
Power and telecommunications outages can disrupt all electronic forms of payments, such as debit and credit card payments. Customers and employees remaining in, or evacuating from, affected areas may need unexpectedly large amounts of cash to pay for critical goods and services. In anticipation of hurricanes or other disasters with advance warning, some financial institutions developed plans for ordering larger shipments of cash prior to the expected onset. These institutions also reported the need to plan for enhanced security precautions.
What if the vault and/or ATMs are damaged?
Damaged vaults and ATMs were significant concerns for some institutions affected by Hurricane Katrina. Currency can be damaged or ruined by water or pollutants. You may want to keep vault cash in clear, waterproof bags to minimize the possibility of contamination from standing water.
Lesson Learned - The financial industry is dependent on numerous critical infrastructure sectors that potentially have competing interests.
What is our level of priority for overall disaster relief efforts?
While the financial system is recognized as a part of the critical infrastructure, 3 financial institutions have to compete with the restoration of other critical components during recovery efforts. Some financial institutions have joined regional coalitions to facilitate critical infrastructure planning efforts. By anticipating and addressing such issues in advance, you can better prepare your staff to overcome unexpected obstacles.
For example, obtaining additional cash (a critical commodity in an affected area) can hinge on whether telecommunications and electrical services have restored power and processing capability to institutions or ATMs, the transportation authorities have reopened traffic routes, and the petroleum industry has provided fuel so armored couriers can enter and leave disaster zones.
You may want to contact local and state officials to understand the priority that will be given to financial institutions to restore critical services. You can reach your state homeland security contact at http://www.dhs.gov/xgovt/editorial_0291.shtm.
Lesson Learned - A financial institution's involvement in neighborhood, city, state, federal, and non-profit or volunteer programs can facilitate a community's recovery from a catastrophic event.
How can we work with other programs?
The Department of Homeland Security recognized that non-governmental organizations, such as non-profit, volunteer, and private sector entities, play a fundamental role in response and recovery efforts. These organizations can contribute in ways that are, in many cases, key to a community's successful recovery after a catastrophic event.4 You may want to contact local chapters of these entities to discuss ways the organizations might work together to benefit the community.
What can regulatory agencies do to assist us?
During the past hurricane season, the regulatory agencies communicated with the industry and the public through a variety of media, including television and radio broadcasts, websites, and national call centers. You may want to maintain a list of regulatory points of contact and reference data to establish clear lines of communication between your institution and primary regulator. A current list of some important regulatory telephone numbers and website addresses is included below.
- Emergency Agencies
- Federal Regulatory Agencies
- State Regulatory Agencies
- Emergency Contact Information
1 The Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, National Credit Administration, Office of the Comptroller of the Currency, and Office of Thrift Supervision.
2 For guidance on business continuity see the FFIEC IT Examination Handbook, Business Continuity Planning (BCP) Booklet - PDF 2.50mb (PDF Help) (www.ffiec.gov)
3 Refer to the USA Patriot Act of 2001 - PDF and the Critical Infrastructure Protection Act of 2001 - PDF. (PDF Help)
4 Department of Homeland Security, The Federal Response to Hurricane Katrina: Lessons Learned, February 2006. (www.whitehouse.gov)