A binder collecting dust is not enough to protect the company from total collapse when a catastrophic event occurs.
Many organizations mistake risk assessment as the same thing as having the ability to recover from disruptions.
They put together high-level documents outlining general risks and responsibilities but assume that they are covered if there is an interruption in business.
When a ransomware attack occurs, locking down their Active Directory, or a major supplier goes bankrupt overnight; they realize quickly that their global plans are not based on real circumstances.
Operationally, there is only one way to ensure that your business is prepared for disruption.
True continuity is not a static document, but rather an operational process for dealing with disruption.
Based on our analysis of continuity failures within Large Enterprises, we find a common risk source between a large percentage of continuity failure plans.
The weakest plans rely on very broad advice without the implementation details and contingencies needed to turn the plan into action.
The strongest plans center around trade-offs in decision-making, bottlenecks to continuity operations, and what occurs during the first hour of a crisis.
Below you find how modern businesses are able to keep things running during times of complete disruption.
Modern day continuity planning
True Business Continuity is focused on recovery limits based on real world and practical limits on recovery as opposed to arbitrary IT timing based limits.
To create a plan for continuity you will need to start with a harsh Business Impact Analysis (BIA) that establishes a ranking of critical processes based on the maximum time that will be tolerated for downtime, not just arbitrary IT timing goals.
Planners should establish a strict Recovery Time Objective (RTO) as well as Recovery Point Objective (RPO), and understand the large-scale consequences of establishing aggressive RTO/RPO goals.
Testing of the plan should not be just polite quarterly tabletop exercises, but a real simulation of chaos to validate both Technology Failover and Human Communication processes/workflows.
The same level of attention as what you would give to cloud backups and server redundancy must be given to non-IT continuity mechanisms (alternate suppliers, manual work-arounds, etc.) as they relate to the availability of alternate physical locations.
Why do standard business continuity strategies fail?
The basic design of corporate planning is wrong.
The process of assessing your risk, building a plan, creating roles and responsibilities, and testing is a loop that gives the false impression that you have managed your risk.
In reality, the standard framework fails, because it lacks a branch structure.
It makes the assumptions that you will be able to communicate during the crisis, the people in the roles will be available, and there will be a rapid recovery back to your business being normal.
Natural disasters do not follow optimal conditions.
If you need approval to tell your IT department to take down a compromised network, what happens if there is a delay?
If your contact list is outdated, or your recovery workflow relies solely on a single database administrator, what happens if that database administrator is not reachable?
Standard templates rarely present bottlenecks in workflows.
These same templates do not account for the fact that personnel on the front-line of your business who are working to restore business operations will not have access to the alternate systems.
In order for strategies to be successful, they must define what actions need to be taken in the first hour of the crisis, who will be responsible for the decisions, and they must provide the fallback workflows that can be implemented.
The core framework for making decisions
Most guides to the difference between business continuity and disaster recovery waste time discussing a definition of the two types of plans.
The distinction between business continuity and disaster recovery is purely academic; disaster recovery is the IT component of a larger continuity framework.
The real framework for making decisions is a financial and operational one in nature.
Downtime tolerance vs. cost
The cost of resilient solutions increases dramatically as the recovery point objective approaches zero downtime.
Assuming that a company can restore every system immediately upon experiencing an outage will lead to financial ruin for an IT department.
It is essential for decision-makers to weigh the cost of resilient solutions versus the cost of downtime.
For example, in a customer-facing payment gateway scenario, a five-minute recovery period could warrant a significant investment in a dual-region active-active failover solution.
On the other hand, a back-office vacations request portal would not warrant such a high investment.
It is the responsibility of leadership to categorize these systems based on their importance.
Tier 1 systems are considered mission-critical. Revenue generation would cease altogether if a tier 1 system were to experience an outage.
Tier 2 systems are important but could sustain limited downtime through manual workarounds.
Tier 3 systems are deferrable and would not significantly impact revenue generation if they were offline for up to one week following the primary outbreak.
Evaluating realistic RTO and RPO goals when over-stressed
Recovery Time Objective (RTO) defines how quickly you want a system to be available again. Recovery Point Objective (RPO) defines how much data you are willing to lose.
These are the two critical foundations for continuity recovery standards. However, RTO and RPO are frequently created in a vacuum.
For instance, declaring an RTO of four hours on a large database is of no value if the bandwidth available at the alternate site makes it impossible to download the data within four hours.
Additionally, establishing an RPO of fifteen minutes would require an ongoing and aggressive method to synchronize data between the primary and alternate sites, which could adversely affect the performance of the primary site.
These two metrics must be validated against the actual capabilities of the staff executing the continuity plan when they are working under high-stress conditions.
A critical underlying component of continuity planning is how quickly exhausted personnel can execute a continuity plan.
Real-world situations: Strategy implementation in practice
Strategies only have real value if they provide solutions to the most serious and time-consuming events that negatively impact your business.
Excellent strategies employ branching mechanisms to handle various situations related to crisis.
Supplier chain disruption
A manufacturer is heavily dependent on an offshore supplier of a critical component.
Due to a localized natural disaster, the supplier’s facilities are closed indefinitely. There is no IT disaster recovery solution to this issue.
This is purely a question of business continuity. A realistic plan will not simply say "source alternate suppliers."
It needs to identify three vetted suppliers, include information such as service level agreements (SLAs), average lead times, and emergency contact numbers for each supplier.
The financial premium that the company is willing to accept for expedited delivery must also be included.
In addition, scripts are provided for customer support personnel to manage expectations for product delivery delays.
Business continuity issues that are not related to Information Technology often continue to be overlooked, yet many businesses today face the greatest risk in terms of supplier instability.
Ransomware complete shutdown
A healthcare provider or financial organization discovers that ransomware has infected all of their primary storage systems.
The first decision you will make will impact the long-term viability of your organization.
Your continuity plan must clearly define who is authorized to disconnect external network access and initiate a complete shutdown of your systems.
Your plan should designate a successor to the Chief Information Officer if the CIO is unable to be reached at that time.
Ransomware presents an unusual issue regarding timelines. System restoration cannot occur just from a backup, as it must be verified extensively.
If the backup is also compromised, the time to recover from an incident will extend from a few hours to several weeks.
In order to have an effective strategy, it should provide a step-by-step process for restoring immutable backups into a clean environment, verifying the integrity of that data, and determining the order in which critical patient and financial systems will be restored.
Facility loss / work force evacuation
A fire may burn down a company's original office building, or a regional crisis may force the evacuation of an entire workforce.
While the rise of hybrid and remote working has reduced some risks to physical locations, there will always be some level of significant risk.
For example, what happens to operations leaders who currently receive their physical mail at their place of employment?
How do they meet local compliance requirements when processing information from outside of the jurisdiction in which they were initially received?
As such, continuity plans should clearly outline how to manage alternative site logistics, as well as establish acceptable forms of manual workaround processes should remote access to the main server be unavailable due to an event or circumstance.
For example, if fifty per cent of the workforce is not able to work, the continuity plan should clearly identify the absolute minimum staffing level required to perform Tier 1 operations.
Building a continuity matrix that can be used
Continuity planners must create an operational playbook to avoid requiring the creation of a useless, text-heavy binder. The best way to present this information is to use the continuity matrix format.
Using the continuity matrix format allows continuity personnel to present information in a manner that provides strict, actionable guidance in the case of an unexpected operational disruption.
Planners can quickly reference the matrix during the chaotic nature of a disruptive event.
System / owner allocations
For every critical system and process, there will be a corresponding row in the continuity matrix.
The row will contain the system name, the primary business process owner, the IT recovery owner, the respective Recovery Time Objective (RTO), and the respective Recovery Point Objective (RPO).
The matrix will eliminate any ambiguity regarding these allocations.
The sales director will know which member of IT is accountable for the restoration of the CRM and the IT leader will understand what the expected timeline is for that restoration.
The first 60 minutes
The matrix should contain a column specifically for first actions to be taken.
The first hour following the declaration of disruption will be spent determining what should be restored first and what decisions must be made to deal with immediate limitations.
Who initiates the emergency communications bridge; who alerts the legal contact for potential data breach compliance?
If the organization does not have a written script for the first 60 minutes, then they will spend that entire hour determining who is in charge of restoring services.
Alternate sites and manual workarounds
It takes time to recover from technical failures. The business cannot simply stop production whilst waiting on servers to boot up.
The matrix should dictate an immediate manual workaround for every Tier 1 and Tier 2 process.
For instance, if the automated order processing system has failed, then the plan should outline what the alternative is: either the processing will be done using paper forms or using spreadsheets.
It must include where manual forms will be kept, who may process them, and how the manual data will eventually be reconciled after the Primary System is restored.
Rethinking testing & validation
A plan that has not been tested is just a theory; it may be a good theory, but it is not a reliable theory.
Regulatory bodies require tests, and typically this results in organizations performing minimal, box check exercises.
Organizations are relying on high level claims and just a handful of records to support the fact that they are ready.
The next phase beyond basic tabletop exercises
Tabletop Simulations (where many leaders gather in the same meeting room to discuss a fictional situation) are a great way to get started since they help you identify the most obvious basic logical flaws in your plan.
However, the fact that you completed a Tabletop Simulation once every three months is simply the starting point and is not enough to demonstrate that your recovery systems work and can be utilized for successful recovery.
To substantiate the Recovery Time Objective (RTO) and Recovery Point Objective (RPO) listed in the plan, it is essential for an organization to focus on performing live failover tests that actually transfer production workloads from the primary recovery environment to the secondary recovery environment and simulate the actual scenarios IT team members will face during a live failover.
It challenges the IT team by forcing them to adapt to the reality of DNS updates, latency problems, and authentication failures.
Additionally, an organization should conduct a full annual Recovery Simulation to demonstrate that the stated RTO and RPO can be achieved.
The communications exercise
A significant failure mode associated with continuity events is the failure of communication.
When a cyberattack targets a primary network, it is common for it to affect the corporate internal email and message platforms.
Testing should also include conducting Communications Drills.
For example, how do leaders communicate with one another if they cannot access their corporate accounts?
A plan should outline the secure, external communication platform that will be used for coordinating during an emergency.
Companies should deliver and regularly conduct scheduled exercises to confirm that all key personnel have installed the communication app, remember their username and password credentials, and are available during off-hours.
Trends in cloud resilience & automated failover testing
The evolution of IT infrastructure has completely changed how businesses design for continuity.
Cloud architecture provides the speed and flexibility to provide resilience that previously could only be found at the mega-enterprise level.
Multi-region recovery
Using one data center leaves an organization at risk of experiencing significant downtime due to natural disasters, hardware failures, or other unforeseen catastrophic events.
Evolving strategies argue that the focus is now on Multi-Region Recovery.
Modern companies use geographically dispersed cloud regions to protect themselves from disaster by replicating the data and application states.
If an entire cloud availability zone becomes inaccessible due to, for example, a power grid failure, the routing will automatically move traffic to the secondary region.
This approach to disaster recovery greatly reduces overall downtime, but requires sophisticated data synchronization to prevent data corruption as traffic is shifted over.
Cybersecurity & artificial intelligence
Cybersecurity today is no longer just a separate discipline.
It is heavily intertwined with business continuity.
With this in mind, increasing numbers of businesses are using artificial intelligence to monitor their networks for abnormal behavior — such as subtle changes in data transfer patterns, which can be indicative of a possible ransomware attack.
In the event of a ransomware attack, automated response protocols can immediately isolate the servers from the rest of the network and protect the company from the effects of a localized breach, thereby minimizing the impact to the entire organization.
However, maintaining and fine-tuning these autonomous defenses still requires skilled threat hunters and SOC analysts, driving continuous growth and specialization across the global cyber security job market.
The role of the human factor in disaster recovery
While technology is the solution to many data-related problems, it is still the people who provide the real answers to business-related problems.
Automated failover solutions, no matter how sophisticated, always require human judgment to determine whether a given outage warrants triggering a failover.
In addition, there are inherent risks associated with failovers, including potential data loss or degradation of service.
Finally, it is critical that organizations provide ongoing training to their staff on both the technical processes for failover as well as the psychological impact of an extended outage.
Decision-making fatigue develops quickly in times of crisis, so providing clear paths for escalation will prevent lower-tier engineers from becoming paralyzed when faced with making critical business decisions.
The final verdict
A business continuity strategy, also known as a core function of a company, does not serve merely as a safety net; this statement indicates that every organization should implement resiliency in daily operations as opposed to only utilizing a strategy when a disruption occurs.
In order to achieve success, business planners need to find and use scenario-driven matrices.
Additionally, being able to quickly determine what functions are critical, the enormous cost of quickly recovering from a major disruption, and to continually test their systems to the breaking point to determine viability.
Organizations cease to fear downtime when their continuity plan combines cyber recovery, redundancy in supply chains, and defined duty responses into a single, seamless entity.
Organizations will prepare for, manage, and sustain operations during disruption.
Frequently asked questions
What is the distinction between business continuity and disaster recovery?
The business continuity plan covers the strategy as a whole to maintain business requirements and processes before and after disruptions.
How the business completes its work through alternative suppliers as well as what physical location and means of communication with its employees can be used is part of business continuity.
Disaster recovery focuses on how to restore the IT infrastructure, applications, and data after the company has incurred an interruption in service caused by either natural or cyber-related threats.
How frequently should a business continuity plan be tested?
At the very minimum, when a company has leadership, quarterly tabletop assessments should be held to keep response plans current and account for personnel changes.
Technical failover testing on critical systems must be conducted twice each year.
Entirely testing and auditing a successful recovery simulation with both IT and business units must take place at least annually.
What are the indicators of an effective business impact analysis?
The effective BIA makes difficult financial choices by illustrating which departments are not equally important within an organization; each specific system has a dollar amount attached for every hour of downtime.
As a result, the organization's leadership can correctly budget for disaster recovery resources in a manner that best aligns with the business' actual risks.
How do cloud failovers reduce an organization's recovery time objectives?
The use of cloud failovers allows an organization to create and access virtual servers in a matter of minutes instead of having to wait for physical repairs to failover or restore critical business functions.
Instead, however, if either the data replication from the primary to cloud-based environment is delayed, the organization will face complications in developing a Response Implementation Timeline (RIT) as those incidents can create significant data loss, particularly when an organization incurs a sudden disruption before the synchronization process is completed.