"Anything that can go wrong will go wrong." -Murphy's Law
Oftentimes misadventures and quirky failures are attributed to the Fates and Murphy's Law, as if we should have a reasonable expectation that everything will go smoothly all the time. Of course, given even the shortest amount of thought, the notion is absurd; especially if you work in IT! Whether we like it or not, we are dealing with complex systems every day, whether those be computers or cars or trains or planes or humans. The amount we don't know pretty much always exceeds what we do know.
As such, it's time that we embrace Murphy's Law. Instead of fighting the inevitable, as has been the modus operandi of security the past few decades, we need to adopt a survivability mentality that focuses on defensible and recoverable systems and processes. Murphy's Law enlightens us greatly in this regard: if we don't embrace failure, then failure will embrace us. And, as no position is absolutely defensible, it seems that a good place to start embracing Murphy's Law is in enhancing system and process recoverability.
There seem to be four key areas where a recovery mindset should always be applied:
* Hardware Failures: Hard drives fail. Network links go down. Cooling fans die. Power cables get cut. These are facts, not FUD or innuendo.
* Schedule Failures: It's not always possible to get the right people in the right place at the right time. It's not always possible to get the right equipment ordered and delivered on the schedule desired.
* People Issues: We like to believe that people are capable of being consistent and reliable, and for the most part this is true. However, Murphy's Law tells us that we should expect key people to encounter unforeseen issues, such as sickness or family emergencies, at the least opportune time.
* Unclear Requirements: One of the more fatal flaws in managing projects or people is failing to clearly articulate the expectation for performance. Yet, even when requirements are specified clearly, concisely, it can be difficult achieving a common understanding. As such, one should expect fuzziness around requirements, and thus gaps between expected and actual performance.
To address some of these challenges, and to help embrace Murphy and his Law, these key practices are recommended:
- Update Policies: Policies provide your first line of due diligence effort when it comes to planning for the unexpected. Organizations should be familiar with business continuity planning and disaster recovery plans (BCP/DRP), but remember to expand those to account for more than just your typical break-fix scenarios. Additionally, policies for sick leave and remote access should be brought into the current era by allowing for extraordinary circumstances. Despite the media hype surrounding Avian Influenza and Swine Flu (H1N1), these plans should take into consideration pandemic scenarios (this Fall has already seen particularly virulent cold and flu strains). Plans should also consider natural disasters, man-made disasters, etc.
- Ensure Remote Access Capabilities: One key consideration in the face of schedule and people challenges is finding ways to bring people together online when face-to-face collaboration isn't possible. Teleconference solutions, unified communications using VoIP and instant messaging, and video conference technologies have all matured well in the past few years to meet some of these needs. In terms of remote access, one additional consideration is to discuss spike license agreements with your VPN vendor, such as to be used in the case of a pandemic or weather disaster that would necessitate a largely remote work force for a short period of time.
- Have a Communication Plan: It is imperative that organizations have communication plans in place, and that they provide personnel with routine awareness training about the communication plan. Severe weather, such as blizzards, ice storms, or tornadoes, can bring commuting to a standstill. The sudden emergence of a quickly spreading pandemic can force a switch into an emergency remote-worker configuration. In all cases, it's important to establish multiple communication vehicles, make personnel aware of those vehicles, and then follow the plan as needed. Incidentally, don't just rely on a single web site for your status communication, since the loss of your computing facilities could make it rather difficult to get the message out. Instead, make sure your communication plan is suitably diverse, making use of two or more communication vehicles for primary communication with personnel.
- Test It! One of the worst things you can do is write policies and plans without testing them. In the middle of a crisis is the wrong time to learn that you made an error in planning. Instead, test plans on a regular basis (at least annually). This advice goes double for failover sites. If you don't test failover plans, then how do you know that they'll work? The last thing you want to do is compound an event by having additional failures. An ounce of prevention is worth a pound of cure.
(Note: this was originally cross-posted from a previous employer's site, but they let me go, so I'm yanking the link-back.)
