IT Disaster Recovery Plan

 Policy

The equipment, software systems, and databases that comprise the electronic Private Health Information (ePHI) System are critical components that enable RACS to function in an effective manner.  The purpose of this plan is to provide the framework for recovering from any disaster that might affect these systems, to minimize downtime, and to assist users in meeting their critical processing requirements.

IT Disaster Recovery Plan ExampleIT Disaster Recovery Plan Example

HIPAA Security Regulations Addressed In This Policy

164.308(a)(7) Disaster Recovery Plan

IT Disaster Recovery Plan: Attachments

Attachment A (Disaster Recovery Team), B (Items for Complete System Recovery), C (System Requirements)

 

IT Disaster Recovery Plan: Guidelines

 

  1. The decision to implement IT disaster recovery plan procedures is the responsibility of the IT Manager or her designee. The Disaster Recovery Team, (see Attachment A), will convene as soon as possible after a disaster has occurred to assess damages and make recommendations to the IT Manager.
  2. This plan is distributed by the IT Analyst and used by those persons responsible for its implementation and operation.  These individuals are identified in Attachment A, Disaster Recovery Team Members.  This document will be maintained and updated by RACS IT Staff whenever significant changes occur.  Updates are performed at the direction of the IT Manager.  The IT Manager forwards the updated document to the Leadership Team for approval.

 

C.  The Disaster Recovery Team is established and organized to assess the damage to the computer systems and capabilities, to implement and coordinate recovery/backup actions, and to make recommendations to the IT Manager. The team consists of persons responsible for one or more of the following functions:

1.   Recovery administration

a.   Insurance Notification

b.   Supplies

c.   Organization

2.   Systems Software

3.   Application Software

4.   Communications

5.   Operations

6.   Facilities

7.   Hardware

 

The IT Manager serves as the Chairperson of the Disaster Team. In the absence of the IT Manager, the IT Analyst is to serve as the Chairperson. The Disaster Recovery Team meets on an as needed basis as changes occur, or, at least, annually to discuss current documentation and make recommendations for changes.  The IT Manager must approve all changes before they are forwarded to the Leadership Team for final approval.

In the event of a disaster or major failure, members of the team assess system and infrastructure damages.

 

D.  Daily backups of the complete system and databases, critical to the restoration of service, are stored in a fireproof container or file cabinet on site (see IT Backup Policy). A weekly backup will be stored offsite in a bank safety deposit box. The IT Backup policy insures that the most current full system backup and the most recent full database backup are stored far enough away from their respective servers to be safe in the case of fire or natural disaster.

E.   In the case of fire or natural disaster it may become necessary to move the computer room to a backup location or alternate site. It may not be necessary to move the computer room; alternate equipment may be put into service as a temporary measure.

 

IT Disaster Recovery Plan: Procedures

 

Several recovery processes are identified, depending on the circumstances:

Disaster Preparation Plan
Being ready and planning ahead is the easiest way to quickly and fully recover from a disaster.  This section outlines the minimum steps needed to insure full recovery from a disaster:

The IT Manager or designee is responsible for insuring implementation of and compliance with the following requirements:

  1. The disaster plan is kept current and all of the employees on the recovery team are made aware of any changes.
  2. The off-site storage area is inspected; at least quarterly, to insure it is clean, organized and that the correct backups are in storage.
  3. A full set of the latest version of every file, patch, driver and software package needed to “recreate” the server is created and stored off-site in a locked fire-proof container.  The IT manager, IT Staff and certain key designees have the only keys to this storage container and they are each individually responsible for the security of her key.
  4. Any fire fighting systems or equipment located in network server rooms are properly maintained.
  5. Employees are advised of the consequences of a disaster and what they can do while recovery is in progress.
  6. Minimum & Recommended specifications for replacement hardware, located in Attachment C, are kept current.  The minimum requirements allow for flexibility in acquiring temporary replacement equipment to restore basic services (either full or degraded) during the acquisition of new permanent replacement equipment.

 

In the event that there is warning of an impending disaster, e.g. potential flood situations, severe weather activity in the immediate area, fire or potential building damage, the following steps are initiated by the first recovery team member to be made aware of the situation.

  1. Notify as many recovery team members as possible.
  2. The Executive Director and the IT Manager are briefed and a decision is made whether or not to shut down the systems.
  3. The recovery team convenes and develops a remedial plan of action.

 

Emergency Response.
These are the first actions taken in an emergency situation, designed to bring the computer systems back to operation. The computer systems may be functioning in a degraded state or not at full capacity temporarily.

  1. The IT Manager or designee is notified by the initial Disaster Recovery team member and made aware of an emergency situation as soon as possible.
  2. The IT Manager or designee insures that the Disaster Recovery Team members are notified and assembled as soon as reasonable under the circumstances.
  3. Team members assess damages in their individual areas of expertise.
  4. Team members advise the IT Manager as to the extent of damage and recovery procedures necessary so that the decision to move the computer center or restart on an alternate equipment platform can be made.
  5. All unit directors are informed, by the designated team member, of the decision and given an estimated time to the return to either full or degraded service.
  6. The unit directors will notify their staff via the best means available.
  7. Disaster Recovery Team members will supervise their own area of expertise.
  8. The IT Manager, Chief Financial Officer and unit directors are contacted, by the designated team member, to determine if needed replacements are available in-house or if emergency purchase orders will need to be created.

Disaster Recovery Procedures.

These are the procedures designed to return the computer systems to a fully operational, or a degraded state, including bringing up the alternate site or equipment as circumstances necessitate.

Recovery from a complete failure to a degraded mode of service may be necessary. In this case it may be possible to bring up individual locations on a priority basis.  The decision to operate in a degraded mode and the order in which locations are brought back into service is made by the IT Manager in consultation with the Disaster Recovery Team.

If it is decided to transfer the computer center to the alternate site the designated team members will take the following steps:

  1. Insure that the basic Emergency Response procedures have been followed.
  2. Create an inventory of the status of existing equipment (functional or damaged) and files (OK or corrupted).
  3. Coordinate the movement of equipment.
  4. Contact IT Staff and determine equipment availability.  If necessary, insure that emergency Purchase Orders are created for replacement equipment.
  5. Determine if a new offsite (backup) storage facility is required.  If a new site is required, immediately identify the site and coordinate its activation.
  6. Test all hardware systems as soon as they are available.
  7. Install Network Operating System (NOS), and other low-level software.  Create NOS volumes emulating the configuration of the downed server as outlined in Attachment B, Section 2.
  8. Communications, networking, operations and applications software employees prepare to install and or setup their individual function in the appropriate order.
  9. Advise Executive Director of the progress and/or impediments at regular intervals.

 

If it is decided to transfer operations to alternate equipment but not move the data center location, the designated team member(s) will take the following steps:

  1. Insure that the basic Emergency Response procedures have been followed.
  2. Determine the extent of the damage and develop a plan to bring the system back on line.
  3. Test all hardware systems as soon as they are available.
  4. Install Network Operating System (NOS), and other low-level software.  Create NOS volumes emulating the configuration of the downed server as outlined in Attachment B, Section 2.  Prepare to support and or adjust individual components
  5. Advise Executive Director of the progress and/or impediments at regular intervals.

Recovery Timetable–The following timetable does not take into account the amount of time required to input data held on hard copy during the recovery period, or inputting data that may have been lost during recovery.   Phases represent units of measure that vary in length depending on the severity of the disaster.

     Phase I        Convene the disaster recovery team and assess damages, determine equipment needs and initiate replacement, discuss options and possible solutions.

     Phase II       Restore programs and data, test integrity of programs and data. Begin restoring communications and networking capabilities.

     Phase III      Restore partial operation to priority locations.

     Phase IV     Determine priority of data processing.

     Phase V      Take delivery and setup new equipment. Restore full communications and networking capabilities. Work with departments to verify data and operation of applications.

IT Disaster Recovery Plan Review — The following steps are taken to insure that the Disaster Recover Plan is current, feasible and effective:

  1. During November of every year the Disaster Recovery Team convenes to review the Plan and Appendices.  Updates or revisions will be made at this time.
  2. The contents of the off-site disaster backup tape storage are subjected to unannounced periodic audits by the IT Manager or her designee.   Results of the audit will be documented and reported in writing to the IT manager.

 

 

Attachment A: Disaster Recovery Team

IT Manager

 

IT Analyst

 

Other Designated IT Staff

 

QI Director

 

Chief Financial Officer

 

Attachment B: Items for Complete System Recovery

 

Section 1: Software, Data and Documentation

Items needed for complete system recovery of the current servers:

‚                    CMHC  IBM 7025/F50 RS/6000

‚                    RACSB-APPS-1  Dell Power Edge 1750

‚                    RACSB- FILE -1 Dell Power Edge 1750

‚                    Webserver-BUI  IBM eServer X Series 235

 

All the below listed items are kept, as per IT Data Backup Policy, in a fireproof storage box or file cabinet.

   Dell OpenManage Server Assistant Software to configure the new server’s hardware.

   Bootable tape to create a Root partition on the CMHC server.

   Emergency Recovery Disks or tapes for all servers (CMHC,RACSB-APPS-1,RACSB- FILE -1,Webserver-BUI)

   CONFIGURE.txt file of configuration information needed to restore the servers and network connections with detailed instructions of procedures.

   WindowsEnterpriseServer 2008

   All current Service Packs and Upgrades for Windows Enterprise Server 2008

   UNIX AIX 4.x OS

   Copy of Cisco IOS configuration files

   Cisco IOS 12.3 (for Router)

   CMHC BUI Install disks

   Symantec Antivirus 8.1

   Veritas Backup Exec 9.0

   Microsoft Exchange 2008 and all related service packs and upgrades

   All required drivers for peripherals and interface cards.

   Backup tapes.

 

Section 2: Recovery Procedure

 

RACSB-APPS-1, RACSB-FILE – 1 procedure:

         Boot up from Dell Server Assistant CD to install drivers and Windows Server 2008 Operating System

         Create RAID  partitions and then create a 25 Gigabyte Operating System Partition.

         Reinstall Windows Server 2003 Operating System, Service Pack(s), and Security Updates from Microsoft, in OS partition

         Recreate Data Partitions if necessary.

         Install Exchange 2008, service packs and security updates (if needed), Symantec Antivirus, Dell Systems Management software and Backup Exec software.

         Restore for the most recent backup tape.

         Reconfigure network interface settings.

         Reconnect to network

Webserver BUI- procedure:

 

‚                    Boot from Install Disk for Windows Server 2008 and install OS and all service packs and security updates to primary partition.

‚                    Restore basic partitions, if needed.

‚                    Install IBM drivers for SCSI card and tape drive

‚                    Install  BUI Software and configure IIS and Apache applications.

‚                    Verify connection to CMHC UNIX server and BUI sign-ons.

 

CMHC UNIX procedure:

Contact vendor for additional support, if needed.

 

‚                    Boot up server from last complete backup tape by pressing 5 key on keyboard to boot from tape.

‚                    Load UNIX Sysback and restore data.

 

 

 

Attachment C: System Requirements

Minimum Specifications For RACS Server(s)

The minimum requirements for a temporary CMHC server are:

         4 GB DAT backup tape drive

         Processor: 166 PowerPC RISC 604e

         Memory: 768MB

         Storage space: 8GB SCSI (2 4GB for mirroring)

         SCSI CD ROM

         Network Interface Card

         14.4 Modem

         IBM UNIX AIX 4.3 OS

 

The minimum requirements for a temporary RACSB Windows-based or BUI server are:

 

         DAT backup tape drive

         Processor: 1.2GHz

         Memory: 512MB

         Storage space: 40GB

         CD ROM

         Network Interface Card (and modem for BUI)

         Windows Enterprise Server 2008 (Windows Advanced Server 2003, if 2008 isn’t available)

 

Recommended Specifications For IT Server(s)

The recommended server specifications for running the <software system> for 80 users and at the current development level of the software are:

 

         DAT backup tape drive

         Processor : 1.5GHz Multi-Processor

         Memory: 1GB

         Dual-channel Ultra2 SCSI controller

         Storage Space: 72GB SCSI (2 36.4GB for mirroring)

         Network Interface Card

         v.92 56K Modem

         SCSI CD ROM

         IBM UNIX AIX 5.x

 

The recommended server specifications for running the RACSB Windows based or BUI servers for 150 users and at the current development level of the software are:

 

         DAT backup tape drive

         Processor: 2.3 GHz Dual Processor or better

         Memory: 1GB

         Dual-channel Ultra2 SCSI controller

         Storage space: 320GB

         Network Interface Card (and modem for BUI)

         CD-ROM

         WindowsEnterpriseServer 2008