The show must go on! Broadcasting belongs to activities where disaster recovery is a key requirement. Redundant solutions are used to stay on air in case of failure of any piece of equipment. Disaster Recovery is an ultimate solution that keeps the station on air even during catastrophic failures, natural disasters and intense cyber-attacks.
Dedicated Disaster Recovery (DR) Site
Because of the fortunately small chance of needing to use a DR site, many customers choose to build a site with minimal functionality that has the ability to stay on air at the lowest cost. This solution is designed with an operational model that has the least disruption to existing workflows and requires minimal, if any new operator training.
Dual Purpose Sites
An alternative solution is one or more dual purpose sites that can be used for every day operations but have additional channels that can be quickly activated. This is generally an existing location of the same media company that has resources which could be utilized if a remote site goes off air.
Considerations for a DR site
Below are some considerations in designing a DR solution. This is not a comprehensive list, but can be used as a starting point to select among Aveco’s many options.
- Video duplication - video material needs to be constantly updated at the DR site. Ideally the automation systems will monitor the playlists and automatically move the required files to the site in order of its priority in the playlist thus keeping both sites synchronized.
- Playlist Duplication, short term - a 24 hour playlist is the recommended minimum that needs to be always available at the DR site to provide protection for the next 24 hours. Automation can generate a 24 hour playlist which can then be continually synchronized with the DR site.
- Playlist Generation, longer term - if the DR site is designed to run for more than 24 hours, which would generally be the case, considerations for generating playlists for the next 2-3 or more days need be planned. While a separate traffic system at the DR site would be optimal - but expensive, at minimum, simple tools like Excel can be used to create a playlist if the main traffic system is also down or unavailable. Aveco can import simple text based schedules from everyday tools like Excel or even simple text editors.
- Broadcast Equipment Chain - one of the major costs for a DR site is the equipment. The functionality requirements of the DR site determine the equipment needs. For a basic DR site, minimal functionality can result in significant cost savings. Items like switchers, graphics, routers and other equipment should each be analyzed for minimum required features. In some cases, not all existing types of equipment may be needed. The goal is to stay on air — not necessarily look fancy. A video server is one of the basic requirements. Adding graphics for informational crawls, logos or simple announcements are generally a good idea especially in emergencies. Switchers and graphics might be eliminated for very simple designs without any type of studio. To keep development and operational costs low, keeping the same type of equipment (but maybe simpler functionality) as in the main facility maintains similar interfaces to automation and minimize any additional operational training.
- News Studio - often it is desirable to have live broadcasting capability at the DR site for reporting current events. A studio can be a major cost but again can be done simply without many of the fancy features in today's News studios. Single camera operation with minimal graphics may be adequate.
- Automation - to control the DR site, having the same automation system as the main site is beneficial if it is relatively up-to-date and fits into your long term growth plans. If your existing system is old and ready for replacement or is no longer or poorly supported, a new Aveco system may be in order. Since it is easy for automation systems to import playlists from traffic, having a different automation system at the DR site should be a consideration rather than buying an older technology that does not fit your long term plans. The only downside is additional operator training if only the DR site is upgraded but since much will be running without operator intervention, this may not be a big issue. Ideally you should take the opportunity to update both sites.
- Operations and Workflow - to minimize disruptions to current workflows and minimize operator training, a DR site workflow should be as similar to the main facility's workflow as possible. Since these are infrequent events, it is critical that training and workflows remain as similar as possible. Look for a system, like Aveco, that can easily be adopted to various workflows.
- Ingest - the DR site needs to be designed with ingest capability including live feeds and manual ingest. If the station or network does news, often you will want to broadcast videos shot by the public in addition to professional formats, so a basic transcoding platform would be provisioned – Aveco integrates with all major transcoding systems.
- Facility Restoration - this step is often overlooked but moving operations back to the main site needs to be part of the overall DR workflow. Syncing all new material can be manual by tape or electronically over the network but all material will need to be available to continue current stories from the main site.
- Monitoring and Testing - the worst thing that can happen is to experience an emergency and have the equipment at the DR site be non-operational when needed for service. Thus a plan to verify that all the backup equipment and workflows are operational is required. Not only the physical equipment needs periodic testing but operators need to refresh their training so everyone knows their roles and responsibilities when needed. Technology such as SNMP remote monitoring can enable you to automatically monitor the hardware at all sites. Aveco’s client-server software architecture is well-suited to local-and-remote operation and is recommended.
Financial Aspects of a DR Site
Significant thought and planning need to go into justifying the investment of a disaster recovery site. There are many options and Aveco experts will be honored to provide their expertise for consideration. Here are a few operational and financial concerns to consider.
- Cost of going off-air – during times of natural disasters, the community service of broadcasters is at a peak, and if a station is off-air, it’s a major blow to the broadcast company. This can result in long-term image losses that will be very difficult to regain.
- Short-term Make-Goods - one easy analysis is the lost revenue for every hour off air. Your program costs stay the same if you broadcast a program or not but you will have to make good on the spots that did not air or lose that revenue. Depending on when during the day you are off-air, this can be a significant cost.
- Disaster damage - if a disaster occurs that does physical damage to the site, then off-air times will be measured in weeks or months while repairs are being made. Without advanced planning, trying to find alternate broadcast facilities and equipment will be an expensive and time-consuming scramble often resulting in mixed results. Even a basic facility will get you back on air with minimal functionality quickly. Loss of revenues for a lengthy amount of time can be devastating, even with insurance coverage, not to mention loss of income for employees, possible loss of key personal, and loss of community service.
- Dual Use Sites - to minimize the expense of a DR site, building it so the backup resources can be used during normal times will help financially as well as providing excess capacity. An additional studio could be used for live events without tying up the main studio. The additional equipment could be used for ingest and QA during busy times. Maintenance of equipment at the main facility can easier be performed by taking it off-line and using the DR site's equipment.
Aveco has the solutions described in this paper available today and can help with your planning and implementation. Our ASTRA MCR and News solutions are built with a unique networked, distributed architecture that controls equipment at remote sites as easily as local equipment is controlled. A disaster recovery system can be designed such that two or more sites operate technically as a single site. In a DR situation, one site can go down and control can pass to another site seamlessly with a redundant, mirrored design.
In times of stress, good planning and forethought can minimize the disruption to a station's programming and finances during disasters while at the same time provide a valuable service to the community. Many options are available to you based on your budget and needs but the key is to start planning today. Aveco industry experts are available today to help with your planning.