News Report
The U.S. DoD has recently published the Defense Department's Director, Operational Test and Evaluation FY2011 Annual Report. The document reports the findings of a systematic review by the Office of Director, Operational Test & Evaluation (DOT&E) of recent major U.S. acquisition programs that experienced delays. These programs were examined to determine the causes and lengths of program delays, and the marginal cost of operational test and evaluation.
The report emphasizes that each U.S. Service operational test agency has developed methods for rapidly evaluating systems fulfilling urgent operational needs, including combining testing with the training of the first unit to be equipped and conducting quick reaction assessments. A typical example of such approach is the Network Integration Evaluation (that has been extensively covered by this blog, and that resulted one of the key C4I trends of the last period). Network Integration Evaluations (NIEs) are executed twice a year at Fort Bliss, Texas, and White Sands Missile Range, New Mexico, in order to provide a venue for operational testing of U.S. Army acquisition programs with a particular focus on the integrated testing of programs related to tactical communications networks supporting command and control. The exercises are also intended to provide an operationally realistic environment to evaluate new emerging capabilities that are not formal acquisition programs.
Together with NIESs, a substancial effort was dedicated by U.S. DoD to assess the situation concerning Cyber Warfare, Information Assurance and Interoperability. U.S. DoD recognizes the importance of applying the same approach of the NIE to these domains, and in fact it is planned that by the end of 2014 the Department should have in place all the capabilities and processes to perform selected evaluations of offensive and defensive cyber-warfighting capabilities in representative cyber‑threat environments. This will allow to assess how well U.S. fighting forces can defend against or fight through the most serious cyber attacks, as well as perform defensive and appropriate response. In order to apply these enhanced capabilities across all major exercises and acquisition programs, the Department will need to identify additional resources to expand the capacity and capabilities of the Red Teams who portray advanced cyber adversaries. This would include funding the cyber-ranges and modeling and simulation capabilities that provide operationally realistic environments for those activities inappropriate for live networks, as well as assessment teams to develop rigorous plans to ensure the cyber adversary is accurately portrayed, and assess the effects of representative cyber adversary activities.
According to the DOT&E report, hoevewev, important analyeses and assessments on cyber, interoperability and information assurance have been carried out already in 2011. In February 2011, the Chairman of the Joint Chiefs issued an Executive Order directing that all major exercises include realistic cyber-adversary elements as a training objective to ensure critical missions can be accomplished in cyber-contested environments. During 2011, the DOT&E Information Assurance (IA) and Interoperability (IOP) Assessment Program performed 23 assessments during combatant command and Service exercises; four of these assessments involved units preparing to deploy (or already deployed) to Iraq or Afghanistan.
The results were not so good...
Information Assurance and Network Defense
The IA posture observed during the assessed exercises was considered not sufficient to prevent an advanced adversary from adversely affecting the missions that were being exercised. DOT&E also observed modest improvements in certain areas of network defense, but there were also several areas in which prior progress has declined. In general, information technology and personnel were not fully prepared to operate in realistic and contested cyberspace conditions. Red Teams generally overcame defenses during exercises by only moderately increasing their level of effort over previous years.
Specifically, most Red Teams reported increased difficulty in penetrating network defenses, but results show that with sufficient time, Red Teams routinely managed to penetrate networks and systems. Detection rates of network intrusions remained low, and the ability of network defenders to detect subsequent exploitations of network data was minimal; most assessments witnessed large exfiltrations of operationally significant data. The extracted data was available, in only a few cases, to the exercise opposition force for tactical/strategic exploitation, which in effect created a more benign exercise environment than postulated by DIA and the intelligence community.
The assessments also showed a decrease in the use of backup files and systems, proper audit logging and reviews, logical access controls, incident planning, and vulnerability management. There was an overall increase in high-risk vulnerabilities observed (indicating a decrease in effective patch management), as well as a decrease in effective use of anti-virus tools and software (including failures to routinely update virus signatures). Although the ongoing fielding of the Host Based Security System (HBSS) has resulted in many local improvements in network protection from intrusion as well as intrusion detection, the majority of HBSS suites observed were found to be incorrectly or ineffectively configured.
Interoperability
The 2011 assessments found that interoperability issues encountered by the training audience typically hindered, but rarely prevented, mission accomplishment; this is due primarily to operators who developed and executed workarounds that may have preserved the timeliness and accuracy of mission data at the cost of the efficiency or level of effort required. Even though missions were generally accomplished, the workarounds usually increased operator workload, and often resulted in degraded effectiveness in completing mission tasks. Assessment teams documented measurable impacts to the timeliness, accuracy, and efficiency of operational data handling in these assessments.
A major source of poor interoperability was often found to be an incomplete set of interface requirements, or uncoordinated upgrades and updates to interdependent systems. Some of the observed mission impacts include: delays in critical battlefield situational awareness, reductions in forces available for operational tasking due to delays or inaccuracies in planning systems, re-allocation of personnel from less critical tasks to support increased manual efforts for critical ones, large-scale exfiltration of operationally significant data from force planning systems, modification of blue-force operational data by opposition force actors, manual transfers of information between systems unable to automatically interoperate.
It was also found that less than one-third of all systems observed during assessments had been fully certified for interoperability, although configuration management and documentation was satisfactory in almost 9 of 10 systems reviewed. Despite the lack of interoperability testing/certification, local authorities certified these systems for network operation. In some instances, major software suites were found to be in operational use despite having not completed operational testing or interoperability certification.
Unresolved interoperability issues, coupled with low-to-moderate level threats, were observed to be sufficient to adversely affect the quality and security of mission critical information in a way that could, and did degrade, mission accomplishment. Interoperability and IA problems are rarely observed in isolation from each other, but are frequently interrelated.
In 2012, DOT&E will continue to support the implementation of more realistic cyber threats in exercises and will report both the IA and IOP results of these assessments.
References: U.S. DoD (1)
