Abstract: Evaluation and testing are major steps in the development of any information system, particularly if it is to be used in high-risk contexts such as conflicts. While thus far there are various approaches for testing against technology requirements; usability or usefulness, there is a lack of a comprehensive evaluation framework that combines the three elements. The lack of such a framework and commonly agreed standards constitutes a barrier for innovation, and at the same time imposes risks to responders if the technology is introduced without proper testing. This paper aims to close this gap. Based on a reviewing of evaluation methods and measurement metrics, we design a comprehensive evaluation framework including common code quality testing metrics, usability testing methods, subjective usefulness questionnaires, and performance indicators. We demonstrate our approach by using the example of an integrated system for the safety and security of humanitarian missions, and we highlight how our approach allows measuring the system?s quality and usefulness.

Abstract: While different governments worldwide have published sets of recommendations-or even laws-for increasing preparedness, a reference framework to assess the level of compliance of organizations is still to come. For instance, emergency plans often remain stored in closets where they stay until some emergency or major legal change occurs. Consequently, achieving actual preparedness is difficult to assess. QuEP is a framework for the assessment and improvement of the management of emergency plans within organizations. It is inspired by the Total Quality Management strategy, and provides a hierarchy of emergency plan management maturity levels. The aim of QuEP is to guide organizations to assess and improve their emergency preparedness by following a set of principles, practices and techniques at the technical, human and strategic levels. In this paper, we show the model underlying the framework, and give details of the current framework evaluation processes.

Abstract: In this article the focus is how to train collaboration and communication between emergency authorities in two countries (Norway and Sweden) by using a web-based tool supporting tabletop like exercises. The exercises are accomplished in three steps: Scenario design; exercise design and realization; and evaluation to examine the results of the exercises and for feedback to new scenario designs. The software ties all three steps together. The process is iterative, and involves users from each emergency authority. The preliminary results after two years show that the approach is promising. To be able to better foresee what will happen during an exercise the need for a simulator has appeared as one desirable and possible direction for further research.

Abstract: Failures in coordination and information sharing between relief agencies have increased the number of calls for innovative information system (IS) designs. While both the academic and the industrial communities have proposed many IS designs, methodologies for the ex-ante evaluation of such IS designs are scarce. Consequently, disaster IS architects are offered little guidance in the ex-ante evaluation process. Not only is it difficult to evaluate IS designs in practice, it is also difficult to include the conditions of disaster situations in the evaluation process. This paper explores the difficulties of ex-ante evaluation and discusses the suitability of the gaming-simulation methodology for the evaluation of principle-based IS designs. Gaming-simulation entails the use of professionals, scenarios and prototypes and can be adapted to a quasi-experimental form enabling researchers to control contextual interferences and rule out alternative explanations. This paper concludes with some discussions on the advantages and pitfalls of employing gaming-simulation for IS evaluation.

Abstract: We present work on developing and evaluating a serious GIS spatial thinking game called SerGIS. We conducted a spatial think-aloud study with ten participants new to crisis response who used SerGIS with a coastal city hurricane scenario. Four themes emerged from participant responses: processes of reasoning, tools of representation, overlay and dissolve operation, and geographic information concept learning and knowledge gaps. The first three themes match directly with the spatial thinking theory and evaluation underlying SerGIS. The fourth theme identified addresses GIS and spatial thinking crisis response educational issues. Furthermore, statistical evidence indicates there is likely a relationship between participants from spatially-oriented backgrounds (but with no GIS experience) performing better with SerGIS than participants from non-spatial backgrounds. Finally, we found that with a game scenario based on established disaster management practitioner literature, participants could focus on spatial thinking tasks and not be limited by understanding the game scenario itself.

Abstract: In an earthquake disaster, having proper estimation about destructed buildings and the degree of destruction, can considerably facilitate decision-making and planning for disaster managers. Using this information, the managers can estimate disaster area and number of victims to determine and allocate required resources. Scientific studies and historical data show that the faults around Tehran, the capital of Iran, are capable to create strong earthquakes which would bring the largest damages in the world history to the city. So it is necessary to be prepared for a rapid and knowledge-based response to such an earthquake. Therefore, development of a knowledge-based model to estimate destruction of buildings is ongoing. The model is going to be developed by using different spatial data obtained from the buildings and its environment in Tehran. This paper outlines the initial results of this research.

Abstract: The undefined flow of execution of activities in an evaluation process hampers its implementation. A consistent evaluation process defines interrelated methodological steps that make it easier for the evaluator to lead the process. This article presents a process model for the evaluation of simulated field exercises in the emergency domain, including their sub processes and activities. The proposed model was derived from observations made during real situations of a simulated evacuation exercise of communities in high-risk areas in Rio de Janeiro (Brazil). The motivation came from the finding that the assessment of simulated field exercises is conducted by completing an activity report that does not follow a structural model, an evaluation program or a formal standard. The results of this research show the experts' satisfaction with the application of the model proposed for the development of an evaluation process. The same occurs when comparing to reports currently used by them for this purpose.

Abstract: When a disaster or emergency occurs, one of the most pressing needs is to establish a communication network for the first responders at the scene. Establishing and accessing a reliable communication infrastructure at a crisis site is crucial in order to have accurate and real-time exchange of information. Failure in the exchange of timely and crucial information or delay in allocating resources impedes early response efforts, potentially resulting in loss of life and additional economic impact. At a disaster site, the existing communication infrastructure may be damaged and therefore partially or totally unavailable; or, there may not have been previously existing infrastructure (as in the case of remote areas). A communication infrastructure within the context of emergency applications should be reliable, easily configurable, robust, interoperable in a heterogeneous environment with minimum interdependencies, and quickly deployable at low cost. A disaster scene is a chaotic environment which requires a systematic approach to abstract the system, study the flow of information and collaboration among different disciplines and jurisdictions to facilitate response and recovery efforts. We have deployed the wireless mesh infrastructure in several drills at the university campus and in the city as part of the California Institute for Telecommunications and Information Technology (Calit2) NSF-funded RESCUE project (Responding to Crises and Unexpected Events). To evaluate network performance and identify the source(s) of bottleneck, we have captured the network traffic. The lessons learned from test bed evaluations of the network based on real-world scenarios can be applied to future applications to enhance the network design and performance.

Abstract: Unreliable communication networks, chaotic environments and stressful conditions can make communication during crisis events difficult. The current practice in crisis management can be improved by introducing ICT systems in the process. However, much experimentation is needed to determine where and how ICT can aid. Therefore, we propose a framework in which predefined modules can be connected in an ad hoc fashion. Such a framework allows for rapid development and evaluation of such ICT systems. The framework offers recognition of various communication modalities including speech, lip movement, facial expression, handwriting and drawing, body gesture, text and visual symbols. It provides mechanisms to fuse these modalities into a context dependent interpretation of the current situation and generate appropriate the multimodal information responses. The proposed toolbox can be used as part of a disaster and rescue simulation. We propose evaluation methods, and focus on the technological aspects of our framework.

Abstract: This paper reports on the added value of combining different types of sensor data and geographic information for fire incident management. A survey was launched within the Belgian fire community to explore the need of added value and the use of new types of sensor data during a fire incident. This evaluation revealed that people are visually-oriented and that video footages and images are of great value to gain insights in a particular problem. However, due to the limited available time (i.e., fast decisions need to be taken) and the large amount of cameras it is not feasible to analyze all video footages sequentially. To solve this problem we propose a video summarization mechanism and a video highlight selection tool based on the automatic generated image and video tags.

Abstract: Disaster management exercises are a core component of humanitarian organizations’ preparedness strategies. They entail diverse purposes, from training capabilities of participants to testing response plans to enhancing collaboration between organizations and many more. However, it is uncertain how much exercises contribute to preparedness. Rigorous evaluation is needed to exploit learning opportunities of an exercise. Therefore, exercises must target evaluable objectives, which is complicated by the socio-technical openness of the exercise system, the heterogeneity of organizational needs, and the scarcity of resources. Many different tools aim to support evaluation but are limited to specific use cases, resulting in a fragmented overview for practitioners. Due to the excessive effort involved, practitioners often consider exercise evaluation to be of secondary importance. This study thus proposes the conceptual design of a combined toolkit that supports the practitioners in a more rigorous but resource-efficient evaluation to make disaster management exercises more evaluation-driven.

Abstract: Artesis Virtual and VSTEP BV developed a virtual scenario training for leading firefighters. We evaluated the usability of the training application and measured the satisfaction of 53 trainees. The trainees were positive about the training application, the training itself, and the interaction with the interface. We observed that trainees experienced navigation difficulties and sometimes misjudged visual information. Evaluations of new versions of this application, should address the influence of interface problems on performance.