Abstract: Insight into performance ability is crucial for successfully implementing AI solutions in real-world applications. Unanticipated input can lead to false positives (FP) and false negatives (FN), potentially resulting in false alarms in fire detection scenarios. Literature on fire detection models shows varying levels of complexity and explicability in evaluation practices; little supplementary information on performance ability outside of accuracy scores is provided. We advocate for a standardized evaluation dataset that prioritizes the end-user perspective in assessing performance capabilities. This leads us to ask what an evaluation dataset needs to constitute to enable a non-expert to determine the adequacy of a model's performance capabilities for their specific use case. We propose using data augmentation techniques that simulate interventions to remove the connection to the original target label, providing interpretable counterfactual explanations into a model's predictions.

Abstract: Real-time information from microposts like Twitter is useful for applications in the crisis management domain. Currently, that potentially valuable information remains mostly unused by the command staff, mainly because the sheer amount of information cannot be handled efficiently. Sentiment analysis has been shown as an effective tool to detect microposts (such as tweets) that contribute to situational awareness. However, current approaches only focus on two or three emotion classes. But using only tweets with negative emotions for crisis management is not always sufficient. The amount of remaining information is still not manageable or most of the tweets are irrelevant. Thus, a more fine-grained differentiation is needed to identify relevant microposts. In this paper, we show the systematic evaluation of an approach for sentiment analysis on microposts that allows detecting seven emotion classes. A preliminary evaluation of our approach in a crisis related scenario demonstrates the applicability and usefulness.

Abstract: The evaluation of an emergency scenario is often based on the use of simulation models. The specificity of these models involves the need for a complex evaluation of the problem domain, including the physical conditions behind the considered threat. Based on emergency occurrences data, provided by the Portuguese National Civil Protection Authority, we are currently developing a methodology for evaluating a real situation, based on past occurrences. The aim is to develop a platform that will enable the evaluation of a risk scenario based on existing civil protection data. The methodology under development should enable the evaluation of different scenarios based on the collected available data. This will be achieved thanks to the facilitated configuration of several aspects, such as the geographical region and relevant properties of the considered threat. In this paper, we describe the methodology development process and the current state of the platform for risk evaluation.

Abstract: In 2014, the Swedish Civil Contingencies Agency (MSB) carried out a comprehensive crisis communication multi-sector exercise. To evaluate communication in exercises there is a need for involving the public as they are the primary target group in most crises. Systematically involving the public in evaluation of exercises has only been done a few times in Sweden and there are no known international attempts. We therefore developed a method where a representative sample of 2 000 persons was drawn from the Swedish population register and invited to assess communication during the exercise. The so-called ?Public Network? finally consisted of 99 people who contributed extensively with 395 assessment questionnaires. This paper describes the method, the results and experiences. We encourage other organizations to involve the public in exercises for evaluation purposes. The results show that there is a willingness among the public to contribute to the development of crisis preparedness.

Abstract: Agent-based simulations enable the simulation of social phenomenon by representing human behaviors using agents. Human actions such as evacuating to safe havens or extinguishing fires in disaster areas are important during earthquakes. The inclusion of human actions in calculating the damage at disaster sites provides useful data to local governments for planning purposes. In order to practically apply these simulation results, these results should be tested using actual data. Further, these results should be analyzed and explained in a manner that people who are not agent programmers can also understand easily. First, the possibility of applying agent-based approaches to social tasks is shown by comparing the simulation results with those obtained from other methods. Next, we propose a method to present agent behaviors using a probability model and discuss the results of applying this method to the RoboCup Rescue simulation data. These will delve into future research topics for developing agent based social simulations to practical ones.

Abstract: The current work describes and evaluates an early warning system for meteorological risk in Lisbon that has been functioning in SMPC since February 2008. The system aims to integrate multiple sources of information and facilitate cross checking observations, forecasts and warnings, allowing for an efficient and timely evaluation of the alert level to issue. Currently, it comprises hourly weather and tide level forecasts and automated warnings for Lisbon city, given by MM5 and WRF models running at IST. Results show MM5 performing better than WRF except for warm weather. The overall skill of the warning system is 40% with some false alarm ratios, mainly for forecasts with more than 3 days in advance. This is a reasonable characteristic for early warning since a potentially problematic situation can be anticipated and checked avoiding unnecessary economic expenditures if the warnings do not persist.

Abstract: One problem affecting crisis management planning teams is overconfidence- An inflated belief that a plan will be successful. In this paper we compared the effect of several different methods for reducing individual team member confidence levels and compared each to a baseline control condition. One hundred and seventy-eight people participated in one of five conditions to evaluate an H1N1 flu epidemic plan in a university context. Over the course of evaluating the plan, participants provided several ratings of confidence in the plan's success and their understanding. We compared several techniques commonly used, such as critique, Pro/Cons generation, Cons only generation and a newer technique, PreMortem, to a baseline condition. The Pro/Cons generation, Cons only generation and the PreMortem technique all reliably reduced confidence levels more than baseline condition. Furthermore, the Premortem method, imagining that a plan has failed and then generating reasons to explain why, reliably reduced confidence more than each of the other conditions, and therefore can be a useful tool for combating overconfidence in crisis management planning. We discuss the results in the context of sensemaking and decision making theory.

Abstract: Coordination deficiencies have been identified after the March 2011 earthquakes in Japan in terms of scheduling and allocation of resources, with time pressure, resource shortages, and especially informational uncertainty being main challenges. We suggest a decision support model that accounts for these challenges by drawing on fuzzy set theory and fuzzy optimization. Based on requirements from practice and the findings of our literature review, the decision model considers the following premises: incidents and rescue units are spatially distributed, rescue units possess specific capabilities, processing is non-preemptive, and informational uncertainty through linguistic assessments is predominant when on-site units vaguely report about incidents and their attributes, or system reports are not exact. We also suggest a Monte Carlo-based heuristic solution procedure and conduct a computational evaluation of different scenarios. We benchmark the results of our heuristic with results yielded through applying a greedy approach. The results indicate that using our Monte Carlo simulation to solve the decision support model inspired by fuzzy set theory can substantially reduce the overall harm. © 2012 ISCRAM.

Abstract: In Emergency Management and Business Continuity Planning, scenarios are a widely used tool. Existing scenario creation systems allow distributed groups to create scenarios together but have limited collaboration support. This study developed and evaluated a solution to provide various types of collaboration support around a knowledge structure at the core of a collaborative scenario creation system called Collario. Following the Design Science paradigm, it evolved through four iterations into a working prototype. Several evaluation methods, including protocol analysis and field study, were employed to evaluate the design effects and obtain user feedback. The results of the first field trial are described in this paper. They indicate that the system is useful to support creation and discussion of emergency scenarios in virtual teams and to share knowledge and experiences among geographically distributed emergency professionals and researchers. It was also found that the system is not hard to learn and use.