|
Schmidt-Colberg, A., & Löffler-Dauth, L. (2023). A Human-Centric Evaluation Dataset for Automated Early Wildfire Detection from a Causal Perspective. In Jaziar Radianti, Ioannis Dokas, Nicolas Lalone, & Deepak Khazanchi (Eds.), Proceedings of the 20th International ISCRAM Conference (pp. 933–943). Omaha, USA: University of Nebraska at Omaha.
Abstract: Insight into performance ability is crucial for successfully implementing AI solutions in real-world applications. Unanticipated input can lead to false positives (FP) and false negatives (FN), potentially resulting in false alarms in fire detection scenarios. Literature on fire detection models shows varying levels of complexity and explicability in evaluation practices; little supplementary information on performance ability outside of accuracy scores is provided. We advocate for a standardized evaluation dataset that prioritizes the end-user perspective in assessing performance capabilities. This leads us to ask what an evaluation dataset needs to constitute to enable a non-expert to determine the adequacy of a model's performance capabilities for their specific use case. We propose using data augmentation techniques that simulate interventions to remove the connection to the original target label, providing interpretable counterfactual explanations into a model's predictions.
|
|
|
Jan Maarten Schraagen, Aletta Eikelboom, Kees Van Dongen, & Guido Te Brake. (2005). Experimental evaluation of a critical thinking tool to support decision making in crisis situations. In B. C. B. Van de Walle (Ed.), Proceedings of ISCRAM 2005 – 2nd International Conference on Information Systems for Crisis Response and Management (pp. 181–189). Brussels: Royal Flemish Academy of Belgium.
Abstract: Building up proper situation awareness is one of the most difficult tasks in the beginning stages of large-scale accidents. As ambiguous information about the events becomes available, decision makers are often tempted to quickly choose a particular story to explain the events. Subsequent information that contradicts the initial story may easily be discarded and cognitive tunnel vision takes over. Our approach, as part of the COMBINED Systems project, is to prevent tunnel vision by providing critical thinking support. In a laboratory experiment with 60 participants, we tested this hypothesis by comparing the Critical Thinking tool with a 'no support' control condition and a 'minimal support' condition. Participants acted as crisis managers determining the likely cause of an accident based on different pieces of information. The results show a positive impact of the tool on both the decision making process and decision making effectiveness.
|
|
|
Axel Schulz, Tung Dang Thanh, Heiko Paulheim, & Immanuel Schweizer. (2013). A fine-grained sentiment analysis approach for detecting crisis related microposts. In J. Geldermann and T. Müller S. Fortier F. F. T. Comes (Ed.), ISCRAM 2013 Conference Proceedings – 10th International Conference on Information Systems for Crisis Response and Management (pp. 846–851). KIT; Baden-Baden: Karlsruher Institut fur Technologie.
Abstract: Real-time information from microposts like Twitter is useful for applications in the crisis management domain. Currently, that potentially valuable information remains mostly unused by the command staff, mainly because the sheer amount of information cannot be handled efficiently. Sentiment analysis has been shown as an effective tool to detect microposts (such as tweets) that contribute to situational awareness. However, current approaches only focus on two or three emotion classes. But using only tweets with negative emotions for crisis management is not always sufficient. The amount of remaining information is still not manageable or most of the tweets are irrelevant. Thus, a more fine-grained differentiation is needed to identify relevant microposts. In this paper, we show the systematic evaluation of an approach for sentiment analysis on microposts that allows detecting seven emotion classes. A preliminary evaluation of our approach in a crisis related scenario demonstrates the applicability and usefulness.
|
|
|
Seungwon Yang, Haeyong Chung, Xiao Lin, Sunshin Lee, Liangzhe Chen, Andrew Wood, et al. (2013). PhaseVis1: What, when, where, and who in visualizing the four phases of emergency management through the lens of social media. In J. Geldermann and T. Müller S. Fortier F. F. T. Comes (Ed.), ISCRAM 2013 Conference Proceedings – 10th International Conference on Information Systems for Crisis Response and Management (pp. 912–917). KIT; Baden-Baden: Karlsruher Institut fur Technologie.
Abstract: The Four Phase Model of Emergency Management has been widely used in developing emergency/disaster response plans. However, the model has received criticism contrasting the clear phase distinctions in the model with the complex and overlapping nature of phases indicated by empirical evidence. To investigate how phases actually occur, we designed PhaseVis based on visualization principles, and applied it to Hurricane Isaac tweet data. We trained three classification algorithms using the four phases as categories. The 10-fold cross-validation showed that Multi-class SVM performed the best in Precision (0.8) and Naïve Bayes Multinomial performed the best in F-1 score (0.782). The tweet volume in each category was visualized as a ThemeRiver[TM], which shows the 'What' aspect. Other aspects – 'When', 'Where', and 'Who' – Are also integrated. The classification evaluation and a sample use case indicate that PhaseVis has potential utility in disasters, aiding those investigating a large disaster tweet dataset.
|
|
|
André Simões, Armanda Rodrigues, Patricia Pires, & Luis Sá. (2011). Evaluating emergency scenarios using historic data: Flood management. In E. Portela L. S. M.A. Santos (Ed.), 8th International Conference on Information Systems for Crisis Response and Management: From Early-Warning Systems to Preparedness and Training, ISCRAM 2011. Lisbon: Information Systems for Crisis Response and Management, ISCRAM.
Abstract: The evaluation of an emergency scenario is often based on the use of simulation models. The specificity of these models involves the need for a complex evaluation of the problem domain, including the physical conditions behind the considered threat. Based on emergency occurrences data, provided by the Portuguese National Civil Protection Authority, we are currently developing a methodology for evaluating a real situation, based on past occurrences. The aim is to develop a platform that will enable the evaluation of a risk scenario based on existing civil protection data. The methodology under development should enable the evaluation of different scenarios based on the collected available data. This will be achieved thanks to the facilitated configuration of several aspects, such as the geographical region and relevant properties of the considered threat. In this paper, we describe the methodology development process and the current state of the platform for risk evaluation.
|
|
|
Marnix W.B. Eysink Smeets, & Simone Sillem. (2005). Intelligent SMS as an effective public warning system: The inspiring results of a dutch pilot project. In B. C. B. Van de Walle (Ed.), Proceedings of ISCRAM 2005 – 2nd International Conference on Information Systems for Crisis Response and Management (pp. 317–321). Brussels: Royal Flemish Academy of Belgium.
Abstract: The classic Dutch disaster-warning system (the 'siren') is not as effective as it should be. The alarm is not heard by 37% of the population, not all the people that hear the siren do take it seriously. A public-warning system was developed based on so-called 'intelligent SMS'. This system was tested in 2004 among 700 inhabitants of the city of Vlaardingen (region Rotterdam-Rijnmond). The University of Delft conducted an intensive evaluation. The evaluation shows that use of SMS is technically feasible. It diminishes the part of the population that is not reached is by approx. 50%. The public is now not only warned that 'something is going on' but is informed by SMS of the nature of the threat ánd on what to do. The public perceives intelligent SMS as the most effective warning system. Based on the pilot, the disaster-management authority of the region Rotterdam-Rijnmond, with some 1.2 million inhabitants, decided to structurally implement the system in the whole region in 2005.
|
|
|
Sofie Ivarsson. (2015). New method for evaluation of crisis communication in exercises ? involve the public. In L. Palen, M. Buscher, T. Comes, & A. Hughes (Eds.), ISCRAM 2015 Conference Proceedings ? 12th International Conference on Information Systems for Crisis Response and Management. Kristiansand, Norway: University of Agder (UiA).
Abstract: In 2014, the Swedish Civil Contingencies Agency (MSB) carried out a comprehensive crisis communication multi-sector exercise. To evaluate communication in exercises there is a need for involving the public as they are the primary target group in most crises. Systematically involving the public in evaluation of exercises has only been done a few times in Sweden and there are no known international attempts. We therefore developed a method where a representative sample of 2 000 persons was drawn from the Swedish population register and invited to assess communication during the exercise. The so-called ?Public Network? finally consisted of 99 people who contributed extensively with 395 assessment questionnaires. This paper describes the method, the results and experiences. We encourage other organizations to involve the public in exercises for evaluation purposes. The results show that there is a willingness among the public to contribute to the development of crisis preparedness.
|
|
|
Sofie Pilemalm, Jaziar Radianti, Bjørn Erik Munkvold, Tim A. Majchrzak, & Kristine Steen-Tveit. (2021). Turning Common Operational Picture Data into Double-loop Learning from Crises – can Vision Meet Reality? In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 417–430). Blacksburg, VA (USA): Virginia Tech.
Abstract: This study proposes a framework for double-loop learning from crises, using common operational pictures (COP). In most crises, a COP is of outmost importance to gain a common understanding among inter-organizational response. A COP is typically expressed through a map visualization. While the technologies to support COP progress rapidly, the corresponding practice of evaluating the COP and situational awareness is not yet established. Tools that enable responders to learn after the crisis, look back in time on the COP devel-opment and detect the barriers that prevent the COP establishment, still seem absent. Double-loop learning is an organizational practice to learn from previous actions widely adopted in the safety domain, and lately used in crisis management. This paper addresses the perceived gap by presenting the technical, organizational and structural requirements derived from document analysis, observation, and a workshop with multiple crisis management stakeholders, and integrating them to an initial framework.
|
|
|
Tomoichi Takahashi. (2007). Agent-based disaster simulation evaluation and its probability model interpretation. In K. Nieuwenhuis P. B. B. Van de Walle (Ed.), Intelligent Human Computer Systems for Crisis Response and Management, ISCRAM 2007 Academic Proceedings Papers (pp. 369–376). Delft: Information Systems for Crisis Response and Management, ISCRAM.
Abstract: Agent-based simulations enable the simulation of social phenomenon by representing human behaviors using agents. Human actions such as evacuating to safe havens or extinguishing fires in disaster areas are important during earthquakes. The inclusion of human actions in calculating the damage at disaster sites provides useful data to local governments for planning purposes. In order to practically apply these simulation results, these results should be tested using actual data. Further, these results should be analyzed and explained in a manner that people who are not agent programmers can also understand easily. First, the possibility of applying agent-based approaches to social tasks is shown by comparing the simulation results with those obtained from other methods. Next, we propose a method to present agent behaviors using a probability model and discuss the results of applying this method to the RoboCup Rescue simulation data. These will delve into future research topics for developing agent based social simulations to practical ones.
|
|
|
Toshihisa Toyoda. (2008). Economic impacts of Kobe Earthquake: A quantitative evaluation after 13 years. In B. V. de W. F. Fiedrich (Ed.), Proceedings of ISCRAM 2008 – 5th International Conference on Information Systems for Crisis Response and Management (pp. 606–617). Washington, DC: Information Systems for Crisis Response and Management, ISCRAM.
Abstract: The importance of distinguishing between direct and indirect losses of disasters is stressed. In order to estimate indirect losses, a conceptual framework of direct and indirect losses is presented. For the case of the Great Hanshin-Awaji (Kobe) Earthquake of 1995, direct stock losses of both the manufacturing and the commercial sectors record almost same size of big damage. As for indirect flow losses, the commercial and the other services sectors show far greater damage than the manufacturing sector. A careful statistical analysis of indirect losses using the gross regional product in the stricken area presents a new finding that the lost product and income in terms of estimated indirect losses are quite large and continue to arise for longer than 10 years, mounting to some 14 trillion yen (about US$0.13 trillion). Disaster management policy should be improved by paying attention not only to direct losses but also to indirect losses.
|
|
|
Ana Rosa Trancoso, José Delgado Domingos, Maria João Telhado, & João Corte-Real. (2011). Early warning system for meteorological risk in Lisbon municipality: Description and quality evaluation. In E. Portela L. S. M.A. Santos (Ed.), 8th International Conference on Information Systems for Crisis Response and Management: From Early-Warning Systems to Preparedness and Training, ISCRAM 2011. Lisbon: Information Systems for Crisis Response and Management, ISCRAM.
Abstract: The current work describes and evaluates an early warning system for meteorological risk in Lisbon that has been functioning in SMPC since February 2008. The system aims to integrate multiple sources of information and facilitate cross checking observations, forecasts and warnings, allowing for an efficient and timely evaluation of the alert level to issue. Currently, it comprises hourly weather and tide level forecasts and automated warnings for Lisbon city, given by MM5 and WRF models running at IST. Results show MM5 performing better than WRF except for warm weather. The overall skill of the warning system is 40% with some false alarm ratios, mainly for forecasts with more than 3 days in advance. This is a reasonable characteristic for early warning since a potentially problematic situation can be anticipated and checked avoiding unnecessary economic expenditures if the warnings do not persist.
|
|
|
Richelle Van Rijk, & Marcel Van Berlo. (2004). Using CrisisKit and MOPED to improve emergency management team training. In B. C. B. Van de Walle (Ed.), Proceedings of ISCRAM 2004 – 1st International Workshop on Information Systems for Crisis Response and Management (pp. 161–166). Brussels: Royal Flemish Academy of Belgium.
Abstract: In order to reduce the effects of a disaster, people in the emergency management organization have to be trained. In recent years training emergency management teams has become a bigger issue. A realistic and effective training of emergency management teams however is a difficult matter. We search for ways to improve this kind of training and to reduce the costs. In this paper two tools that can be used to improve emergency management training, CrisisKit and MOPED, will be discussed. © Proceedings ISCRAM 2004.
|
|
|
Beth Veinott, Gary L. Klein, & Sterling Wiggins. (2010). Evaluating the effectiveness of the PreMortem technique on plan confidence. In C. Zobel B. T. S. French (Ed.), ISCRAM 2010 – 7th International Conference on Information Systems for Crisis Response and Management: Defining Crisis Management 3.0, Proceedings. Seattle, WA: Information Systems for Crisis Response and Management, ISCRAM.
Abstract: One problem affecting crisis management planning teams is overconfidence- An inflated belief that a plan will be successful. In this paper we compared the effect of several different methods for reducing individual team member confidence levels and compared each to a baseline control condition. One hundred and seventy-eight people participated in one of five conditions to evaluate an H1N1 flu epidemic plan in a university context. Over the course of evaluating the plan, participants provided several ratings of confidence in the plan's success and their understanding. We compared several techniques commonly used, such as critique, Pro/Cons generation, Cons only generation and a newer technique, PreMortem, to a baseline condition. The Pro/Cons generation, Cons only generation and the PreMortem technique all reliably reduced confidence levels more than baseline condition. Furthermore, the Premortem method, imagining that a plan has failed and then generating reasons to explain why, reliably reduced confidence more than each of the other conditions, and therefore can be a useful tool for combating overconfidence in crisis management planning. We discuss the results in the context of sensemaking and decision making theory.
|
|
|
Vittorio Nespeca, Kenny Meesters, & Tina Comes. (2018). Evaluating Platforms for Community Sense-making: Using the Case of the Kenyan Elections. In Kees Boersma, & Brian Tomaszeski (Eds.), ISCRAM 2018 Conference Proceedings – 15th International Conference on Information Systems for Crisis Response and Management (pp. 924–934). Rochester, NY (USA): Rochester Institute of Technology.
Abstract: The profusion of information technology has created new possibilities for local communities to self-organize and respond to disruptive events. Along with the opportunities, there is also a series of challenges that need to be addressed in order to improve societal resilience. One of these challenges is to make sense of the continuous stream of information to create a coherent understanding and improve coordination. The research presented in this paper focuses on the socio-technical requirements of IT platforms that support sense-making and coordination. Using a comprehensive evaluation exercise based on real data from the 2017 Kenyan elections, we examine the development, workflows and use of this shared situational awareness in a group decision making process. In this manner, we identify requirements for resilience platforms and identify further research directions.
|
|
|
Felix Wex, Guido Schryen, & Dirk Neumann. (2012). Operational emergency response under informational uncertainty: A fuzzy optimization model for scheduling and allocating rescue units. In Z.Franco J. R. L. Rothkrantz (Ed.), ISCRAM 2012 Conference Proceedings – 9th International Conference on Information Systems for Crisis Response and Management. Vancouver, BC: Simon Fraser University.
Abstract: Coordination deficiencies have been identified after the March 2011 earthquakes in Japan in terms of scheduling and allocation of resources, with time pressure, resource shortages, and especially informational uncertainty being main challenges. We suggest a decision support model that accounts for these challenges by drawing on fuzzy set theory and fuzzy optimization. Based on requirements from practice and the findings of our literature review, the decision model considers the following premises: incidents and rescue units are spatially distributed, rescue units possess specific capabilities, processing is non-preemptive, and informational uncertainty through linguistic assessments is predominant when on-site units vaguely report about incidents and their attributes, or system reports are not exact. We also suggest a Monte Carlo-based heuristic solution procedure and conduct a computational evaluation of different scenarios. We benchmark the results of our heuristic with results yielded through applying a greedy approach. The results indicate that using our Monte Carlo simulation to solve the decision support model inspired by fuzzy set theory can substantially reduce the overall harm. © 2012 ISCRAM.
|
|
|
Yangyang Meng, Xiangliang Tian, Chang Liu, Zhongwen Li, Zhijie Zhou, & Maohua Zhong. (2020). Research on Emergency Capability Evaluation of Network Operation-Based Urban Rail Transit. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 530–544). Blacksburg, VA (USA): Virginia Tech.
Abstract: In the process of network operation, it is of great significance to evaluate the emergency capability for the safety and resilience of urban rail transit. In this work, we proposed an emergency capability evaluation model of network operation-based urban rail transit by building a four-level index system. AHP (Analytic Hierarchy Process) method demonstrated the effectiveness of the evaluation index system. The ranking of index importance Wi characterizes the emergency capability of network operation-based urban rail transit. Taking Shenzhen Metro as an example, this study analyzed the risk in the actual network operation, evaluated the emergency capability of network operation and calculated the comprehensive score of emergency capability. Furthermore, based on the correlation analysis results from the emergency capability indexes, we put forward some measures to improve the weaker indexes in the evaluation. The results indicate that the emergency capability evaluation method of network operation-based urban rail transit proposed in this study can better guide the emergency management of network operation-based urban rail transit.
|
|
|
Xiang Yao, Murray Turoff, & Starr Roxanne Hiltz. (2010). A field trial of a collaborative online scenario creation system for emergency management. In C. Zobel B. T. S. French (Ed.), ISCRAM 2010 – 7th International Conference on Information Systems for Crisis Response and Management: Defining Crisis Management 3.0, Proceedings. Seattle, WA: Information Systems for Crisis Response and Management, ISCRAM.
Abstract: In Emergency Management and Business Continuity Planning, scenarios are a widely used tool. Existing scenario creation systems allow distributed groups to create scenarios together but have limited collaboration support. This study developed and evaluated a solution to provide various types of collaboration support around a knowledge structure at the core of a collaborative scenario creation system called Collario. Following the Design Science paradigm, it evolved through four iterations into a working prototype. Several evaluation methods, including protocol analysis and field study, were employed to evaluate the design effects and obtain user feedback. The results of the first field trial are described in this paper. They indicate that the system is useful to support creation and discussion of emergency scenarios in virtual teams and to share knowledge and experiences among geographically distributed emergency professionals and researchers. It was also found that the system is not hard to learn and use.
|
|
|
Hans Zimmermann. (2005). Recent developments in emergency telecommunications. In B. C. B. Van de Walle (Ed.), Proceedings of ISCRAM 2005 – 2nd International Conference on Information Systems for Crisis Response and Management (pp. 327–334). Brussels: Royal Flemish Academy of Belgium.
Abstract: Recent Developments in the telecommunication technology offer a number of additional tools not only for the providers of emergency and disaster response, but may also facilitate early warning. Their application depends, however, on the regulatory framework governing telecommunications and on their appropriateness for the specific requirements in the pre-, peri- and postimpact phases of an event. Telecommunications are a key element for to the success of emergency preparedness and response, and the application of all available technologies and networks saves lives. The recent events in the Indian Ocean Region will have to be the subject of detailed analysis and evaluation of all existing mechanisms, hopefully leading to improvements in the organizational and operational field. First information available already in the immediate aftermath of the events of 26 December 2004 has been applied to the considerations in the present paper.
|
|