Kuntke, F., Bektas, M., Buhleier, L., Pohl, E., Schiller, R., & Reuter, C. (2023). How Would Emergency Communication Based On LoRaWAN Perform? Empirical Findings of Signal Propagation in Rural Areas. In Jaziar Radianti, Ioannis Dokas, Nicolas Lalone, & Deepak Khazanchi (Eds.), Proceedings of the 20th International ISCRAM Conference (pp. 1042–1050). Omaha, USA: University of Nebraska at Omaha.
Abstract: Low Power Wide Area Network (LPWAN) technologies are typically promoted for Internet-of-Things (IoT) applications, but are also of interest for emergency communications systems when regular fixed and mobile networks break down. Although LoRaWAN is a frequently used representative here, there are sometimes large differences between the proposed range and the results of some practical evaluations. Since previous work has focused on urban environments or has conducted simulations, this work aims to gather concrete knowledge on the transmission characteristics in rural environments. Extensive field studies with varying geographic conditions and comparative tests in urban environments were performed using two different hardware implementations. Overall, it was found that the collected values in rural areas are significantly lower than the theoretical values. Nevertheless, the results certify that LoRaWAN technology has a high range that cannot be achieved with other common technologies for emergency communications.
|
|
Henke, S., Widera, A., & Hellingrath, B. (2023). Evaluation-driven Disaster Management Exercises: A Collaborative Toolkit. In Jaziar Radianti, Ioannis Dokas, Nicolas Lalone, & Deepak Khazanchi (Eds.), Proceedings of the 20th International ISCRAM Conference (pp. 989–999). Omaha, USA: University of Nebraska at Omaha.
Abstract: Disaster management exercises are a core component of humanitarian organizations’ preparedness strategies. They entail diverse purposes, from training capabilities of participants to testing response plans to enhancing collaboration between organizations and many more. However, it is uncertain how much exercises contribute to preparedness. Rigorous evaluation is needed to exploit learning opportunities of an exercise. Therefore, exercises must target evaluable objectives, which is complicated by the socio-technical openness of the exercise system, the heterogeneity of organizational needs, and the scarcity of resources. Many different tools aim to support evaluation but are limited to specific use cases, resulting in a fragmented overview for practitioners. Due to the excessive effort involved, practitioners often consider exercise evaluation to be of secondary importance. This study thus proposes the conceptual design of a combined toolkit that supports the practitioners in a more rigorous but resource-efficient evaluation to make disaster management exercises more evaluation-driven.
|
|
Schmidt-Colberg, A., & Löffler-Dauth, L. (2023). A Human-Centric Evaluation Dataset for Automated Early Wildfire Detection from a Causal Perspective. In Jaziar Radianti, Ioannis Dokas, Nicolas Lalone, & Deepak Khazanchi (Eds.), Proceedings of the 20th International ISCRAM Conference (pp. 933–943). Omaha, USA: University of Nebraska at Omaha.
Abstract: Insight into performance ability is crucial for successfully implementing AI solutions in real-world applications. Unanticipated input can lead to false positives (FP) and false negatives (FN), potentially resulting in false alarms in fire detection scenarios. Literature on fire detection models shows varying levels of complexity and explicability in evaluation practices; little supplementary information on performance ability outside of accuracy scores is provided. We advocate for a standardized evaluation dataset that prioritizes the end-user perspective in assessing performance capabilities. This leads us to ask what an evaluation dataset needs to constitute to enable a non-expert to determine the adequacy of a model's performance capabilities for their specific use case. We propose using data augmentation techniques that simulate interventions to remove the connection to the original target label, providing interpretable counterfactual explanations into a model's predictions.
|
|
Landsberg, L., Gleibs, T., & Mudimu, O. A. (2023). Design of a Systems Theory Approach for the Evaluation of C2-Systems. In Jaziar Radianti, Ioannis Dokas, Nicolas Lalone, & Deepak Khazanchi (Eds.), Proceedings of the 20th International ISCRAM Conference (pp. 692–700). Omaha, USA: University of Nebraska at Omaha.
Abstract: The course of large-scale incidents as well as disasters can reveal weaknesses in command and control (C2) systems, which make adjustments necessary. Also, new technologies may require C2-systems to be adapted to achieve their full potential for improving incident command. This paper deals with an approach to enable the comparison and evaluation of different C2-systems or their adaptations in order to find the best possible customizations for C2-systems. To this purpose, systems theory is used to unify the approaches of different research disciplines. Within the C2-system boundaries, distinctions were made to represent three different levels of evaluation: “Physical Characteristics”, “Structures and Processes” as well as “C2-system-effectiveness”. During the implementation of the evaluation methods from the different research disciplines into the systems theory approach, it became apparent that the comprehensive approach is desirable, but that broad knowledge and expertise is necessary, especially at the highest evaluation level “C2-system-effectiveness”.
|
|
Radianti, J., Pilemalm, S., Steen-Tveit, K., & Rustenberg, K. (2023). Enhancing Learning from Incidents by Reconstruction of Events: Using the SQUARE Tool for Evaluation. In Jaziar Radianti, Ioannis Dokas, Nicolas Lalone, & Deepak Khazanchi (Eds.), Proceedings of the 20th International ISCRAM Conference (pp. 663–675). Omaha, USA: University of Nebraska at Omaha.
Abstract: Evaluation after exercise and incidents—often called After Action Review or AAR—are important to enhance the emergency management (EM) response capability. However, evaluation supporttools for event reconstruction after incidents and exercises are not yet fully available. We engaged EM stakeholders in a tabletop exercise based on snow chaos and car accidents scenarios to evaluate a fully functional prototype. The tool assists information sharing in real-time and enhances debriefing process of any EM response and exercise. Analysis of reflective discussions and an open question survey shows that the EM participants consider the features of the prototype to facilitate better learning from incidents. However, they have different attitudes concerning the adoption, management, and application of the tool in organizations. For instance, more security features are required to comply with regulations. We argue that the tool is an important first step to fill the gap on the need for “event reconstruction-based evaluation.”
|
|