Abstract: Evaluation and testing are major steps in the development of any information system, particularly if it is to be used in high-risk contexts such as conflicts. While thus far there are various approaches for testing against technology requirements; usability or usefulness, there is a lack of a comprehensive evaluation framework that combines the three elements. The lack of such a framework and commonly agreed standards constitutes a barrier for innovation, and at the same time imposes risks to responders if the technology is introduced without proper testing. This paper aims to close this gap. Based on a reviewing of evaluation methods and measurement metrics, we design a comprehensive evaluation framework including common code quality testing metrics, usability testing methods, subjective usefulness questionnaires, and performance indicators. We demonstrate our approach by using the example of an integrated system for the safety and security of humanitarian missions, and we highlight how our approach allows measuring the system?s quality and usefulness.

Abstract: Computer and virtual simulation-based training (CST) offer several benefits for emergency response and management preparedness. However, organizations responsible for training are often hesitant to use CST, based on cost and perceived lack of benefit when compared to live simulation training (LST). This paper investigates how CST can complement LST, and how it contributes to achieving the necessary learning objectives for level one fire and rescue service incident commanders (ICs). Data and examples come from an experimental study with students from different fire and rescue services trained in the role of the IC in LST and CST, in a similar scenario. Results show the cost and benefits of the CST implementation based on evaluations from learners, instructors and responsible managers. Participants had a positive attitude towards using virtual simulations, but the results also point to barriers regarding the suitable design of learning scenarios and implementation.

Abstract: Mass casualty incidents (MCI) are highly dynamic situations in which limited available resources need to be quickly and efficiently allocated. In this paper, we suggest considerable extensions to an allocation method that we presented in earlier work. The extensions address two major challenges: First, the need to balance real-world resource usage and second, the need to adapt to changing situations. Additionally, a theoretical evaluation of the efficiency of the suggested approach is described. © 2012 ISCRAM.

Abstract: Artesis Virtual and VSTEP BV developed a virtual scenario training for leading firefighters. We evaluated the usability of the training application and measured the satisfaction of 53 trainees. The trainees were positive about the training application, the training itself, and the interaction with the interface. We observed that trainees experienced navigation difficulties and sometimes misjudged visual information. Evaluations of new versions of this application, should address the influence of interface problems on performance.

Abstract: Although new technology may benefit rural first responders to help them serve their communities, to date little is known about what communication technology problems rural first responders most need addressed and what future technology they desire. To explore the context of use and communication technology problems and needs of rural first responders, semi-structured interviews were conducted with 63 rural first responders across four disciplines: Communications (Comm) Center & 9-1-1 Services, Emergency Medical Services, Fire Service, and Law Enforcement. Using qualitative data analysis, interview data were sorted into problems and needs categories. Rural first responders' greatest problems were with reliable coverage/connectivity, interoperability, implementation/information technology (IT) infrastructure, and physical ergonomics. Rural first responders' greatest need for new technology was to address their current problems, but they were interested in new technology that leverages real-time technology and location tracking. Implications for researchers and developers of public safety communication technology are discussed.

Abstract: Social media platforms contain abundant data that can provide comprehensive knowledge of historical and real-time events. During crisis events, the use of social media peaks, as people discuss what they have seen, heard, or felt. Previous studies confirm the usefulness of such socially generated discussions for the public, first responders, and decision-makers to gain a better understanding of events as they unfold at the ground level. This study performs an extensive analysis of COVID-19-related Twitter discussions generated in Australia between January 2020, and October 2022. We explore the Australian Twitterverse by employing state-of-the-art approaches from both supervised and unsupervised domains to perform network analysis, topic modeling, sentiment analysis, and causality analysis. As the presented results provide a comprehensive understanding of the Australian Twitterverse during the COVID-19 pandemic, this study aims to explore the discussion dynamics to aid the development of future automated information systems for epidemic/pandemic management.

Abstract: Multiple disaster mobile applications (apps) already exist for public use; however, availability does not automatically translate to continued usage. Limited research has explored whether disaster apps are usable and whether the apps' usability affects users' intent for continued use. The paper presents a work-in-progress study that aims to test a usability-continuance intention model for the specific context of disaster apps. The study theorises seven usability factors that influence continued intention to use. An online usability survey was used to gather user experience data on disaster apps. Initial findings, through structural equational modelling, showed that five of the seven usability factors have a significant relationship to continuance intention. Although the relationships have different weights and directions, key influencers to users' intent to continue usage are app utility, app dependability, interface output, interface input, and interface graphics. The next step of the study will investigate the mediating effects of the factors and the moderating effects of users' experience and technological comfort.

Abstract: In the safety- And time-critical context of pre-hospital medical care, usability is a major challenge that needs to be addressed, because interaction problems or errors may have serious consequences for patients as well as employees of Emergency Medical Services (EMS). In this regard, a user-centered human-computer interface is a crucial factor for efficient usage as well as user acceptance. Well-established design principles can serve as directives throughout the analysis, design, implementation, and evaluation of applications. For standard PCbased applications the ISO standard 9241-110 serves as a practical guide since many years. However, new types of devices, especially mobile systems impose new challenges and ways of interpreting usability standards and guidelines. In this paper, we apply the ISO-Standard 9241-110 dialogue principles to mobile applications in EMS, considering the challenges of designing a system suitable for supporting users in both regular day-to-day services as well as rare Mass Casualty Incidents (MCIs).

Abstract: Today's decision support systems for crisis management are mostly designed to support a fixed process that integrates a given set of information sources. This means policies that govern the crisis management process are tightly integrated with the implementation, which makes it hard to adapt the system to changing requirements. Modern systems are expected to be adaptable and need to evolve along with the availability of new information sources and changing business processes. Previous work suggested using workflow systems to manage crisis management processes. Current approaches that use workflow systems are not end-user friendly or not flexible enough. In this paper we present our approach that combines workflows and decision tables for creating more flexible decision support systems. While workflows are used to orchestrate services and implement information logistics in the decision support processes, embedded rule sets are used to provide flexibility and adaptability of workflows. The rule sets are authored using decision tables which are an easy-to-use representation that allows end-users to express rules in an intuitive way. © 2012 ISCRAM.

Abstract: Insight into performance ability is crucial for successfully implementing AI solutions in real-world applications. Unanticipated input can lead to false positives (FP) and false negatives (FN), potentially resulting in false alarms in fire detection scenarios. Literature on fire detection models shows varying levels of complexity and explicability in evaluation practices; little supplementary information on performance ability outside of accuracy scores is provided. We advocate for a standardized evaluation dataset that prioritizes the end-user perspective in assessing performance capabilities. This leads us to ask what an evaluation dataset needs to constitute to enable a non-expert to determine the adequacy of a model's performance capabilities for their specific use case. We propose using data augmentation techniques that simulate interventions to remove the connection to the original target label, providing interpretable counterfactual explanations into a model's predictions.

Abstract: Crisis and disaster management are increasingly characterized by interactive systems intended to be valuable support for professionals and volunteers in preventing, preparing for, responding to and recovering from major incidents and accidents. Therefore, usability in terms of safe and efficient usage of computer-based solutions becomes a crucial factor for successful crisis and disaster management. In order to ensure usability, it has to be addressed systematically throughout any development process. In this paper, established engineering approaches to crisis and disaster management systems are summarized. Subsequently, resemblances (e.g. diversity of users and devices) and differences (e.g. scalability) between safety-critical contexts of medical device design and crisis management are outlined. Following this, recommendations for applying usability engineering processes to disaster management are derived from standards and guidelines according to medical device design (IEC 62366-1:2015, ISO 14971:2007). Particularly, relationships and interactions between usability engineering and risk managements measures (e.g. hazard-related use scenarios) are described.