Anna Kruspe. (2020). Detecting Novelty in Social Media Messages During Emerging Crisis Events. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 860–871). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media can be a highly valuable source of information during disasters. A crisis' development over time is of particular interest here, as social media messages can convey unfolding events in near-real time. Previous approaches for the automatic detection of information in such messages have focused on a static analysis, not taking temporal changes and already-known information into account. In this paper, we present a novel method for detecting new topics in incoming Twitter messages (tweets) conditional upon previously found related tweets. We do this by first extracting latent representations of each tweet using pre-trained sentence embedding models. Then, Infinite Mixture modeling is used to dynamically cluster these embeddings anew with each incoming tweet. Once a cluster reaches a minimum number of members, it is considered to be a new topic. We validate our approach on the TREC Incident Streams 2019A data set.
|
Antone Evans Jr., Yingyuan Yang, & Sunshin Lee. (2021). Towards Predicting COVID-19 Trends: Feature Engineering on Social Media Responses. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 792–807). Blacksburg, VA (USA): Virginia Tech.
Abstract: During the course of this pandemic, the use of social media and virtual networks has been at an all-time high. Individuals have used social media to express their thoughts on matters related to this pandemic. It is difficult to predict current trends based on historic case data because trends are more connected to social activities which can lead to the spread of coronavirus. So, it's important for us to derive meaningful information from social media as it is widely used. Therefore, we grouped tweets by common keywords, found correlations between keywords and daily COVID-19 statistics and built predictive modeling. The features correlation analysis was very effective, so trends were predicted very well. A RMSE score of 0.0425504, MAE of 0.03295105 and RSQ of 0.5237014 in relation to daily deaths. In addition, we found a RMSE score of 0.07346836, MAE of 0.0491152 and RSQ 0.374529 in relation to daily cases.
|
Apoorva Chauhan, & Amanda Hughes. (2021). COVID-19 Named Resources on Facebook, Twitter, and Reddit. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 679–690). Blacksburg, VA (USA): Virginia Tech.
Abstract: Crisis Named Resources (CNRs) are social media accounts and pages named after a crisis event. They are created soon after an event occurs. CNRs share a lot of information around an event and are followed by many. In this study, we identify CNRs created around COVID-19 on Facebook, Twitter, and Reddit. We analyze when these resources were created, why they were created, how they were received by members of the public, and who created them. We conclude by comparing CNRs created around COVID-19 with past crisis events and discuss how CNR owners attempt to manage content and combat misinformation.
|
Cody Buntain, Richard Mccreadie, & Ian Soboroff. (2021). Incident Streams 2020: TRECIS in the Time of COVID-19. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 621–639). Blacksburg, VA (USA): Virginia Tech.
Abstract: Between 2018 and 2019, the Incident Streams track (TREC-IS) has developed standard approaches for classifying the types and criticality of information shared in online social spaces during crises, but the introduction of SARS-CoV-2 has shifted the landscape of online crises substantially. While prior editions of TREC-IS have lacked data on large-scale public-health emergencies as these events are exceedingly rare, COVID-19 has introduced an over-abundance of potential data, and significant open questions remain about how existing approaches to crisis informatics and datasets built on other emergencies adapt to this new context. This paper describes how the 2020 edition of TREC-IS has addressed these dual issues by introducing a new COVID-19-specific task for evaluating generalization of existing COVID-19 annotation and system performance to this new context, applied to 11 regions across the globe. TREC-IS has also continued expanding its set of target crises, adding 29 new events and expanding the collection of event types to include explosions, fires, and general storms, making for a total of 9 event types in addition to the new COVID-19 events. Across these events, TREC-IS has made available 478,110 COVID-related messages and 282,444 crisis-related messages for participant systems to analyze, of which 14,835 COVID-related and 19,784 crisis-related messages have been manually annotated. Analyses of these new datasets and participant systems demonstrate first that both the distributions of information type and priority of information vary between general crises and COVID-19-related discussion. Secondly, despite these differences, results suggest leveraging general crisis data in the COVID-19 context improves performance over baselines. Using these results, we provide guidance on which information types appear most consistent between general crises and COVID-19.
|
Congcong Wang, Paul Nulty, & David Lillis. (2021). Crisis Domain Adaptation Using Sequence-to-Sequence Transformers. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 655–666). Blacksburg, VA (USA): Virginia Tech.
Abstract: User-generated content (UGC) on social media can act as a key source of information for emergency responders incrisis situations. However, due to the volume concerned, computational techniques are needed to effectively filter and prioritise this content as it arises during emerging events. In the literature, these techniques are trained using annotated content from previous crises. In this paper, we investigate how this prior knowledge can be best leveraged for new crises by examining the extent to which crisis events of a similar type are more suitable for adaptation tonew events (cross-domain adaptation). Given the recent successes of transformers in various language processing tasks, we propose CAST: an approach for Crisis domain Adaptation leveraging Sequence-to-sequence Transformers. We evaluate CAST using two major crisis-related message classification datasets. Our experiments show that ourCAST-based best run without using any target data achieves the state of the art performance in both in-domain and cross-domain contexts. Moreover, CAST is particularly effective in one-to-one cross-domain adaptation when trained with a larger language model. In many-to-one adaptation where multiple crises are jointly used as the source domain, CAST further improves its performance. In addition, we find that more similar events are more likely to bring better adaptation performance whereas fine-tuning using dissimilar events does not help for adaptation. To aid reproducibility, we open source our code to the community.
|
Congcong Wang, Paul Nulty, & David Lillis. (2021). Transformer-based Multi-task Learning for Disaster Tweet Categorisation. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 705–718). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media has enabled people to circulate information in a timely fashion, thus motivating people to post messages seeking help during crisis situations. These messages can contribute to the situational awareness of emergency responders, who have a need for them to be categorised according to information types (i.e. the type of aid services the messages are requesting). We introduce a transformer-based multi-task learning (MTL) technique for classifying information types and estimating the priority of these messages. We evaluate the effectiveness of our approach with a variety of metrics by submitting runs to the TREC Incident Streams (IS) track: a research initiative specifically designed for disaster tweet classification and prioritisation. The results demonstrate that our approach achieves competitive performance in most metrics as compared to other participating runs. Subsequently, we find that an ensemble approach combining disparate transformer encoders within our approach helps to improve the overall effectiveness to a significant extent, achieving state-of-the-art performance in almost every metric. We make the code publicly available so that our work can be reproduced and used as a baseline for the community for future work in this domain.
|
Dilini Rajapaksha, Kacper Sokol, Jeffrey Chan, Flora Salim, Mukesh Prasad, & Mahendra Samarawickrama. (2023). Analysing Donors’ Behaviour in Non-profit Organisations for Disaster Resilience. In V. L. Thomas J. Huggins (Ed.), Proceedings of the ISCRAM Asia Pacific Conference 2022 (pp. 258–267). Palmerston North, New Zealand: Massey Unversity.
Abstract: With the advancement and proliferation of technology, non-profit organisations have embraced social media platforms to improve their operational capabilities through brand advocacy, among many other strategies. The effect of such social media campaigns on these institutions, however, remains largely underexplored, especially during disaster periods. This work introduces and applies a quantitative investigative framework to understand how social media influence the behaviour of donors and their usage of these platforms throughout (natural) disasters. More specifically, we explore how on-line engagement – as captured by Facebook interactions and Google search trends – corresponds to the donors’ behaviour during the catastrophic 2019–2020 Australian bushfire season. To discover this relationship, we analyse the record of donations made to the Australian Red Cross throughout this period. Our exploratory study reveals that social media campaigns are effective in encouraging on-line donations made via a dedicated website. We also compare this mode of giving to more regular, direct deposit gifting.
|
Ferda Ofli, Firoj Alam, & Muhammad Imran. (2020). Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 802–811). Blacksburg, VA (USA): Virginia Tech.
Abstract: Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning techniques. Specifically, we utilize convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation. Extensive experiments on real-world disaster datasets show that the proposed multimodal architecture yields better performance than models trained using a single modality (e.g., either text or image).
|
Guillermo Romera Rodriguez. (2023). Parler, Capitol Riots, Alt-Right and Radicalization in Social Media. In V. L. Thomas J. Huggins (Ed.), Proceedings of the ISCRAM Asia Pacific Conference 2022 (pp. 268–277). Palmerston North, New Zealand: Massey Unversity.
Abstract: Social media platforms have risen in popularity since their inception. These platforms have since then come to be at the forefront of controversies, from being accused of election interference to, more recently, disseminating fake news and campaigns to sway political behavior. One such episode took place on January 6 when a group of individuals stormed the United States Capitol, and the social media platform Parler came under scrutiny. The platform was accused of being a place for right-wing extremists and Trump supporters who claimed the 2020 election was fraudulent. Initial reports suggested these individuals used Parler to organize and call others to action. This paper explores the feasibility of using social media to detect alt-right radicalization and examines its possible relation to the Capitol Insurrection and Parler. Moreover, we examine if those events could have been detected and averted through the investigation of the platform.
|
Haiyan Hao, & Yan Wang. (2020). Hurricane Damage Assessment with Multi-, Crowd-Sourced Image Data: A Case Study of Hurricane Irma in the City of Miami. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 825–837). Blacksburg, VA (USA): Virginia Tech.
Abstract: The massive crowdsourced data generated on social networking platforms (e.g. Twitter and Flickr) provide free, real-time data for damage assessment (DA) even during catastrophes. Recent studies leveraging crowdsourced data for DA mainly focused on analyzing textual formats. Crowdsourced images can provide rich and objective information about damage conditions, however, are rarely researched for DA purposes. The highly-varied content and loosely-defined damage forms make it difficult to process and analyze the crowdsourced images. To address this problem, we propose a data-driven DA method based on multi-, crowd-sourced images, which includes five machine learning classifiers organized in a hierarchical structure. The method is validated with a case study investigating the damage condition of the City of Miami caused by Hurricane Irma. The outcome is then compared with a metric derived from NFIP insurance claims data. The proposed method offers a resource for rapid DA that supplements conventional DA methods.
|
Hannah Van Wyk, & Kate Starbird. (2020). Analyzing Social Media Data to Understand How Disaster-Affected Individuals Adapt to Disaster-Related Telecommunications Disruptions. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 704–717). Blacksburg, VA (USA): Virginia Tech.
Abstract: Information is a critical need during disasters such as hurricanes. Increasingly, people are relying upon cellular and internet-based technology to communicate that information--modalities that are acutely vulnerable to the disruptions to telecommunication infrastructure that are common during disasters. Focusing on Hurricane Maria (2017) and its long-term impacts on Puerto Rico, this research examines how people affected by severe and sustained disruptions to telecommunications services adapt to those disruptions. Leveraging social media trace data as a window into the real-time activities of people who were actively adapting, we use a primarily qualitative approach to identify and characterize how people changed their telecommunications practices and routines--and especially how they changed their locations--to access Wi-Fi and cellular service in the weeks and months after the hurricane. These findings have implications for researchers seeking to better understand human responses to disasters and responders seeking to identify strategies to support affected populations.
|
Hongmin Li, Doina Caragea, & Cornelia Caragea. (2021). Combining Self-training with Deep Learning for Disaster Tweet Classification. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 719–730). Blacksburg, VA (USA): Virginia Tech.
Abstract: Significant progress has been made towards automated classification of disaster or crisis related tweets using machine learning approaches. Deep learning models, such as Convolutional Neural Networks (CNN), domain adaptation approaches based on self-training, and approaches based on pre-trained language models, such as BERT, have been proposed and used independently for disaster tweet classification. In this paper, we propose to combine self-training with CNN and BERT models, respectively, to improve the performance on the task of identifying crisis related tweets in a target disaster where labeled data is assumed to be unavailable, while unlabeled data is available. We evaluate the resulting self-training models on three crisis tweet collections and find that: 1) the pre-trained language model BERTweet is better than the standard BERT model, when fine-tuned for downstream crisis tweets classification; 2) self-training can help improve the performance of the CNN and BERTweet models for larger unlabeled target datasets, but not for smaller datasets.
|
James A. Reep, & Andrea Tapia. (2020). Toward an Organizational Technology Adoption Process (OTAP) for Social Media Integration in a PSAP. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 718–729). Blacksburg, VA (USA): Virginia Tech.
Abstract: Integration of social media in emergency response environments presents specific organizational challenges, such as lack of resources or information credibility. Additionally, there exists individual resistance to change in these environments that could potentially discourage adoption. To identify and understand these challenges, we conducted semi-structured group interviews with emergency call takers and dispatchers. We find that these PSAP operators desire participation and explanation of changes throughout the organizational change process. Participants also articulated they desired training regarding change even when not directly affected. Though change management procedures often call for these strategies, they are commonly overlooked, leaving individuals to imagine worse case scenarios that manifest as additional stress in an already stressful work environment. It is suggested that a formalized change management process which directly addresses the identified challenges within the organizational technology adoption process (OTAP) is needed in order to mitigate undue stress.
|
Jens Kersten, Jan Bongard, & Friederike Klan. (2021). Combining Supervised and Unsupervised Learning to Detect and Semantically Aggregate Crisis-Related Twitter Content. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 744–754). Blacksburg, VA (USA): Virginia Tech.
Abstract: Twitter is an immediate and almost ubiquitous platform and therefore can be a valuable source of information during disasters. Current methods for identifying and classifying crisis-related content are often based on single tweets, i.e., already known information from the past is neglected. In this paper, the combination of tweet-wise pre-trained neural networks and unsupervised semantic clustering is proposed and investigated. The intention is to (1) enhance the generalization capability of pre-trained models, (2) to be able to handle massive amounts of stream data, (3) to reduce information overload by identifying potentially crisis-related content, and (4) to obtain a semantically aggregated data representation that allows for further automated, manual and visual analyses. Latent representations of each tweet based on pre-trained sentence embedding models are used for both, clustering and tweet classification. For a fast, robust and time-continuous processing, subsequent time periods are clustered individually according to a Chinese restaurant process. Clusters without any tweet classified as crisis-related are pruned. Data aggregation over time is ensured by merging semantically similar clusters. A comparison of our hybrid method to a similar clustering approach, as well as first quantitative and qualitative results from experiments with two different labeled data sets demonstrate the great potential for crisis-related Twitter stream analyses.
|
Jeremy Diaz, Lise St. Denis, Maxwell B. Joseph, Kylen Solvik, & Jennifer K. Balch. (2020). Classifying Twitter Users for Disaster Response: A Highly Multimodal or Simple Approach? In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 774–789). Blacksburg, VA (USA): Virginia Tech.
Abstract: We report on the development of a classifier to identify Twitter users contributing first-hand information during a disaster. Identifying such users helps social media monitoring teams identify critical information that might otherwise slip through the cracks. A parallel study (St. Denis et al., 2020) demonstrates that Twitter user filtering creates an information-rich stream of content, but the best way to approach this task is unexplored. A user's profile contains many different “modalities” of data, including numbers, text, and images. To integrate these different data types, we constructed a multimodal neural network that combines the loss function of all modalities, and we compared the results to many individual unimodal models and a decision-level fusion approach. Analysis of the results suggests that unimodal models acting on Twitter users' recent tweets are sufficient for accurate classification. We demonstrate promising classification of Twitter users for crisis response with methods that are (1) easy to implement and (2) quick to both optimize and infer.
|
Justin Michael Crow. (2020). Verifying Baselines for Crisis Event Information Classification on Twitter. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 670–687). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media are rich information sources during crisis events such as earthquakes and terrorist attacks. Despite myriad challenges, with the right tools, significant insight can be gained to assist emergency responders and related applications. However, most extant approaches are incomparable, using bespoke definitions, models, datasets and even evaluation metrics. Furthermore, it's rare that code, trained models, or exhaustive parametrisation details are openly available. Thus, even confirming self-reported performance is problematic; authoritatively determining state of the art (SOTA) is essentially impossible. Consequently, to begin addressing such endemic ambiguity, this paper makes 3 contributions: 1) replication and results confirmation of a leading technique; 2) testing straightforward modifications likely to improve performance; and 3) extension to a novel complimentary type of crisis-relevant information to demonstrate it's generalisability.
|
Kamol Roy, MD Ashraf Ahmed, Samiul Hasan, & Arif Mohaimin Sadri, P. D. (2020). Dynamics of Crisis Communications in Social Media: Spatio-temporal and Text-based Comparative Analyses of Twitter Data from Hurricanes Irma and Michael. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 812–824). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media platforms play critical roles in information dissemination, communication and co-ordination during different phases of natural disasters as it is crucial to know the type of crisis information being disseminated and user concerns. Large-scale Twitter data from hurricanes Irma (Sept. 2017) and Michael (Oct. 2018) are used here to understand the topic dynamics over time by applying the Dynamic Topic Model, followed by a comparative analyses of the differences in such dynamics for these two hurricane scenarios. We performed a spatio-temporal analyses of user activities with reference to the hurricane center location and wind speed. The findings of spatio-temporal analyses show that differences in hurricane path and the affected regions influence user participation and social media activity. Besides, topic dynamics reveals that situational awareness, disruptions, relief action are among the patterns common for both hurricanes; unlike topics such as hurricane evacuation and political situation that are scenario dependent.
|
Lise Ann St. Denis, Amanda Lee Hughes, Jeremy Diaz, Kylen Solvik, Maxwell B. Joseph, & Jennifer K. Balch. (2020). 'What I Need to Know is What I Don't Know!': Filtering Disaster Twitter Data for Information from Local Individuals. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 730–743). Blacksburg, VA (USA): Virginia Tech.
Abstract: We report on the design, development, and evaluation of a user labeling framework for social media monitoring by emergency responders. By labeling Twitter user accounts based on behavior and content, this novel approach identifies tweets from accounts belonging to Individuals generating Personalized content and captures information that might otherwise be missed. We evaluate the framework using training data from the 2018 Camp, Woolsey, and Hill fires. Approximately 30% of the Individual-Personalized tweets contain first-hand information, providing a rich stream of content for social media monitoring. Because it can quickly eliminate most redundant tweets, this framework could be a critical first step in an end-to-end information extraction pipeline. It may also generalize more easily for new disaster events since it relies on general user account attributes rather than tweet content. We conclude with next steps for refining and evaluating our framework in near real-time during a disaster response.
|
Liuqing Li, & Edward A. Fox. (2020). Disaster Response Patterns across Different User Groups on Twitter: A Case Study during Hurricane Dorian. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 838–848). Blacksburg, VA (USA): Virginia Tech.
Abstract: We conducted a case study analysis of disaster response patterns across different user groups during Hurricane Dorian in 2019. We built a tweet collection about the hurricane, covering a two week period. We divided Twitter users into two groups: brand/organization or individual. We found a significant difference in response patterns between the groups. Brand users increasingly participated as the disaster unfolded, and they posted more tweets than individual users on average. Regarding emotions, brand users posted more tweets with joy and surprise, while individual users posted more tweets with sadness. Fear was a common emotion between the two groups. Further, both groups used different types of hashtags and words in their tweets. Some distinct patterns were also discovered in their concerns on specific topics. These results suggest the value of further exploration with more tweet collections, considering the behavior of different user groups during disasters.
|
Lucia Castro Herrera. (2021). Configuring Social Media Listening Practices in Crisis Management. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 640–654). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media listening practices are increasingly adopted in crisis management and have become an object of interest for researchers and practitioners alike. This article analyzes how these enactments have been studied in the academic literature. Through a systematic review of the available body of knowledge, features from studies involving depictions of practice were extracted, analyzed, and turned into a narrative using an inductive approach. Strategies of improvisation, overreliance on personal and professional networks, manual work, spontaneous coordination, and re-assigning tasks represent the main findings in the multidisciplinary literature. This article is a consolidated overview of experiences from social media listening in practice beyond listing the benefits of social media as a source of information. Moreover, the paper sets the basis for future studies on the range of possible configurations and institutionalization of disruptive crisis management practices.
|
Ly Dinh, Sumeet Kulkarni, Pingjing Yang, & Jana Diesner. (2023). Reliability of Methods for Extracting Collaboration Networks from Crisis-related Situational Reports and Tweets. In V. L. Thomas J. Huggins (Ed.), Proceedings of the ISCRAM Asia Pacific Conference 2022 (pp. 181–195). Palmerston North, New Zealand: Massey Unversity.
Abstract: Assessing the effectiveness of crisis response is key to improving preparedness and adapting policies. One method for response evaluation is reviewing actual response activities and interactions. Response reports are often available in the form of natural language text data. Analyzing a large number of such reports requires automated or semi automated solutions. To improve the trustworthiness of methods for this purpose, we empirically validate the reliability of three relation extraction methods that we used to construct interorganizational collaboration networks by comparing them against human-annotated ground truth (crisis-specific situational reports and tweets). For entity extraction, we find that using a combination of two off-the-shelf methods (FlairNLP and SpaCy) is optimal for situational reports data and one method (SpaCy) for tweets data. For relation extraction, we find that a heuristics-based model that we built by leveraging word co-occurrence and deep and shallow syntax as features and training it on domain-specific text data outperforms two state-of-the-art relation extraction models (Stanford OpenIE and OneIE) that were pre-trained on general domain data. We also find that situational reports, on average, contain less entities and relations than tweets, but the extracted networks are more closely related to collaboration activities mentioned in the ground truth. As it is widely known that general domain tools might need adjustment to perform accurately in specific domains, we did not expect the tested off-the-shelf tools to perform highly accurately. Our point is to rather identify what accuracy one could reasonably expect when leveraging available resources as-is for domain specific work (in this case, crisis informatics), what errors (in terms of false positives and false negatives) to expect, and how to account for that.
|
Matti Wiegmann, Jens Kersten, Friederike Klan, Martin Potthast, & Benno Stein. (2020). Analysis of Detection Models for Disaster-Related Tweets. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 872–880). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media is perceived as a rich resource for disaster management and relief efforts, but the high class imbalance between disaster-related and non-disaster-related messages challenges a reliable detection. We analyze and compare the effectiveness of three state-of-the-art machine learning models for detecting disaster-related tweets. In this regard we introduce the Disaster Tweet Corpus~2020, an extended compilation of existing resources, which comprises a total of 123,166 tweets from 46~disasters covering 9~disaster types. Our findings from a large experiments series include: detection models work equally well over a broad range of disaster types when being trained for the respective type, a domain transfer across disaster types leads to unacceptable performance drops, or, similarly, type-agnostic classification models behave more robust at a lower effectiveness level. Altogether, the average misclassification rate of~3,8\% on performance-optimized detection models indicates effective classification knowledge but comes at the price of insufficient generalizability.
|
Muhammad Imran, Firoj Alam, Umair Qazi, Steve Peterson, & Ferda Ofli. (2020). Rapid Damage Assessment Using Social Media Images by Combining Human and Machine Intelligence. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 761–773). Blacksburg, VA (USA): Virginia Tech.
Abstract: Rapid damage assessment is one of the core tasks that response organizations perform at the onset of a disaster to understand the scale of damage to infrastructures such as roads, bridges, and buildings. This work analyzes the usefulness of social media imagery content to perform rapid damage assessment during a real-world disaster. An automatic image processing system, which was activated in collaboration with a volunteer response organization, processed ~280K images to understand the extent of damage caused by the disaster. The system achieved an accuracy of 76% computed based on the feedback received from the domain experts who analyzed ~29K system-processed images during the disaster. An extensive error analysis reveals several insights and challenges faced by the system, which are vital for the research community to advance this line of research.
|
Nasik Muhammad Nafi, Avishek Bose, Sarthak Khanal, Doina Caragea, & William H. Hsu. (2020). Abstractive Text Summarization of Disaster-Related Documents. In Amanda Hughes, Fiona McNeill, & Christopher W. Zobel (Eds.), ISCRAM 2020 Conference Proceedings – 17th International Conference on Information Systems for Crisis Response and Management (pp. 881–892). Blacksburg, VA (USA): Virginia Tech.
Abstract: Abstractive summarization is intended to capture key information from the full text of documents. In the application domain of disaster and crisis event reporting, key information includes disaster effects, cause, and severity. While some researches regarding information extraction in the disaster domain have focused on keyphrase extraction from short disaster-related texts like tweets, there is hardly any work that attempts abstractive summarization of long disaster-related documents. Following the recent success of Reinforcement Learning (RL) in other domains, we leverage an RL-based state-of-the-art approach in abstractive summarization to summarize disaster-related documents. RL enables an agent to find an optimal policy by maximizing some reward. We design a novel hybrid reward metric for the disaster domain by combining \underline{Vec}tor Similarity and \underline{Lex}icon Matching (\textit{VecLex}) to maximize the relevance of the abstract to the source document while focusing on disaster-related keywords. We evaluate the model on a disaster-related subset of a CNN/Daily Mail dataset consisting of 104,913 documents. The results show that our approach produces more informative summaries and achieves higher \textit{VecLex} scores compared to the baseline.
|
Nilani Algiriyage, Rangana Sampath, Raj Prasanna, Kristin Stock, Emma Hudson-Doyle, & David Johnston. (2021). Identifying Disaster-related Tweets: A Large-Scale Detection Model Comparison. In Anouck Adrot, Rob Grace, Kathleen Moore, & Christopher W. Zobel (Eds.), ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management (pp. 731–743). Blacksburg, VA (USA): Virginia Tech.
Abstract: Social media applications such as Twitter and Facebook are fast becoming a key instrument in gaining situational awareness (understanding the bigger picture of the situation) during disasters. This has provided multiple opportunities to gather relevant information in a timely manner to improve disaster response. In recent years, identifying crisis-related social media posts is analysed as an automatic task using machine learning (ML) or deep learning (DL) techniques. However, such supervised learning algorithms require labelled training data in the early hours of a crisis. Recently, multiple manually labelled disaster-related open-source twitter datasets have been released. In this work, we create a large dataset with 186,718 tweets by combining a number of such datasets and evaluate the performance of multiple ML and DL algorithms in classifying disaster-related tweets in three settings, namely ``in-disaster'', ``out-disaster'' and ``cross-disaster''. Our results show that the Bidirectional LSTM model with Word2Vec embeddings performs well for the tweet classification task in all three settings. We also make available the preprocessing steps and trained weights for future research.
|