Abstract: The increasing use of social media as an information source brings further challenges – social media platforms can be an excellent medium for disseminating public awareness and critical information, that can be shared across large populations. However, misinformation in social media can have immense implications on public health, risking the effectiveness of health interventions as well as lives. This has been particularly true in the case of COVID-19 pandemic, with a range of misinformation, conspiracy theories and propaganda being spread across social channels. In our study, through a questionnaire survey, we set out to understand how members of the public interact with different sources when looking for information on COVID-19. We explored how participants react when they encounter information they believe to be misinformation. Through a set of three behaviour tasks, synthetic misinformation posts were provided to the participants who chose how they would react to them. In this work in progress study, we present initial findings and insights into our analysis of the data collected. We highlight what are the most common reactions to misinformation and also how these reactions are different based on the type of misinformation.

Abstract: This paper summarizes the final year of the four-year Text REtrieval Conference Incident Streams track (TREC-IS), which has produced a large dataset comprising 136,263 annotated tweets, spanning 98 crisis events. Goals of this final year were twofold: 1) to add new categories for assessing messages, with a focus on characterizing the audience, author, and images associated with these messages, and 2) to enlarge the TREC-IS dataset with new events, with an emphasis of deeper pools for sampling. Beyond these two goals, TREC-IS has nearly doubled the number of annotated messages per event for the 26 crises introduced in 2021 and has released a new parallel dataset of 312,546 images associated with crisis content – with 7,297 tweets having annotations about their embedded images. Our analyses of this new crisis data yields new insights about the context of a tweet; e.g., messages intended for a local audience and those that contain images of weather forecasts and infographics have higher than average assessments of priority but are relatively rare. Tweets containing images, however, have higher perceived priorities than tweets without images. Moving to deeper pools, while tending to lower classification performance, also does not generally impact performance rankings or alter distributions of information-types. We end this paper with a discussion of these datasets, analyses, their implications, and how they contribute both new data and insights to the broader crisis informatics community.

Abstract: During natural disasters, automatic information extraction from Twitter posts is a valuable way to get a better overview of the field situation. This information has to be geolocated to support effective actions, but for the vast majority of tweets, spatial information has to be extracted from texts content. Despite the remarkable advances of the Natural Language Processing field, this task is still challenging for current state-of-the-art models because they are not necessarily trained on Twitter data and because high quality annotated data are still lacking for low resources languages. This research in progress address this gap describing an analytic pipeline able to automatically extract geolocatable entities from texts and to annotate them by aligning them with the entities present in Wikipedia/Wikidata resources. We present a new dataset for Entity Linking on French texts as preliminary results, and discuss research perspectives for enhancements over current state-of-the-art modeling for this task.

Abstract: Overload reduction is essential to exploit Twitter text data for crisis management. Often used pre-trained machine learning models require training data for both, crisis-related and off-topic content. However, this task can also be formulated as a one-class classification problem in which labeled off-topic samples are not required. Gaussian processes (GPs) have great potential in both, binary and one-class settings and are therefore investigated in this work. Deep kernel learning combines the representative power of text embeddings with the Bayesian formalism of GPs. Motivated by this, we investigate the potential of deep kernel models for the task of classifying crisis-related tweet texts with special emphasis on cross-event applications. Compared to standard binary neural networks, first experiments with one-class GP models reveal a great potential for realistic scenarios, offering a fast and flexible approach for interactive model training without requiring off-topic training samples and comprehensive expert knowledge (only two model parameters involved).

Abstract: This paper aims at measuring transfer learning performances across different types of crises related to sudden or unexpected events (like earthquakes, terror attacks, explosions, technological incidents) that cannot be foreseen by emergency services and on the occurrence of which they have virtually no control. Although sudden crises are present in most existing crisis datasets, as far as we are aware, no one studied their impact on classifiers performances when evaluated in an out-of-type scenario in which models are tested on a particular type of crisis unseen during training. Our contribution is threefold: (1) A new dataset of about 3,800 French tweets related to four sudden events that occurred in France annotated for both relatedness (i.e., useful vs. not useful for emergency responders) and urgency (i.e., not useful vs. urgent vs. not urgent), (2) A set of monotask and multitask zero-shot learning experiments to transfer knowledge across events and types, and finally, (3) Experiments involving few-shot learning to measure the amount of sudden events instances needed during training to guarantee good performances. When compared to a cross-event setting, our preliminary results are encouraging and show that transfer from predictable ecological crisis to sudden events is feasible and constitutes a first step towards real-time crisis management systems from social media content.

Abstract: Over the years, with the increase in social media engagement, there has been an in increase in various pipelines to analyze, classify and prioritize crisis-related data on various social media platforms. These pipelines utilize various data augmentation methods to counter imbalanced crisis data, sophisticated and off-the-shelf models for training. However, there is a lack of comprehensive study which compares these methods for the various sections of a pipeline. In this study, we split a general crisis-related pipeline into 3 major sections, namely, data augmentation, model selection, and training methodology. We compare various methods for each of these sections and then present a comprehensive evaluation of which section to prioritize based on the results from various pipelines. We compare our results against two separate tasks, information classification and priority scoring for crisis-related tweets. Our results suggest that data augmentation, in general,improves the performance. However, sophisticated, state-of-the-art language models like DeBERTa only show performance gain in information classification tasks, and models like RoBERTa tend to show a consistent performance increase over our presented baseline consisting of BERT. We also show that, though training two separate task-specific BERT models does show better performance than one BERT model with multi-task learning methodology over an imbalanced dataset, multi-task learning does improve performance for more sophisticated model like DeBERTa with a much more balanced dataset after augmentation.

Abstract: The user-base of social media platforms, like Twitter, has grown dramatically around the world over the last decade. As people post everything they experience on social media, large volumes of valuable multimedia content are being recorded online, which can be analysed to help for a range of tasks. Here we specifically focus on crisis response. The majority of prior works in this space focus on using machine learning to categorize single-modality content (e.g. text of the posts, or images shared), with few works jointly utilizing multiple modalities. Hence, in this paper, we examine to what extent integrating multiple modalities is important for crisis content categorization. In particular, we design a pipeline for multi-modal learning that fuses textual and visual inputs, leverages both, and then classifies that content based on the specified task. Through evaluation using the CrisisMMD dataset, we demonstrate that effective automatic labelling for this task is possible, with an average of 88.31% F1 performance across two significant tasks (relevance and humanitarian category classification). while also analysing cases that unimodal models and multi-modal models success and fail.