Abstract: Disaster management exercises are a core component of humanitarian organizations’ preparedness strategies. They entail diverse purposes, from training capabilities of participants to testing response plans to enhancing collaboration between organizations and many more. However, it is uncertain how much exercises contribute to preparedness. Rigorous evaluation is needed to exploit learning opportunities of an exercise. Therefore, exercises must target evaluable objectives, which is complicated by the socio-technical openness of the exercise system, the heterogeneity of organizational needs, and the scarcity of resources. Many different tools aim to support evaluation but are limited to specific use cases, resulting in a fragmented overview for practitioners. Due to the excessive effort involved, practitioners often consider exercise evaluation to be of secondary importance. This study thus proposes the conceptual design of a combined toolkit that supports the practitioners in a more rigorous but resource-efficient evaluation to make disaster management exercises more evaluation-driven.