Deep learning methods achieve state-of-the-art results in data-driven fields such as autonomous driving, computer vision, and healthcare. Yet, DL models operate in high dimensional spaces with interconnected calculations, which makes them difficult to understand. This is particularly critical as DL models become more prevalent, drawing the attention of adversarial attacks. For instance, models should be robust to potential adversarial attacks which operate by boosting weights to one or more features to yield more favourable outcomes.
This project investigates the stability of trained neural networks, paired with complimentary visualisations to provide insight. Instead of applying perturbations to the input data, this research explores the development of a highly-customisable, model-agnostic pipeline for targetted weight perturbations.
The notion of stability
We refer to the ‘Stability’ of a neural network as the model’s ability to resist change. Change is often in reference to the input data. For instance, pixels may be modified on an image to cause a model to change prediction outcome, yet the change may be imperceptible to the human eye. Such explorations of stability are often limited to the individual sample or a small local neighbourhood of samples, as samples reside in a high dimensional space, often distant from each other.
Weight perturbations are global, impacting multiple classes which may share neurons, revealing otherwise unseen behaviour. A key component of this behaviour is how the Decision Zones (Regions of a high dimensional space that correspond to a specific outcome) of a model change shape after perturbations. Samples closest to this boundary are the most vulnerable. Understanding where and how this change occurs is critical to understanding stability.
The visualisations we produce visualise how these decision zones change shape, utilising dimensionality reduction techniques to provide a snapshot of this high dimensional space.
What are we investigating?
This project explores questions including:
- Given a perturbation to neurons associated with a particular outcome (e.g. a class or a numeric output), what other outcomes become predicted as this outcomes?
- Given neurons associated with a particular outcome (e.g. a class or a numeric output), are there any other outcomes that are associated with these neurons that are impacted when the weights are changed?
- Given our approach to targetted neuron perturbations, how does this compare to random perturbations?
- How can our approach yield actionable insights into regressor stability?
- How does the choice of visualisation technique impact the user’s perception of model stability?
Team
- Luke Watkin (PhD student), Newcastle University
- Prof. Daniel Archambault (main supervisor), Newcastle University
- Dr Deepayan Bhowmik (co-supervisor), Newcastle University
Collaborators
- Prof. Alex Telea, Utrecht Univeristy
Publications
Watkin, L., Archambault, D., & Telea, A. (2026). Shapdbm: Exploring decision boundary maps in shapley space. https://doi.org/10.2312/EVS.20261004