Short CommunicationAbstract
Introduction: Neural networks have demonstrated remarkable success in
extracting intricate patterns from complex data. An attempt was made to design a
neural network with the aim of predicting team performance in acute stroke care, in
terms of door to needle time.
Methods: Duration, severity by NIHSS stroke scale, age in years and hour of day at
arrival were identified as the predictive features, and door to needle time of above 45
minutes was set as the target for training the neural network. A three-layered network
was implemented using the Tensor Flow framework and was trained on historical data
and validated on a separate set.
Results: The network after optimization showed precision 63%, recall 67% and F1
64% on the validation set, with an overall accuracy of 60% (defined as percentage of
delay > 45 min correctly predicted).
Conclusion: With the above four parameters, though individually having a weak
association, the neural network was able to construct a reasonable prediction model
for stroke team performance.
Keywords: Machine Learning; Neural Networks; Acute Stroke Care; Thrombolysis
Introduction
Machine learning with neural networks have demonstrated remarkable success in finding patterns in structured, as well as unstructured data. It has been applied to various aspects of health care including early detection and diagnosis, treatment, as well as outcome prediction and prognosis evaluation [1]. Process optimization is another application of machine learning, as it can find intricate trends from historical data. With 15-year historical data from the stroke thrombolysis program at a tertiary care center, an attempt was made to design a neural network with the aim of predicting team performance, in terms of door to needle time (defined as the time interval from arrival at the hospital to administration of the thrombolytic drug). Such information could be of help to warn of potential delay when a particular patient arrives, and the team armed with this information will be better enabled to pre-empt the delay [2].
Methods
Historical records of acute stroke thrombolysis were available from 2003 – 2018. The data was first split into training, validation sets. Data from 2003 – 201, consisting of 221 cases, was used as the training data, and the last 2 years’ data of 61 cases was used for validation. Identification of feature and target measures. (The inputs parameters fed to a machine learning model are called features, and the output it is trained to optimize on, the target). It has been reported from various centers that the door to needle time in acute ischemic stroke tends to have an inverse relation to duration of stroke at presentation [3], and this was replicated from data in our center as well. Similarly, regression and scatter plots were used to identify that: severity of stroke by NIH Stroke Score (NIHSS) scale, age of the patient in years, time of the day (rounded to the nearest hour) were also showing an association with the door to needle time, though individually the effects were weak (Figure 1). Thus, these four features were selected to train the network. The target parameter was the door to needle time, parametrized as above or below 45 minutes, as this was the cut off that showed the best predictive performance. Cases with delay <= 45 minutes were labelled as 0 and others as [1] (Table 1).
Table 1: Features and target.
Neural Network Architecture
A 3-layered fully connected architecture was used and implemented using the open source Tensor Flow framework [4]. The number of neurons in each layer were 4, 6 and 1, from input to output. The first two layers consisted of rectified linear activation units (Re LU), and the final unit used the sigmoid activation function, to give output in the range of 0 to 1. (Figure 1). The source code of the implementation is available at https://github.com/mlnoone/ nnstroketeamperf/
Training
The network was trained using batch gradient descent, with total iterations of 1500. Sigmoid cross entropy cost of the 45 min delay prediction (ranging 0 to 1) vs actual status (0 vs 1) was used as the minimization target. The learning rate was optimized based on performance on the training set and a value of 0.15 achieved the best result as shown in (Figures 2 & 3).
Figure 1: Scatter plots showing the (weak) correlations with door to needle time in minutes (on the vertical axis in all) of duration (A), Age(B), Severity (C) and Hour of day (D)
Figure 2: The neural network architecture.
Figure 3: Plot of cost vs iterations as the network was being trained at a learning rate of 0.15
Hyper Parameter Tuning
Based on performance in the training and the validation sets,
the hyper parameters of the network were optimized. Performance
on the validation set was assessed using the following parameters:
a) Accuracy, defined as percentage of delay > 45 min correctly
predicted.
b) Sensitivity and positive predictive value (also called recall and
precision in machine learning parlance) which were combined
into the F1 score, defined as: 2 x precision x recall / (precision
+ recall).
These were then used to select the optimum hyper parameters
for the network which, in addition to the learning rate of 0.15,
included:
a) The number of neurons in each layer as mentioned above was
set at 4, 6 and 1 which resulted in optimum performance on
the training set (accuracy 60%, precision 68%, recall 71%, F1
0.68).
b) L2 Regularization was used to minimize overfitting which
was observed initially, a regularization parameter (lambda) of
0.02 was chosen based on observing the result over a range of
parameters to minimize error on the validation set.
Results
The network after optimization showed precision (positive predictive value) 63%, recall (sensitivity) 67% and F1 0.64 on the validation set, with an overall accuracy of 60% (Table 2).
Table 2: Performance measures on the validation set.
Discussion
Neural networks are capable of extracting intricate patterns from complex data, which are otherwise nearly impossible to delineate. They have been applied with remarkable success in fields like advertising and manufacturing to optimize performance based on structured data. Though the success of neural network with unstructured data like computer vision, speech and language are more widely appreciated, they are also remarkably effective in analyzing structured data. As seen here, with four weakly associated parameters it was possible to develop a reasonable prediction model for stroke team performance with 60% accuracy and 63% positive predictive value. Application of neural network in this context of optimizing acute care based on a predictive model has not so far been reported in literature. With increasing application of care pathways, neural networks could be a natural choice to build predictive models to identify potential for errors and delays.
References
- Jiang F, Jiang Y, Zhi H, Yi Dong, Hao Li, et al. (2017) Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol 2(4): 230-243.
- Maestre Moreno JF, Arnáiz Urrutia C, Del Saz Saucedo P, KA Vatz, I Feria Vilar, et al. (2007) Impact of warning against procrastination of thrombolysis in ischemic stroke. Article in Spanish. Rev Neurol 44(11): 643-646.
- Strbian D, Michel P, Ringleb P, Heikki Numminen, Lorenz Breuer, et al. (2013) Relationship between onset-to-door time and door-to-thrombolysis time: a pooled analysis of 10 dedicated stroke centers. Stroke 44: 2808-2813.
- Abadi M, Barham P, Chen J, Zhifeng Chen, Andy Davis, et al. (2016) TensorFlow: A system for large-scale machine learning, arXiv 1605: 08695.
