Automated Computer Vision Technology Algorithm for Wrist Joint Detection for Rheumatoid Arthritis Volume 51- Issue 4

Xiao Tianle and Carol Anne Hargreaves*

• Department of Statistics and Data Science, National University of Singapore, Singapore

Received: July 10, 2023;   Published: July 24, 2023

*Corresponding author: Carol Anne Hargreaves, Department of Statistics and Data Science, National University of Singapore, Singapore 117546

DOI: 10.26717/BJSTR.2023.51.008148

Abstract PDF

The application of artificial intelligence (AI), including Deep Learning (DL) algorithms, is an emerging ground in the healthcare industry. Rheumatoid arthritis (RA) is a disease that is at the forefront of digital healthcare. The traditional manual approach is time consuming, tedious, and highly subjective. Multiple research paper has demonstrated promising results for the classification of rheumatoid arthritis using the fingers and toes, but little research has shown progress at classifying rheumatoid arthritis using the wrist X-Ray images. The objective of this paper is to develop an algorithm for automatically detecting the wrist joints of rheumatoid arthritis patients using X-Ray images and computer vision technology. In this study, >95% accuracy is obtained for the wrist joint detection. This paper serves as a proof of concept for wrist joint identification. Our initial experiments in wrist joint detection have shown significant improvements to this area of research, bringing automatic wrist joint detection closer than ever before towards a clinical support system. Our automated algorithm also provides the foundation for the classification of RA using wrist joint X-Ray images.

Graphical Abstract

Keywords: Rheumatoid Arthritis; Image Detection; X-Ray; YOLOv7; Computer Vision Technology

Abbreviations: CLAHE: Contrast Limited Adaptive Histogram Equalization; AI: Artificial Intelligence; Dl: Including Deep Learning; RA: Rheumatoid Arthritis; IOU: Intersection Over Union

Problem

There is no known method to automatically detect wrist joints using X-Ray images for the classification of rheumatoid arthritis patients.

What is Already Known

Multiple research papers have demonstrated promising results for the classification of rheumatoid arthritis using finger and toes X-Ray images, but little research has shown progress for the wrist joint identification.

Our Contributions to Wrist-Joint X-Ray Image Detection

The objective of this study was to construct a deep learning algorithm that provides automatic detection of the wrist joints using the state of the art computer vision technology, YOLOv7. In this preliminary study, >95% accuracy was obtained for wrist joint detection. This paper serves as a proof of concept for wrist joint identification. It also provides the foundation for the classification of wrist joints for rheumatoid arthritis. Rheumatoid arthritis (RA) is a chronic autoimmune disease that primarily affects the joints. The global RA prevalence estimate was 0.46% in 2015 [1], and this number is expected to increase. Its symbolic symptoms include inflammation in the synovial membrane (the lining of the joint), which can cause pain, stiffness, and swelling in the affected area. Over time, RA can lead to damage to the joints characterized by joint space narrowing, erosions in subchondral bone, and joint deformity [2], which may eventually result in irreversible abnormality and loss of function. Patients at early stage of RA often develop disease-related or therapy-induced osteopenia and joint space narrowing caused by the dissolution of cartilage tissue. These morphological alterations can be detected by CAD techniques before further deterioration [3]. Early detection of RA is crucial so that doctors could intervene in the deterioration progress with drug treatments. Several attempts have been made to develop software and algorithms to automate this process. Computerized measurement provides quantitative data on JSW that can be more reproducible than data obtained with traditional scoring systems [4-6]. Automated detection has been shown to be more sensitive to changes than SvdH scores [7]. Jin et al. illustrated that eye diseases could be related to RA development in patients above age of 50 [8].
Initial experiments involving machine learning algorithms were first attempted [9]. O’Neil et al. designed a regression model to give prediction whether a patient is at high risk of developing RA [10]. Deep learning is a type of AI that proves to be powerful in healthcare industry, such as medical imaging and text from electronic health records [11]. Deep learning-based models are used to conquer a broad range of missions such in rheumatology such as testing for antinuclear antibodies [12], interpretation of synovial ultrasounds [13], and predicting diagnoses from an EHR [14]. A study found that building accurate models to forecast complex disease outcomes using electronic health record data is possible [15]. More recently, convolutional neural networks have become the dominant method for medical image classification [16,17], medical image segmentation [18]. An efficient CNN architecture (GRNN) achieves high accuracy for hand X-ray classification [19]. An efficient CNN ResNet-Dwise50 model was designed for the overall scoring of RA in hand X-rays by introducing depth wise separable convolution block and inverted residual block [20]. Another common approach involves a two-step approach to detect finger joint destruction [21,22]. A customed model using SIFT and CNN to extract features outperformed traditional ML classifiers [23]. YOLOv4 and VGG16 were combined for the assessment of RA, osteoarthritis and achieved 90.7% accuracy [24]. Additional efforts constructed a customized CNN model with batch normalization, ReLU, and pre-trained VGG 16 model but failed to improve the accuracy (67.5%) [25]. The main contributions of this project, in which Rheumatoid Arthritis classification was performed on radiographic pictures at wrist area using novel computer vision algorithms, are shown as below.
• A dataset containing 367 pairs of hands was obtained from CLEAR repository and TETRAD study. The X-ray images provide an accurate depiction of various stages of patients in real-life scenarios.
• To understand the distribution of joint space narrowing and bones erosion scores at all wrist joints/bones, data exploration was performed with the given score tables.
• To resolve the issue of imbalance in various classes of scores, 3 joints and 3 bones are selected for further study. The selected bones have a more balanced distribution of scores and are typical targets of inflammation.
• In order to improve the interpretability of given images, several image pre-processing methods were applied in sequence, including cropping, normalization, resizing, padding, contrast increment.
• Manual labelling was conducted to provide the ground truth to the deep learning models. The joints and bones were labelled separately in a free, open-source software. Overall, 6 classes of bounding boxes were drawn on each wrist image.
• YOLOv7 was chosen as the object detection algorithm and the training results were compared. Metrics including mean average precision (mAP), precision, recall, F1-score were used to tune parameters of the model. Different sizes of bounding boxes were also tested, but their results showed negligible difference.
• With the newly publish YOLOv7, modern computer vision has proven to be capable of detecting wrist joints and bones with tremendously high accuracy. In this ongoing research project, the object detection result provides a solid foundation for the future scoring models.

Dataset

The radiographic image dataset used in this project contains 367 pairs of X-ray images of both left hand and right hand of patients, including the wrist area. High-resolution radiographic images of both hands were presented in sequence as JPEG images. The side of hands shown in the picture was indicated in the file name as either LH (Left Hand) or RH (Right Hand). Two separate CSV files were provided with overall erosion scores, individual erosion score for each joint, overall narrowing scores, and individual narrowing score for each joint. The scores were obtained through Sharp/van der Heijde method by certified and recognized physicians.

Data Exploration

Due to the limit of computational resources, we decide to only focus on joints that are comparatively equally distributed. We selected ‘mna’, ‘capnlun’, ‘radcar’, ‘mul’, ‘nav’, and ‘lunate’. We first categorized the scores into 0,1 and the rest. The assumption was samples with score 1 have only minor symptoms. Those at the initial stages of RA do not require too much in house treatment. We indicated zeros and ones as ‘healthy’ and the rest as ‘unhealthy’ for ease of representation. The distribution is roughly 2:1 for joints which is a reasonable range. The distribution of bones is significantly more unequal. We also tried to isolate zeros from the rest and derived the graphs in (Figures 1 & 2) The distribution of this categorization is more likely to give satisfactory results.

Figure 1

Cropping

The raw X-ray images from the dataset were first cropped to focus only on the wrist area, which is the region of interest. For the convenience of the following steps, we removed the redundant parts of images by a fixed proportion. After conducting trials and manual observations, we decided to remove the upper 3/7 of all images as shown in Figure 3. Our aim was to eliminate most of the fingers and palms while still ensuring that all the wrist bones remained clearly visible.

Figure 2

Normalization

Normalization can speed up the convergence rate during training by helping to find the optimal solution more quickly. This is because normalization reduces the range of the input data, which can help the optimizer take larger steps towards the minimum. After normalization, the input features have similar scales and distributions (0-1), which can improve the performance of the model during training.

Resizing and Padding

The cropped images were resized to a dimension of 1400 * 760. Firstly, we obtained the minimum width and length of all the images, and then we multiplied the exact pixel numbers by an arbitrary number to derive reasonable integer pixel numbers of 1400 and 760. We enlarged the images proportionally by a fixed number to make the labelling process easier. Uniform height and width also ensure that the joint detection model can identify areas of interest within a smaller range, thus increasing the success rate. To maintain the original aspect ratio for each image, black pixels were added to areas where necessary. For each image, we compared its height ratio (1400 / height) and width ratio (760 / width). We rescaled the image according to the smaller ratio while keeping the original aspect ratio. We calculated the amount of padding required on all four sides so that 1400 and 760 could be achieved. This ensures that the objects remain centered after resizing. Black pixels were used to fit the grayscale background color.

Contrast Increment

Contrast Limited Adaptive Histogram Equalization (CLAHE) is a proven technique for enhancing the contrast of an image. It extends the traditional Histogram Equalization method by spreading the intensity values of an image evenly across its entire histogram. We applied CLAHE to all the images with a clip limit of 2.0 and a grid size of (8,8). This procedure greatly improved the contrast and made the bones more visible. It also helped alleviate the problem of some pictures being overexposed or underexposed. The visual features are more likely to be captured by the model, thus easing the learning process of joint identification. CLAHE also achieved noise removal. As shown in Figure 3, the image is much clearer after the contrast is increased.

Figure 3

Joint Segmentation (YOLO)

Two separate models were trained for joints and bones, respectively. In the training process, we first download the weight document from the YOLOv7 official GitHub repository, which was pre-trained on the Common Objects in Context (COCO) dataset. This is the starting weight for our training process. The joints and bones models were trained independently of each other. Since the number of classes is the same for each model (Capnlun, Mna, Radcar for joints, and mul, nav, lunate for bones), the same set of parameters could be applied as follows: epochs = 300, batch-size = 2, img-size = 1400, 760, device = ‘0’, workers = 1. The final epoch number was determined after multiple failed attempts at 80, 100, and 200 epochs. The batch size and number of workers were kept low since increasing these values exceeded the maximum memory of the GPU. However, when training one model for each joint, 200 epochs were sufficient to generate bounding boxes independently. Since the three classes of labels were exported together, we had to extract only one line of label when training one model for one joint. Additionally, YOLOv7 only recognizes class labels that start from 0. Therefore, when training an isolated model to detect Mna joints, we first had to extract the lines of label that began with class label 1 (Mna is the second class) and then convert all the 1s to 0s.

Joints Detection Model Testing Result

The confusion matrix in Figure 4 below shows the accuracy of the YOLOv7 wrist detection results, with each box indicating a 100% accuracy for the joint location. For the test Intersection Over Union (IoU) threshold, we set it to the default value of 0.65. As long as the predicted joint area has an intersection of >=65% with the manually labelled area, it is classified as detected. All joints have an accuracy of >=95%, with Capnlun being the lowest, which could be due to differences in the extent of overlap. The F1-score reaches its highest value at a confidence level slightly over 0.6 and then continues to drop. To further improve the detection accuracy of a specific joint, we could train separate models for each joint and tune the parameters independently.

Figure 4

Potential Impacts

This study provides proof of the validity of applying YOLOv7 to detect wrist joints. To our knowledge, no algorithms have been able to automatically classify RA in the wrist area thus far. While classifying RA in fingers and toes has been proven to be reliable, progress in extending this concept to the wrists has been slow. This is primarily due to the overlapping of carpal bones, which makes wrist joints significantly fuzzier to observe. Our study results provide solid evidence that the most advanced computer vision technology can accurately detect wrist joints, which makes an automatic classification algorithm possible. A quick wrist joint detection algorithm will speed up the process for wrist joint scoring and the classification of RA. Even if the final diagnosis must be authorized by a clinician, automated algorithmic wrist detection is valuable in the health care digital transformation and can help clinicians to understand the severity of the RA and prioritise patients with more serious symptoms. A scientific, systematic approach to wrist joint detection and classification also reduces subjectivity, as even experienced clinicians can have conflicting opinions. Prior research has also indicated that clinicians can be negligent in monitoring the RA progress. A categorical score is often vulnerable to minor developments in symptoms, which can be effectively captured using automated algorithms.

Future Work

Unfortunately, due to time constraints, we were only able to complete the wrist joint detection. The advancements in object detection algorithms have given us the opportunity to explore a new frontier in detecting wrist joints for the classification of rheumatoid arthritis. Initial experiments in wrist joint detection have shown significant improvements, bringing automatic detection closer than ever before. We will continue to build on existing experimental results and develop algorithms that can predict scores in targeted areas. Our first goal is to train the model to differentiate healthy joints from unhealthy ones. We define healthy joints as those with a score of 0, and unhealthy ones as those with a score greater than 0. As shown in the data exploration steps, the distribution of healthy vs. unhealthy joints is fairly equal and less prone to underfitting. Once we achieve optimal results, we will build on this foundation and begin training models to make precise predictions on individual scores. One major difficulty we foresee is the imbalance in the distribution of scores. Among all the joints, approximately 80% are healthy, and in some cases, the proportion of healthy joints can be as high as 85% (cmc3, cmc4, cmc5, radius, ulna). The lack of unhealthy joint examples makes it difficult to produce accurate predictions. One proposal is to use techniques such as oversampling (randomly increasing samples in minority classes), under-sampling (randomly decreasing the number of samples in majority classes during training), and ordinal class encoding (making classes with lower number labels subsets of higher order classes).

The authors declare that they have no known competing financial interests or personal relationships that might potentially have influence on this paper.

The datasets described and used in the paper were contributed by University of Alabama at Birmingham. These were obtained through RA2-DREAM Challenge.

1. Almutairi K, Nossent J, Preen D, Keen H, Inderjeeth C, et al. (2020) The global prevalence of rheumatoid arthritis: A meta-analysis based on a systematic review. Rheumatology International 41(5): 863-877.
2. Sun D, Nguyen TM, Allaway RJ, Jelai Wang, Verena Chung, et al. (2022) A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis. JAMA Netw Open 5(8): e2227423.
3. Pfeil A, Böttcher J, Seidl B E, Heyne J P, Petrovitch A, et al. (2007) Computer-aided joint space analysis of the metacarpal-phalangeal and proximal-interphalangeal finger joint: Normative age-related and gender-specific data. Skeletal Radiology 36(9): 853-864.
4. Goligher E C, Duryea J, Liang M H, Wolfe F, Finckh A, et al. (2006). Radiographic joint space width in the fingers of patients with rheumatoid arthritis of less than one year's duration. Arthritis & Rheumatism 54(5): 1440-1443.
5. Sharp J T, Wolfe F, Lassere M, Boers M, Van Der Heijde D, et al. (2004). Variability of precision in scoring radiographic abnormalities in rheumatoid arthritis by experienced readers. The Journal of rheumatology 31(6): 1062-1072.
6. Neumann G, dePablo P, Finckh A, Chibnik L B, Wolfe F, et al. (2011) Patient repositioning reproducibility of joint space width measurements on hand radiographs. Arthritis care & research 63(2): 203-207.
7. Angwin J, Lloyd A, Heald G, Nepom G, Binks M, et al. (2004) Radiographic hand joint space width assessed by computer is a sensitive measure of change in early rheumatoid arthritis. The Journal of rheumatology 31(6): 1050-1061.
8. Jin W, Yao Q, Liu Z, Cao W, Zhang Y, et al. (2021) Do eye diseases increase the risk of arthritis in the elderly population? Aging.
9. Ramasamy U, Sundar S (2022) An illustration of rheumatoid arthritis disease using decision tree algorithm. Informatica 46(1).
10. Momtazmanesh S, Nowroozi A, Rezaei N (2022) Artificial intelligence in rheumatoid arthritis: Current status and future perspectives: A state-of-the-art review. Rheumatology and Therapy 9(5): 1249-1304.
11. Bird A, OakdenRayner L, McMaster C, Luke A Smith, Minyan Zeng, et al. (2022) Artificial intelligence and the future of radiographic scoring in rheumatoid arthritis: a viewpoint. Arthritis Res Ther 24: 268.
12. Mahler M, Meroni PL, Bossuyt X, Fritzler MJ (2014) Current concepts and future directions for the assessment of autoantibodies to cellular antigens referred to as anti-nuclear antibodies. J Immunol Res 2014: 315179.
13. Kaeley GS, Bakewell C, Deodhar A (2020) The importance of ultrasound in identifying and differentiating patients with early inflammatory arthritis: a narrative review. Arthritis Res Ther 22(1): 1.
14. Norgeot B, Glicksberg BS, Trupin L, Lituiev D, Gianfrancesco M, et al. (2019) Assessment of a deep learning model based on electronic health record data to forecast clinical outcomes in patients with rheumatoid arthritis. JAMA Netw Open 2(3): e190606.
15. Li X, Guo Z, Zhang S, Guo X (2021) A new deep model based on part pooling for Medical Image Classification. 2021 IEEE International Conference on Artificial Intelligence and Industrial Design (AIID).
16. Masquelin A H, Cheney N, Kinsey C M, Bates J H (2021) Wavelet decomposition facilitates training on small datasets for medical image classification by Deep Learning. Histochemistry and Cell Biology 155(2): 309-317.
17. Tseng K K, Zhang R, Chen C M, Hassan M M (2020) DNetUnet: A semi-supervised CNN of Medical Image Segmentation for super-computing AI service. The Journal of Supercomputing 77(4): 3594-3615.
18. Mate G S, Kureshi A K, Singh B K (2021) An Efficient CNN for Hand X-Ray Classification of Rheumatoid Arthritis. Journal of healthcare engineering 2021: 6712785.
19. Wang Z, Liu J, Gu Z, Li C (2022) An efficient CNN for hand X-ray overall scoring of rheumatoid arthritis. Complexity 2022: 1-9.
20. Chaturvedi N (2021) DeepRA: Predicting Joint Damage From Radiographs Using CNN with Attention. ArXiv.
21. Hirano T, Nishide M, Nonaka N, Seita J, Ebina K, et al. (2019) Development and validation of a deep-learning model for scoring of radiographic finger joint destruction in rheumatoid arthritis. Rheumatology Advances in Practice 3(2).
22. Ahalya R K, Umapathy S, Krishnan P T, Joseph Raj A N (2022) Automated evaluation of rheumatoid arthritis from hand radiographs using machine learning and Deep Learning Techniques. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine 236(8): 1238-1249.
23. Üreten K, Maraş H H (2022) Automated classification of rheumatoid arthritis, osteoarthritis, and normal hand radiographs with deep learning methods. Journal of Digital Imaging, 35(2): 193-199.
24. Rohrbach J, Reinhard T, Sick B, Dürr O (2019) Bone erosion scoring for rheumatoid arthritis with deep convolutional neural networks. Computers & Electrical Engineering 78: 472-481.
25. W (2023) GitHub - WongKinYiu/yolov7: Implementation of paper - YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. GitHub.