Prediction and detection of side effects severity following COVID-19 and influenza vaccinations: utilizing smartwatches … – Nature.com

Study design and participants

We studied a cohort of 2111 participants (>18 years of age) who were recruited between 14 December 2021 and 29 September 2022 to the PerMed prospective observational study from all across Israel27,36. Of the 2,111 participants, 1932 participants reported receipt of>1 doses of the BNT162b2 mRNA COVID-19 vaccine, and 856 participants reported receipt of>1 doses of the influenza vaccine. Specifically, of the 1932 participants receiving COVID-19 vaccination, during the study, 146 received their first dose, 209 their second dose, 1545 their third dose, and 685 their fourth dose. Of the 856 participants who received influenza vaccination, 791 received the seasonal dose for 20212022, 188 received the seasonal dose for 20222023, and 125 received both seasonal doses. Participant recruitment was conducted via advertisements on social media and word-of-mouth. Each participant signed an informed consent form after receiving a comprehensive explanation of the study from a professional survey company. The participants were equipped with a Garmin Vivosmart 4 smartwatch and installed two apps on their smartphones: a dedicated mobile application (PerMed App), that collected daily self-reported questionnaires37, and an application that passively recorded the smartwatch data. Further information regarding data collection architecture and the PerMed dashboard is provided in our previous works13,25,27 and the Supplementary Material (Appendix B).

We hired a professional survey company to recruit participants as well as to keep them engaged throughout the PerMed study. The survey company was responsible for guaranteeing that the participants met the studys requirements, in particular, that they agreed to wear the smartwatch and fill in the daily questionnaires at least three times a week. We implemented several measures to minimize attrition and churn of participants and consequently improve the quality, continuity, and reliability of the collected data. First, participants who did not fill out the daily questionnaire by 7 p.m. received a notification in their mobile app to fill out the questionnaire. Second, a dedicated dashboard that allowed the survey company to identify participants who continually neglected to complete the daily questionnaires at least three times a week or did not wear their smartwatch for a long duration of time was developed. These participants were contacted by the survey company (either by text messages or phone calls) and were encouraged to better adhere to the study protocol. Third, to strengthen participants engagement, a weekly personalized summary report was generated for each participant and was available inside the PerMed application. Similarly, a monthly newsletter with recent findings from the study and useful tips regarding the smartwatchs capabilities was sent to the participants. At the end of the study, participants will receive all personal insights that were obtained and can keep the smartwatch as a gift.

After joining the PerMed study, participants filled out the enrollment questionnaire, and information on participants sex, age, and underlying medical conditions, was collected. The list of underlying medical conditions consisted of hypertension, diabetes, heart disease, chronic lung disease, immune suppression, cancer, renal failure, and body mass index (BMI)>30 (BMI is defined as weight in kilograms divided by the square of height in meters). Participants filled out a daily questionnaire through the PerMed mobile application27,36. The questionnaire allowed participants to report their signs and symptoms from a closed list of local and systemic reactionspreviously observed in BNT162b2 mRNA COVID-19 or influenza vaccines31,38, with an option to add other symptoms as free text. A detailed description of the questionnaire is provided in the Supplementary Material (Appendix A pp 68).

Participants were equipped with Garmin Vivosmart 4 smart fitness trackers. Among many physiological measurements, the smartwatch provides continuous measures of heart rate, stress, and daily resting heart rate capabilities39. Since the HRV measure is not accessible through Garmins application programming interface, we used Garmins stress level measure instead, which is computed based on the HRV measure40. HRV-based stress is a measure between 1 to 100 computed by Garmin and is categorized into four tiers: rest (125), low (2650), medium (5175), and high (76100)41. Specifically, the Garmin device uses heart rate data to determine the interval between each heartbeat. The variable length of time between each heartbeat is regulated by the bodys autonomic nervous system. Less variability between beats correlates with higher stress levels, whereas an increase in variability indicates less stress41,42,43. When we examined data collected in our study, we identified a heart rate sample approximately every 15s, a stress sample every 180s, and a daily sampling of resting heart rate.

We performed several preprocessing steps on the daily questionnaire data and smartwatch physiological measures before analyzing the data. For the daily questionnaires, if participants filled in the daily questionnaire more than once on a given day we considered only the last entry reported. For the HRV-based stress and heart rate measures collected by the smartwatches, we computed the mean value for each hour of data. Then, to impute missing values, we performed a linear interpolation. Finally, data was smoothed by calculating the moving average value using a 5-h sliding window.

For each participant, we defined the 7days before vaccination as a baseline period. For the analysis which involves self-reported questionnaires and for the machine learning model, we included only participants who filled out at least one questionnaire during the baseline period and at least one questionnaire during the 7days following vaccination. Those questionnaires are required to determine whether symptoms reported by the participants should be considered side effects. We defined a reaction as a post-vaccination side effect if it had not been reported during the baseline period. For the questionnaire data, we calculated the percentage and corresponding 95% CI of participants who reported new systemic reactions in the 7days following vaccination. We used a Beta distribution to calculate the 95% CI.

For the analysis involving smartwatch measurements, we included participants who had at least one overlapping period of data (i.e., the same day of the week and same hour during the day for the heart rate and HRV-based stress measures, and the same day of the week for the daily resting heart rate) during their baseline and post-vaccination periods. The overlapping periods are required for computing the change in measurement values between the baseline and post-vaccination periods. To calculate the changes in continuous Garmin smartwatch measurements (heart rate and HRV-based stress measures) over the 07days post-vaccination, with those of the baseline period, we calculated for each participant the difference between the measurement of each hour during the seven days post-vaccination and that of the corresponding hours in the baseline period (keeping the same day of the week and the same hour during the day). For the daily resting heart rate, we calculated the differences in the same manner (keeping the same day of the week). A Randomized control trial31, and prior studies analyzing physiological measures via smartwatches and self-reported questionnaires13, demonstrate a significant decrease in local and systemic reactions within 72h post-vaccination. Consequently, our classification problem focused on determining whether a moderate to severe reaction occurred within this 72h post-inoculation period.

Based on data from the Centers for Disease Control and Prevention, we stratified the participant-reported post-vaccination side effects by the severity of the reactions they reported in the questionnaire in the post-vaccination period by the appearance of symptoms, as follows:

No reaction

Mild reaction: abdominal pain, back or neck pain, feeling cold, muscle pain, weakness, headache, dizziness, vomiting, sore throat, diarrhea, cough, leg pain, ear pain, loss of taste and smell, swelling of the lymph nodes.

Moderate to severe reaction: fast heartbeat, hypertension, chest pain, dyspnea (shortness of breath), fever, confusion, and chills.

Participants were classified into one of the three categories, based on the most severe symptom that was reported in their post-vaccination period.

We stratified participants by the severity of their reactions. Participants who did not report a reaction, or had a Mild reaction following vaccination were classified in the No- or Mild-reaction group, and the remaining participants were classified in the Moderate to severe reaction group. We developed ML models to predict and detect the participant-reported side effect severity following COVID-19 or influenza vaccinations. The prediction model utilized sociodemographics, side effects from previous doses collected from questionnaires, and smartwatch information, but only before the vaccine, while the detection model also utilized the smartwatch measures 72h post-vaccination information.

The entire data set has been randomly divided into 5 separate non-overlapping test sets. For each test set, a model is trained using all the remaining data, ensuring an equal percentage of positive cases between train and test sets to take into account imbalanced positive and negative classes.

Several machine learning techniques were evaluated for both models: Gradient Boosted Decision Trees (XGBoost), Random Forest (RF), Multi-Layer Perceptron (MLP), and K-Nearest Neighbors (KNN).

The XGBoost package was used for training Gradient Boosted Decision Trees44, while the Scikit-learn machine learning library was used to implement the other models45.

Performances of the testing samples from each model are reported by mean AUROC, sensitivity (SE), and specificity (SP). SE and SP are defined as the fraction of positive and negative individuals correctly classified, respectively. These values are based on the point in the ROC that optimized the Youden index46,47. For each classifier, we applied a grid search within our stratified cross-validation framework and optimized our model selection using the mean AUROC.

The interpretable nature of the decision tree model allows for the evaluation of feature importance estimates48. The XGBoost in-model feature importance was used to demonstrate each predictor variables effect on the detection of the participant-reported side effect severity.

For evaluation of the differences in terms of AUROC, a bootstrap test (n=1000) for the difference was used. We repeatedly sampled the dataset with replacement in a stratified manner. We trained the prediction and detection models for each bootstrap sample and computed AUROC on the unique data points that were not selected in the current bootstrap sample. Each model is trained and tested with its subset features and the best hyperparameters. For each bootstrap sample, we computed the AUROC difference between the prediction and detection models and generated a distribution of bootstrapped differences. Finally, we calculated the p-value which is the proportion ofbootstrapped differences that is less than or equal to 0.

Before participating in the study, all subjects were advised, both orally and in writing, as to the nature of the study and gave written informed consent to the study protocol, which was approved by the Tel Aviv University Institutional Review Board (0002522-1). All methods were performed in accordance with the relevant guidelines and regulations.

Continue reading here:

Prediction and detection of side effects severity following COVID-19 and influenza vaccinations: utilizing smartwatches ... - Nature.com