All research

Applied machine learning · 2021

Machine learning analysis of COVID-19 vaccine attitudes in Qatar

A survey of 175 residents of Qatar, analysed with statistical tests, machine learning and dominance analysis to find what drives willingness and hesitancy toward COVID-19 vaccination.

Qatar University MSc course project

Four bar charts comparing twelve classifiers under five-fold cross-validation. K-nearest neighbours leads on accuracy (80.00%), precision (81.07%), recall (84.43%) and F1 score (80.26%). Random forest, extra trees and XGBoost follow on accuracy and precision but have recall near or below 52%. AdaBoost scores lowest on every metric.
Figure 11 from Hafizh et al. (2021). Testing (a) accuracy, (b) precision, (c) recall and (d) F1 score of twelve classifiers for the four attitude classes, averaged over five folds.View full-size figureSource articleCC BY-NC-ND 4.0

Approach

What makes residents of Qatar willing or hesitant to take the COVID-19 vaccine, and can their survey answers predict that attitude?

By early 2021 Qatar had approved COVID-19 vaccines and made its contact-tracing app, Ehteraz, mandatory in public spaces, yet some residents remained hesitant. Evidence on what drives willingness and hesitancy in the region was limited. This study surveyed residents of Qatar online in English and Arabic, tested how attitudes differed between groups, trained machine learning models to predict attitude from survey answers, and ranked the most influential factors with dominance analysis.

  1. Survey residents online
  2. Compare attitudes between groups
  3. Predict attitudes and rank factors

Surveying residents of Qatar

Journal of Human Behavior in the Social Environment · 2022

A short pilot with three participants and open-ended questions shaped the survey. A high health risk and the wish to travel favoured vaccination, while unknown long-term side effects held people back. One participant wanted clearer public information and to see leaders vaccinated first.

The main survey ran on Google Forms in English and Arabic, took 5 to 10 minutes and collected no identifying information. It asked about background, willingness and hesitancy on seven-point scales, and beliefs about past vaccines, religion, public figures, restrictions and the Ehteraz contact-tracing app. Of 175 respondents, 59 (33.71%) had already been vaccinated.

Willingness and hesitancy scores above 3 counted as willing or hesitant. Combining the two gave four attitude classes: willing and not hesitant, willing but hesitant, not willing and hesitant, and not willing and not hesitant.

Respondents
175
Already vaccinated
33.71%
Survey answers used as predictors
19
Weighted mean response on the seven-point scale, by vaccination status. From Table 2 of the paper.
VaccinatedNot vaccinated
I am willing to take the vaccine75
I have some doubts or hesitations about taking the vaccine43
Past vaccines make me concerned about the COVID-19 vaccine33
Taking the vaccine should be a personal choice if I follow safety procedures66
Seeing public figures take the vaccine increases my willingness54
My religious background makes me unwilling to take the vaccine22
More willing if lockdown restrictions are reduced65
More willing if Ehteraz shows a special colour for vaccinated people64
More willing if I no longer have to use Ehteraz64
More willing if I avoid home quarantine after travelling75

Comparing attitudes across groups

Descriptive statistics, ANOVA and t-tests

Willingness was high and hesitancy low for both women and men, and two-way ANOVA found no significant difference between them at the 95% confidence level. Younger respondents were the most willing and least hesitant.

Splitting respondents at age 45 showed that willingness differed significantly between those aged 18 to 44 and those 45 and over (two-tailed p < 0.001), while hesitancy did not (p = 0.87). Past flu vaccination showed no significant interaction with attitude, and religious affiliation was not linked to hesitancy.

These tests look at one factor at a time and cannot rank how much each answer contributes. That question was taken up with machine learning and dominance analysis.

Willingness, ages 18–44 vs 45 and over
p < 0.001
Hesitancy, same age groups
p = 0.87
Confidence level for all tests
95%
Two grouped bar charts of the share of respondents at each level from 1 to 7, for women and men. Willingness peaks at level 7, with 23.43% for each gender. Hesitancy peaks at level 1, with 17.14% of women and 16.00% of men.
Figure 5 from Hafizh et al. (2021). (a) Willingness and (b) hesitancy levels for women and men.View full-size figureSource articleCC BY-NC-ND 4.0
Two grouped bar charts of the share of respondents at each level from 1 to 7, for five age groups from 18–24 to 55 and over. Willingness is concentrated at level 7 in every group, highest for ages 35–44 at 12.00%. Hesitancy is most common at level 1, led by ages 18–24 at 8.00%.
Figure 6 from Hafizh et al. (2021). (a) Willingness and (b) hesitancy levels by age group.View full-size figureSource articleCC BY-NC-ND 4.0

Predicting attitude from survey answers

Machine learning

The 19 answers were encoded as 68 dummy variables, and twelve classifiers were compared in Python with five-fold cross-validation and an 80:20 split. The task was to predict each respondent’s attitude class. Ensemble and tree-based methods were the most stable, and random forest gave the lowest mean absolute error, 0.2057.

K-nearest neighbours had the best average across the four testing metrics, with 80.00% accuracy and 84.43% recall, while many other models had much lower recall. It worked best with two neighbours, which suggests the responses do not form large clusters.

Predicting the seven-level scores directly was harder, and hesitancy was harder to predict than willingness. Random forest again performed best, with a mean absolute error of 0.35 for willingness and 0.71 for hesitancy.

Accuracy with k-nearest neighbours
80.00%
Classifiers compared
12
Lowest mean absolute error (random forest)
0.2057
Testing results for selected classifiers, averaged over five folds (%). Values from Figure 11 of the paper.
AccuracyPrecisionRecallF1 score
K-nearest neighbours80.0081.0784.4380.26
Random forest77.7177.7852.1777.39
Extra trees76.5776.8051.6176.19
XGBoost74.2978.7137.2976.09
Multilayer perceptron71.4375.3178.3372.61
Logistic regression65.7169.4673.6166.84
Line chart with a data table of k-nearest-neighbour accuracy, precision, recall and F1 score for 1 to 11 neighbours. All four metrics peak at two neighbours, with accuracy 0.8286, precision 0.8579, recall 0.8389 and F1 score 0.8317, then fall as more neighbours are used. Recall drops most, to about 0.35 to 0.50.
Figure 12 from Hafizh et al. (2021). Performance of k-nearest neighbours for 1 to 11 neighbours.View full-size figureSource articleCC BY-NC-ND 4.0
Left: multi-class ROC curves for k-nearest neighbours, with areas of 0.84 for the micro average, 0.77 for class 0, 0.54 for class 1 and 1.00 for class 2; class 3 has no curve. Right: confusion matrices for k-nearest neighbours with two neighbours, which correctly classifies 22, 6 and 1 respondents in classes 0 to 2, and for XGBoost, which correctly classifies 19 and 7 in classes 0 and 1.
Figure 13 from Hafizh et al. (2021). (a) Multi-class ROC curves for k-nearest neighbours; confusion matrices for (b) k-nearest neighbours with two neighbours and (c) XGBoost.View full-size figureSource articleCC BY-NC-ND 4.0

Ranking what drives willingness and hesitancy

Dominance analysis

Dominance analysis measured how much each answer explains, on its own, alongside all others and across subsets of them, and ranked the top 15 answers for willingness and for hesitancy.

Willingness was driven most by practical gains: avoiding home quarantine after travel, fewer lockdown restrictions and a special Ehteraz colour for vaccinated people, followed by seeing public figures vaccinated. Hesitancy was swayed most by seeing public figures vaccinated, followed by whether respondents had been vaccinated in Qatar, the view that vaccination should be a personal choice, and sex. Only a few of the many questions carried most of the weight.

Retraining on only the ten top-ranked answers lowered the willingness error for several models, including gradient boosting (0.53 to 0.31) and random forest (0.35 to 0.31).

Relative importance of avoiding quarantine for willingness
23.4%
Relative importance of public figures for hesitancy
31.1%
Top factors ranked for each attitude
15
Top four factors from dominance analysis, with relative importance. From the dominance analysis tables in the paper.
WillingnessHesitancy
1Avoiding home quarantine after travel (23.4%)Seeing public figures vaccinated (31.1%)
2Fewer lockdown restrictions (14.1%)Whether vaccinated in Qatar (13.5%)
3Special Ehteraz colour for vaccinated people (13.8%)Vaccination as a personal choice (10.6%)
4Seeing public figures vaccinated (12.1%)Biological sex (8.2%)
Two grouped bar charts of mean absolute error for twelve classifiers using all answers or only the ten top-ranked answers. For willingness, the top-ten version lowers error for several models, such as gradient boosting from 0.53 to 0.31, random forest from 0.35 to 0.31 and XGBoost from 0.71 to 0.49. For hesitancy, errors are higher overall and change in both directions.
Figure 15 from Hafizh et al. (2021). Mean absolute error with all answers and with the ten top-ranked answers, for (a) willingness and (b) hesitancy.View full-size figureSource articleCC BY-NC-ND 4.0

My contribution

I contributed to the quantitative and machine-learning analysis, interpretation, and manuscript preparation.

Study scope

The 175 respondents were recruited online rather than sampled at random, and their answers reflect attitudes around April 2021, when vaccines were newly available. Few respondents were unwilling but not hesitant, so that class could not be evaluated, and the factors found are associations rather than causes.