Alert - This a masked draft version for internal use only
3. Main Analyses Country Comparison
Load Data
Code
# load data for analysisload("../data/wrangled_data/dt_ana_full.RData")
Analysis Strategy
We evaluate whether the predictive structure for violent intent differs across samples (countries; with a focal interest in the Jihadist sample) using two complementary strategies:
a multilevel model that separates within- from between-country effects and, in a second step, allows country-specific slopes;
a two-step meta-analysis that fits the same model within each country, then pools coefficient estimates and tests heterogeneity.
Why both?
The multilevel model performs one-step partial pooling (stabilizing noisy country estimates via shrinkage), while the meta-analysis gives two-step, fully transparent country-by-country estimates with classic heterogeneity statistics (Q, τ², I²). Agreement between the two strengthens credibility.
Prepare Data
We use group-mean centering (suffix _cw) for individual-level predictors to estimate within-country relationships, and include country means (grand-mean centered) to estimate between-country differences. This avoids conflating contextual differences with individual-level effects.
This model allows country-specific baselines for violent intent while estimating common within-country slopes for the centered predictors and between-country effects for country means.
We now allow country-specific slopes for the individual-level predictors. This addresses the question: do predictors of violent intent vary across samples? If so, we should see non-zero variance in the random slopes and improved model fit.
model_comp <-anova(mod_ml_ri_ml, mod_ml_rs_ml) # LRT for added random slopesdf_model_comp <-as.data.frame(model_comp)df_model_comp %>%kable(., caption ="Comparing fixed and random slope model") %>%kable_styling(full_width = F, latex_options =c("hold_position", "scale-down"))
Comparing fixed and random slope model
npar
AIC
BIC
logLik
-2*log(L)
Chisq
Df
Pr(>Chisq)
mod_ml_ri_ml
35
33757.23
34010.94
-16843.61
33687.23
NA
NA
NA
mod_ml_rs_ml
62
33402.92
33852.36
-16639.46
33278.92
408.3083
27
0
Model fit improves drasticially with random slopes (not suprisingly given the many different countries). We now assess how the Jihadist sample differs from the other samples.
The following shrinkage (partial-pooling) contrasts, for each country and each predictor, the stand-alone country OLS (orinary least squares) slope with the multilevel Best Linear Unbiased Prediction (BLUP) slope which comes from the random-slopes model.
Hollow circle (OLS): the slope estimated by fitting a separate linear model within that country (no pooling).
Filled circle (BLUP): the country’s slope implied by the multilevel model = fixed effect + random deviation (i.e., after partial pooling).
Segment connecting them: the amount and direction of shrinkage from OLS → BLUP.
Long segments = noisier country estimates (often small n) get pulled more toward the overall mean slope.
Short segments = precise country estimates change little.
OLS
For each country, we fit a simple regression model just on that country’s data. The slope we get is the country-specific OLS slope.
Properties:
Uses only that country’s sample.
Very unbiased, but high variance if the country has a small sample size (noisy, unstable).
BLUB
The multilevel model’s estimate of a country’s slope = the global fixed slope plus that country’s random deviation.
Properties:
Shrinks noisy country slopes toward the overall mean slope (partial pooling).
Countries with large N or strong signal stay close to their OLS.
Countries with small N or noisy estimates get pulled toward the overall slope.
Code
# 3) Shrinkage plot: per-country OLS slopes vs. multilevel BLUPs (highlight Jihadist)# Map *_gmz terms to *_cw to align labels# 1) Recode gmz predictors to their CW counterparts (for easier matching)ols <- country_effects %>%mutate(term =str_replace(term, "_gmz$", "_cw"))# 2) Get fixed effects (all CW terms) from the random-slope modelfixef_df <- broom.mixed::tidy(mod_ml_rs, effects ="fixed") %>%filter(str_detect(term, "_cw$")) %>%select(term, fix = estimate)# 3) Get random deviations (all CW terms) per countryre_df <- broom.mixed::tidy(mod_ml_rs, effects ="ran_vals") %>%filter(str_detect(term, "_cw$")) %>%select(country = level, term, ran = estimate)# 4) Compute BLUPs = fixed slope + random deviationblup <- re_df %>%left_join(fixef_df, by ="term") %>%mutate(blup = fix + ran)# 5) Combine OLS and BLUPs into a single plotting datasetplot_df <- ols %>%rename(ols = estimate) %>%select(country, term, ols) %>%inner_join(blup %>%select(country, term, blup), by =c("country","term")) %>%mutate(group =if_else(country =="Jihadist", "Jihadist", "Other"),term_lab = term %>%str_remove("_cw$") %>%str_replace_all("_", " ") %>%str_to_title() # nice readable labels )# 6) Plot# Helper: one predictor's shrinkage plotshrinkage_plot_one <-function(df, term_label) { d <- df %>% dplyr::filter(term_lab == term_label)ggplot(d, aes(y = forcats::fct_reorder(country, blup))) +geom_segment(aes(x = ols, xend = blup, yend = country), alpha =0.5, color ="grey60") +geom_point(aes(x = ols), shape =1, size =2) +geom_point(aes(x = blup, color = group), size =2) +geom_vline(xintercept =0, linetype =2) +labs(x ="Slope",y =NULL,color =NULL,title =paste0("Partial pooling: OLS vs BLUP — ", term_label) ) +theme_minimal() +theme(axis.text.y =element_text(size =6),legend.position ="top" )}# Emit a single tabset with one tab per predictorrender_shrinkage_tabs <-function(df) { labs <-sort(unique(df$term_lab))# blank line before fenced div is IMPORTANTcat("\n\n::: {.panel-tabset}\n\n")for (lb in labs) {# each heading becomes a tab (no {.tabset} here)cat("\n")cat("#### ", lb, "\n\n", sep ="")print(shrinkage_plot_one(df, lb))cat("\n") }cat(":::\n")}# Render the tabsrender_shrinkage_tabs(plot_df)
To compare effects across countries transparently, we refitted the same country-level linear model using grand-mean standardized predictors (suffix _gmz), extracted per-country coefficients and standard errors. We now meta-analyze them with a random-effects model (REML). This yields an overall average effect and heterogeneity statistics (Q, τ², I²).
Code
meta_results <- country_effects %>% tidyr::nest(data =c(country, estimate, std.error)) %>%# one row per term dplyr::mutate(.results = purrr::map(data, ~ metafor::rma(yi = .x$estimate,sei = .x$std.error,method ="REML",slab = .x$country )) ) %>% dplyr::select(term, .results)
Code
# 4) Forest + Baujat (meta-analytic echo), adapted to generic slopes/SEs# Helper: draw ggplot-style forest make_forest_generic <-function(rma_obj,main_title ="Meta-analysis",order_by =NULL,show_prediction_interval =TRUE,study_labels =NULL,highlight ="Jihadist", # <- NEW: what to highlight (character pattern(s) or logical vector)highlight_color ="red") { # <- NEW: highlight color# robust SE: prefer $sei if present, otherwise sqrt(vi) sei_vec <-if (length(rma_obj$sei) >0) as.numeric(rma_obj$sei) elsesqrt(as.numeric(rma_obj$vi)) wi <-1/ (sei_vec^2) slab_vec <-if (!is.null(study_labels)) study_labels else rma_obj$slab# build highlight flag (case-insensitive substring match OR logical vector) hl_flag <-rep(FALSE, length(slab_vec))if (is.logical(highlight) &&length(highlight) ==length(slab_vec)) { hl_flag <- highlight } elseif (is.character(highlight) &&length(highlight) >0) {for (pat in highlight) hl_flag <- hl_flag |grepl(pat, slab_vec, ignore.case =TRUE) } df <-tibble(slab = slab_vec,yi =as.numeric(rma_obj$yi),sei = sei_vec,ci_l = yi -qnorm(.975) * sei_vec,ci_u = yi +qnorm(.975) * sei_vec,w = wi,hl =ifelse(hl_flag, "highlight", "other") )if (!is.null(order_by) && order_by %in%names(df)) { df <-arrange(df, .data[[order_by]]) }# overall pooled effect (+ CI + optional PI) overall <-tibble(slab ="Overall (REML)",yi =as.numeric(rma_obj$b),ci_l =as.numeric(rma_obj$ci.lb),ci_u =as.numeric(rma_obj$ci.ub) )# prediction interval if available pi_band <-NULLif (isTRUE(show_prediction_interval)) { pr <-tryCatch(predict(rma_obj, transf =NULL), error =function(e) NULL)if (!is.null(pr) &&!any(is.na(c(pr$pi.lb, pr$pi.ub)))) { pi_band <-c(pr$pi.lb, pr$pi.ub) } } g <-ggplot(bind_rows(df %>%select(-w, -sei), overall),aes(x = yi, y =reorder(slab, yi))) +geom_vline(xintercept =0, linetype =2, linewidth =0.4) +annotate("rect",xmin = overall$ci_l[1], xmax = overall$ci_u[1],ymin =-Inf, ymax =Inf,alpha =0.06) + {if (!is.null(pi_band))annotate("rect",xmin = pi_band[1], xmax = pi_band[2],ymin =-Inf, ymax =Inf,alpha =0.04)} +# Error bars + points ONLY for study rows, colored by highlight flaggeom_errorbarh(data = df,aes(xmin = ci_l, xmax = ci_u, color = hl),height = .15) +geom_point(data = df,aes(color = hl, size =pmin(w /max(w, na.rm =TRUE), 1)),show.legend =FALSE) +# pooled vertical + CIgeom_vline(xintercept = overall$yi[1], linewidth =0.4) +geom_vline(xintercept = overall$ci_l[1], linetype ="dashed", linewidth =0.3) +geom_vline(xintercept = overall$ci_u[1], linetype ="dashed", linewidth =0.3) +scale_color_manual(values =c(other ="black", highlight = highlight_color), guide ="none") +labs(x ="Effect (95% CI)",y =NULL,title = main_title,subtitle =paste0("Pooled = ",formatC(overall$yi[1], digits =3, format ="f")," [", formatC(overall$ci_l[1], digits =3, format ="f"),", ", formatC(overall$ci_u[1], digits =3, format ="f"), "]",if (!is.null(pi_band)) paste0(" | PI [",formatC(pi_band[1], digits =3, format ="f"), ", ",formatC(pi_band[2], digits =3, format ="f"), "]" ) else"" )) +theme_bw() +theme(panel.grid.minor =element_blank())return(g)}# Helper: draw funnel (generic) with highlightingmake_funnel_generic <-function(rma_obj,main_title ="Funnel plot",refline =0,levels =c(90, 95, 99),grayscale =c("white", "gray55", "gray75"),legend_pos ="topright",highlight ="Jihadist", # <- NEWhighlight_color ="red") { # <- NEW# Ensure we have yi/vi/sei yi <-as.numeric(rma_obj$yi) vi <-if (!is.null(rma_obj$vi)) as.numeric(rma_obj$vi) else (as.numeric(rma_obj$sei))^2 sei <-sqrt(vi) slab_vec <- rma_obj$slab# highlight flag (same rules as forest) hl_flag <-rep(FALSE, length(slab_vec))if (is.logical(highlight) &&length(highlight) ==length(slab_vec)) { hl_flag <- highlight } elseif (is.character(highlight) &&length(highlight) >0) {for (pat in highlight) hl_flag <- hl_flag |grepl(pat, slab_vec, ignore.case =TRUE) }# Trim-and-fill on the fitted model tf <-tryCatch(trimfill(rma_obj), error =function(e) rma_obj)# PET / PEESE regressions pet <-tryCatch(rma(yi = yi, sei = sei, mods =~ sei), error =function(e) NULL) peese <-tryCatch(rma(yi = yi, sei = sei, mods =~I(sei^2)), error =function(e) NULL) reml_b <-as.numeric(rma_obj$b)funnel(tf, level = levels, shade = grayscale, refline = refline,legend = legend_pos,main =paste0( main_title, "\n","REML = ", formatC(reml_b, digits =2, format ="f"),if (!is.null(pet)) paste0(", PET = ", formatC(pet$b[1], digits =2, format ="f")) else"",if (!is.null(peese)) paste0(", PEESE = ", formatC(peese$b[1], digits =2, format ="f")) else"",", n = ", length(yi) ))# pooled markerabline(v = reml_b, lty ="dashed")points(x = reml_b, y =0, cex =1.4, pch =17)# PEESE curve + pointif (!is.null(peese) &&length(peese$b) >=2) { sei_grid <-seq(0, max(sei, na.rm =TRUE), length.out =1000) vi_grid <- sei_grid^2 yi_grid <-as.numeric(peese$b[1]) +as.numeric(peese$b[2]) * vi_gridlines(x = yi_grid, y = sei_grid)points(x =as.numeric(peese$b[1]), y =0, cex =1.3, pch =5) }# PET line + point + CI segmentif (!is.null(pet) &&length(pet$b) >=2) {abline(a =-as.numeric(pet$b[1]) /as.numeric(pet$b[2]),b =1/as.numeric(pet$b[2]))points(x =as.numeric(pet$b[1]), y =0, cex =1.2)segments(x0 =as.numeric(pet$ci.lb[1]), y0 =0,x1 =as.numeric(pet$ci.ub[1]), y1 =0,lty ="dashed") }# OVERDRAW highlighted studies in red so they popif (any(hl_flag, na.rm =TRUE)) {points(x = yi[hl_flag], y = sei[hl_flag], pch =19, col = highlight_color, cex =1.2) }invisible(list(trimfill = tf, PET = pet, PEESE = peese))}
Code
plot_meta_suite_tabbed <-function(meta_results,add_funnel =TRUE,highlight ="Jihadist",highlight_color ="red") {# Open ONE tabset containercat("::: {.panel-tabset}\n\n") terms <-unique(meta_results$term)for (tm in terms) { mobj <- meta_results %>% dplyr::filter(term == tm) %>% dplyr::pull(.results) %>% .[[1]] pretty <- tm %>% stringr::str_remove("_gmz$") %>% stringr::str_replace_all("_", " ") %>% stringr::str_to_title()# Each of these headings becomes a tab (NO {.tabset} here)cat("\n")cat("#### ", pretty, "\n\n", sep ="")# Forest (ggplot)print(make_forest_generic( mobj,main_title =paste0("Forest: ", pretty),order_by =NULL,show_prediction_interval =TRUE,highlight = highlight,highlight_color = highlight_color ))if (isTRUE(add_funnel)) {# Make sure the base-graphics funnel is a separate figure graphics::plot.new()make_funnel_generic( mobj,main_title =paste0("Funnel: ", pretty),highlight = highlight,highlight_color = highlight_color ) }cat("\n") # spacing between tabs }# Close the tabset containercat(":::\n")}# Call itplot_meta_suite_tabbed(meta_results, add_funnel =TRUE)