Doing_Meta_Analysis_in_R.tex

\documentclass[]{book}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\usepackage{fixltx2e} % provides \textsubscript
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
\else % if luatex or xelatex
  \ifxetex
    \usepackage{mathspec}
  \else
    \usepackage{fontspec}
  \fi
  \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
\fi
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\usepackage[margin=1in]{geometry}
\usepackage{hyperref}
\hypersetup{unicode=true,
            pdftitle={Doing Meta-Analysis in R and exploring heterogeneity using metaforest},
            pdfauthor={Caspar van Lissa¹},
            pdfborder={0 0 0},
            breaklinks=true}
\urlstyle{same}  % don't use monospace font for urls
\usepackage{natbib}
\bibliographystyle{apalike}
\usepackage{color}
\usepackage{fancyvrb}
\newcommand{\VerbBar}{|}
\newcommand{\VERB}{\Verb[commandchars=\\\{\}]}
\DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}}
% Add ',fontsize=\small' for more characters per line
\usepackage{framed}
\definecolor{shadecolor}{RGB}{248,248,248}
\newenvironment{Shaded}{\begin{snugshade}}{\end{snugshade}}
\newcommand{\AlertTok}[1]{\textcolor[rgb]{0.94,0.16,0.16}{#1}}
\newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.77,0.63,0.00}{#1}}
\newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\BuiltInTok}[1]{#1}
\newcommand{\CharTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\CommentTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}}
\newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}}
\newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{#1}}
\newcommand{\DecValTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\ErrorTok}[1]{\textcolor[rgb]{0.64,0.00,0.00}{\textbf{#1}}}
\newcommand{\ExtensionTok}[1]{#1}
\newcommand{\FloatTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\ImportTok}[1]{#1}
\newcommand{\InformationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}}
\newcommand{\NormalTok}[1]{#1}
\newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.81,0.36,0.00}{\textbf{#1}}}
\newcommand{\OtherTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{#1}}
\newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}}
\newcommand{\RegionMarkerTok}[1]{#1}
\newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\StringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\VariableTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\WarningTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\usepackage{longtable,booktabs}
\usepackage{graphicx,grffile}
\makeatletter
\def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth\else\Gin@nat@width\fi}
\def\maxheight{\ifdim\Gin@nat@height>\textheight\textheight\else\Gin@nat@height\fi}
\makeatother
% Scale images if necessary, so that they will not overflow the page
% margins by default, and it is still possible to overwrite the defaults
% using explicit options in \includegraphics[width, height, ...]{}
\setkeys{Gin}{width=\maxwidth,height=\maxheight,keepaspectratio}
\IfFileExists{parskip.sty}{%
\usepackage{parskip}
}{% else
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}
}
\setlength{\emergencystretch}{3em}  % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{5}
% Redefines (sub)paragraphs to behave more like sections
\ifx\paragraph\undefined\else
\let\oldparagraph\paragraph
\renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
\fi
\ifx\subparagraph\undefined\else
\let\oldsubparagraph\subparagraph
\renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
\fi

%%% Use protect on footnotes to avoid problems with footnotes in titles
\let\rmarkdownfootnote\footnote%
\def\footnote{\protect\rmarkdownfootnote}

%%% Change title format to be more compact
\usepackage{titling}

% Create subtitle command for use in maketitle
\providecommand{\subtitle}[1]{
  \posttitle{
    \begin{center}\large#1\end{center}
    }
}

\setlength{\droptitle}{-2em}

  \title{Doing Meta-Analysis in R and exploring heterogeneity using metaforest}
    \pretitle{\vspace{\droptitle}\centering\huge}
  \posttitle{\par}
  \subtitle{A Hands-on Guide}
  \author{Caspar van Lissa¹}
    \preauthor{\centering\large\emph}
  \postauthor{\par}
      \predate{\centering\large\emph}
  \postdate{\par}
    \date{¹Utrecht University}

\usepackage{booktabs}
\usepackage{amsthm}
\makeatletter
\def\thm@space@setup{%
  \thm@preskip=8pt plus 2pt minus 4pt
  \thm@postskip=\thm@preskip
}
\makeatother

\begin{document}
\maketitle

{
\setcounter{tocdepth}{1}
\tableofcontents
}
\hypertarget{about-this-guide}{%
\chapter{About this Guide}\label{about-this-guide}}

\includegraphics{coverbild.jpg}

\begin{rmdinfo}
This guide shows you how to conduct Meta-Analyses in R from scratch. The
focus of this guide is primarily on clinical outcome research in
psychology. It is designed for staff and collaborators of the
\href{https://www.protectlab.org}{\textbf{PROTECT Lab}}, which is headed
by \textbf{Dr.~David D. Ebert}.
\end{rmdinfo}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\textbf{The guide will show you how to:}

\begin{itemize}
\tightlist
\item
  Get \textbf{R} and \textbf{RStudio} set for your Meta-Analysis
\item
  Get your data into R
\item
  \textbf{Prepare your data} for the meta-analysis
\item
  Perform \textbf{fixed-effect} and \textbf{random-effects} meta-analysis using the \texttt{meta} and \texttt{metafor}packages
\item
  Analyse the \textbf{heterogeneity} of your results
\item
  Tackle heterogeneity using \textbf{subgroup analyses} and \textbf{meta-regression}
\item
  Check if \textbf{selective outcome reporting (publication bias)} is a present in your data
\item
  Control for selective outcome reporting and publication bias
\item
  Analyse the \textbf{risk of bias} in your data
\item
  Do advanced types of meta-analyses, such as

  \begin{itemize}
  \tightlist
  \item
    \textbf{network analyses} or
  \item
    meta-analyses with \textbf{more than one outcome}
  \end{itemize}
\end{itemize}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\textbf{What this guide will not cover}

Although this guide will provide some information on the statistics behind meta-analysis, it will not give you an \textbf{in-depth introduction} into how meta-analyses are calculated statistically.

It is also beyond the scope of this guide to advise in detail which meta-analytical strategy is suited best in which contexts, and on how the search, study inclusion and reporting of meta-analyses should be conducted. The \href{http://handbook-5-1.cochrane.org/}{\emph{Cochrane Handbook for Systematic Reviews of Interventions}}, however, should be a great source to find more information on these topics.

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\textbf{Generally, there a two other sources to recommended when conducting Meta-Analyses:}

\begin{itemize}
\tightlist
\item
  If you're looking for a easily digestable, hands-on introduction on how Meta-Analyses are conducted, we can recommend \textbf{Pim Cuijpers' online courses on Meta-Analysis}. The courses are freely available on YouTube. To have a look, click \href{https://www.youtube.com/watch?v=pP7_VBrG_TY\&list=PL-h5cI5Bkvt0J-O0kq_9J9_aksWFPgR7s}{here}.
\item
  If you're interested in more details on how to conduct Meta-Analyses in R, you can either have a look at Wolfgang Viechtbauer's page for the \texttt{metafor} package (\href{http://metafor-project.org}{Link}). Or you can consult a book on the \texttt{meta} package which was recently published \citep{schwarzer2015meta}.
\end{itemize}

\begin{rmdinfo}
\textbf{How to get the R code for this guide}

\includegraphics{githublogo.png}

All code behind this book is available online on \textbf{GitHub}. We
have created a website containing a \textbf{download link} for all
codes, and a \textbf{quick guide} on how to get the code running on your
computer. The site can be found
\href{https://mathiasharrer.github.io/Doing-Meta-Analysis-in-R/}{here}.
\end{rmdinfo}

\textbf{How to cite this guide}

Harrer, M., Cuijpers, P. \& Ebert, D. D. (2019). Doing Meta-Analysis in R: A Hand-on Guide. \url{https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/}.
\href{https://zenodo.org/badge/latestdoi/152492192}{\includegraphics{https://zenodo.org/badge/152492192.svg}}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\textbf{To get started, proceed to the next chapter!}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{rstudio-basics}{%
\chapter{RStudio \& Basics}\label{rstudio-basics}}

\includegraphics{chap2.jpg}

\begin{rmdinfo}
Before we start with our meta-analysis, we have to download and prepare
a \textbf{computer program} which allows us to use \emph{R} for
programming.

Probably the best option for this at the moment is \textbf{RStudio}.
This program gives us a user interface which makes it easier to overlook
our data, packages and output. The best part is that RStudio is
\textbf{completely free} and can be downloaded anytime.

In this Chapter, we'll focus on how you can install RStudio on your
computer. We'll also provide some general information on R, and how you
can get help if you get error messages.

If you already have RStudio installed on your computer, and if you're an
experienced R user already, all of this might be nothing new for you.
You may \textbf{skip} this chapter then.

Especially if you have \textbf{never used R before, we would like to
consider this Chapter essential}, as it gives you some input on how R
works, and how we can use it for our data analyses.
\end{rmdinfo}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{RStudio}{%
\section{Getting RStudio to run on your computer}\label{RStudio}}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-6-1.pdf}

As a prerequisite for this guide, you need to have \textbf{RStudio} and a few essential \textbf{R packages} installed.

\textbf{You have to follow these steps to get your RStudio set.}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Download RStudio on the \textbf{RStudio} Website (\href{https://www.rstudio.com/products/rstudio/download/}{Link}). It's free!
\item
  If you do not have \textbf{R} installed yet, you will have to install the latest R Version before you can use RStudio. You can get the latest R version \href{https://cran.r-project.org/bin/windows/base/}{here}.
\item
  Once RStudio is running, open the \textbf{Console} on the bottom left corner of your screen.
\item
  We will now install a few packages using R Code. Here's an overview of the packages, and why we need them:
\end{enumerate}

Package

Description

tidyverse

This is a large package containing various functions to tidy up your data

meta

This package is a so-called wrapper for many meta-analytic functions and makes it easy to code different meta-analyses

metaforest

This package is used to explore heterogeneity using the random forests algorithm. It also installs the R-package `metafor', which is one of the most commonly used meta-analysis packages.

5. To install these packages, we use the \texttt{install.packages()} function in R. One package after another, our code should look like this:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{install.packages}\NormalTok{(}\StringTok{"tidyverse"}\NormalTok{)}
\KeywordTok{install.packages}\NormalTok{(}\StringTok{"meta"}\NormalTok{)}
\KeywordTok{install.packages}\NormalTok{(}\StringTok{"metaforest"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{rmdachtung}
Don't forget to put the packages in \texttt{""}.

Otherwise, you will get an error message.
\end{rmdachtung}

\textbf{Continue to the next chapter.}

\emph{Additional, non-essential information on RStudio and troubleshooting below.}

\hypertarget{running-r-code}{%
\subsection{Running R Code}\label{running-r-code}}

In order to get the most out of this guide, it's helpful (but not essential) if you have some programming experience already. If you've never programmed before, you might find \textbf{\emph{Hands-On Programming with R}} \citep{grolemund2014hands} to be a useful primer.

There are three things you need to run the code: \textbf{R}, \textbf{RStudio}, and collection of \textbf{R packages}. Packages are the fundamental units of reproducible R code. They include reusable functions, the documentation that describes how to use them, and sample data.

Gladly, once you've reached this point successfully, these three things are set already. Nevertheless, we will have to install and load a few new packages at some place in this guide, for which you can use the \texttt{install.packages()} the same way as you did before.

Throughout the guide, a consistent set of conventions is used to refer to code:

\begin{itemize}
\tightlist
\item
  Functions are in a code font and followed by parentheses, like
  \texttt{sum()} or \texttt{mean()}.
\item
  Other R objects (like data or function arguments) are in a code
  font, without parentheses, like \texttt{seTE} or \texttt{method.tau}.
\item
  Sometimes, we'll use the package name followed by two colons, like
  \texttt{meta::metagen()}. This is also valid R code. This is used so as to not confuse the functions of different packages with the same name.
\end{itemize}

\hypertarget{getting-help}{%
\subsection{Getting Help}\label{getting-help}}

As you start to apply the techniques described in this guide to your data you will soon find questions that the guide does not answer. This section describes a few tips on how to get help.

\begin{itemize}
\tightlist
\item
  If you get stuck, start with \textbf{Google}. Typically, adding ``R'' to a search is enough to restrict it to relevant results: if the search isn't useful, it often means that there aren't any R-specific results available. Google is particularly useful for error messages. If you get an error message and you have no idea what it means, try googling it. Chances are that someone else has been confused by it in the past, and there will
  be help somewhere on the web. (If the error message isn't in English,
  run \texttt{Sys.setenv(LANGUAGE\ =\ "en")} and re-run the code; you're
  more likely to find help for English error messages.)
\item
  If Google doesn't help, try \href{https://stackoverflow.com}{stackoverflow}. Start by spending a little time searching for an existing answer; including {[}R{]} restricts your search to questions and answers that use R.
\item
  Lastly, if you stumble upon an error (or typos!) in this guide's text or R syntax, feel free to contact \textbf{Caspar van Lissa} at \textbf{\href{mailto:c.j.vanlissa@uu.nl}{\nolinkurl{c.j.vanlissa@uu.nl}}}.
\end{itemize}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{getting-your-data-into-r}{%
\chapter{Getting your data into R}\label{getting-your-data-into-r}}

\includegraphics{chappool.jpg}

\begin{rmdinfo}
This chapter will tell you about how you can \textbf{import} your effect
size data in RStudio. We will also show you a few commands which make it
easier to \textbf{manipulate data} directly in R.

Data preparation can be tedious and exhausting at times, but it is the
backbone of all later steps when doing meta-analyses in R. We therefore
have to pay \textbf{close attention} to preparing the data correctly
before we can proceed.
\end{rmdinfo}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{data-preparation-in-excel}{%
\section{Data preparation in Excel}\label{data-preparation-in-excel}}

\hypertarget{excel_preparation}{%
\subsection{Setting the columns of the excel spreadsheet}\label{excel_preparation}}

To conduct Meta-Analyses in R, you need to have your study data prepared. For a standard meta-analysis, the following information is needed for every study.

\begin{itemize}
\tightlist
\item
  The \textbf{names} of the individual studies, so that they can be easily identified later on. Usually, the first author and publication year of a study is used for this (e.g. ``Ebert et al., 2018'')
\item
  The \textbf{Mean} of both the Intervention and the Control group at the same assessment point
\item
  The \textbf{Standard Deviation} of both the Intervention and the Control group at the same assessment point
\item
  The \textbf{number of participants (N)} in each group of the trial
\item
  If you want to have a look at differences between various study subgroups later on, you also need a \textbf{subgroup code} for each study which signifies to which subgroup it belongs. For example, if a study was conducted in children, you might give it the subgroup code ``children''.
\end{itemize}

As per usual, such data is stored in \textbf{EXCEL spreadsheets}. We recommend to store your data there, because this makes it very easy to import data into RStudio.

However, it is very important how you \textbf{name the columns of your spreadsheet}. If you name the columns of your sheet adequately in EXCEL already, you can save a lot of time because your data doesn't have to be transformed in RStudio later on.

\textbf{Here is how you should name the data columns in your EXCEL spreadheet containing your Meta-Analysis data}

Column

Description

Author

This signifies the column for the study label (i.e., the first author)

Me

The Mean of the experimental/intervention group

Se

The Standard Deviation of the experimental/intervention group

Mc

The Mean of the control group

Sc

The Standard Deviation of the control group

Ne

The number of participants in the experimental/intervention group

Nc

The number of participats in the control group

Subgroup

This is the label for one of your Subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g.~population). In this column, each study should then be given an subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique

Note that it \textbf{doesn't matter how these columns are ordered in your EXCEL spreadsheet}. They just have to be labeled correctly.

There's also no need to \textbf{format} the columns in any way. If you type the column name in the first line of you spreadsheet, R will automatically detect it as a column name.

\begin{rmdachtung}
It's also important to know that the import \textbf{will distort letters
like ä,ü,ö,á,é,ê, etc}. So be sure to transform them to ``normal''
letters before you proceed.
\end{rmdachtung}

\hypertarget{setting-the-columns-of-your-sheet-if-you-have-calculated-the-effect-sizes-of-each-study-already}{%
\subsection{Setting the columns of your sheet if you have calculated the effect sizes of each study already}\label{setting-the-columns-of-your-sheet-if-you-have-calculated-the-effect-sizes-of-each-study-already}}

If you have \textbf{already calculated the effect sizes for each study on your own}, for example using \emph{Comprehensive Meta-Analysis} or \emph{RevMan}, there's another way to prepare your data which makes things a little easier. In this case, you only have to include the following columns:

Column

Description

Author

This signifies the column for the study label (i.e., the first author)

TE

The calculated effect size of the study (either Cohen's d or Hedges' g, or some other form of effect size

seTE

The Standard Error (SE) of the calculated effect

Subgroup

This is the label for one of your Subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g.~population). In this column, each study should then be given an subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{import_excel}{%
\section{Importing the Spreadsheet into Rstudio}\label{import_excel}}

To get our data into R, we need to \textbf{save our data in a format and at a place where RStudio can open it}.

\hypertarget{saving-the-data-in-the-right-format}{%
\subsection{Saving the data in the right format}\label{saving-the-data-in-the-right-format}}

Generelly, finding the right format to import EXCEL can be tricky.

\begin{itemize}
\tightlist
\item
  If you're using a PC or Mac, it is advised to save your EXCEL sheet as a \textbf{comma-separated-values (.csv) file}. This can be done by clicking on ``save as'' and then choosing (.csv) as the output format in EXCEL.
\item
  With some PCs, RStudios might not be able to open such files, or the files might be distorted. In this case, you can also try to save the sheet as a normal \textbf{.xslx} EXCEL-file and try if this works.
\end{itemize}

\hypertarget{saving-the-data-in-your-working-directory}{%
\subsection{Saving the data in your working directory}\label{saving-the-data-in-your-working-directory}}

To function properly, you have to set a working directory for RStudio first. The working directory is a \textbf{folder on your computer from which RStudio can use data, and in which output it saved}.

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Therefore, create a folder on your computer and give it a meaningful name (e.g. ``My Meta-Analysis'').
\item
  Save your spreadsheet in the folder
\item
  Set the folder as your working directory. This can be done in RStudio on the \textbf{bottom-right corner of your screen}. Under the tile \textbf{``Files''}, search for the folder on your computer, and open it.
\item
  Once you've opened your folder, the file you just saved there should be in there.
\item
  Now that you've opened the folder, click on the \textbf{little gear wheel on top of the pane}
\end{enumerate}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-14-1.pdf}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{5}
\tightlist
\item
  Then click on ``\textbf{Set as working directory}''
\end{enumerate}

\textbf{Your file, and the working directory, are now where they should be!}

\hypertarget{loading-the-data}{%
\subsection{Loading the data}\label{loading-the-data}}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  To import the data, simply \textbf{click on the file} in the bottom-right pane. Then click on \textbf{import dataset\ldots{}}
\item
  An \textbf{import assistant} should now pop up, which is also loading a preview of your data. This can be time-consuming sometimes, so you can skip this step if you want to, and klick straight on \textbf{``import''}
\end{enumerate}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-15-1.pdf}

As you can see, the on the top-right pane \textbf{Environment}, your file is now listed as a data set in your RStudio environment.

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{2}
\tightlist
\item
  I also want to give my data a shorter name (``madata''). To rename it, i use the following code:
\end{enumerate}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata<-Meta_Analysis_Data}
\end{Highlighting}
\end{Shaded}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{3}
\tightlist
\item
  Now, let's have a look at the \textbf{structure of my data} using the \texttt{str()} function
\end{enumerate}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(madata)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Classes 'tbl_df', 'tbl' and 'data.frame':    18 obs. of  17 variables:
##  $ Author               : chr  "Call et al." "Cavanagh et al." "DanitzOrsillo" "de Vibe et al." ...
##  $ TE                   : num  0.709 0.355 1.791 0.182 0.422 ...
##  $ seTE                 : num  0.261 0.196 0.346 0.118 0.145 ...
##  $ RoB                  : chr  "low" "low" "high" "low" ...
##  $ Control              : chr  "WLC" "WLC" "WLC" "no intervention" ...
##  $ intervention duration: chr  "short" "short" "short" "short" ...
##  $ intervention type    : chr  "mindfulness" "mindfulness" "ACT" "mindfulness" ...
##  $ population           : chr  "undergraduate students" "students" "undergraduate students" "undergraduate students" ...
##  $ type of students     : chr  "psychology" "general" "general" "general" ...
##  $ prevention type      : chr  "selective" "universal" "universal" "universal" ...
##  $ gender               : chr  "female" "mixed" "mixed" "mixed" ...
##  $ mode of delivery     : chr  "group" "online" "group" "group" ...
##  $ ROB streng           : chr  "high" "low" "high" "low" ...
##  $ ROB superstreng      : chr  "high" "high" "high" "low" ...
##  $ compensation         : chr  "none" "none" "voucher/money" "voucher/money" ...
##  $ instruments          : chr  "DASS" "PSS" "DASS" "other" ...
##  $ guidance             : chr  "f2f" "self-guided" "f2f" "f2f" ...
\end{verbatim}

Although this output looks kind of messy, it's already very informative. It shows the structure of my data. In this case, i used data for which the effect sizes were already calculated. This is why the variables \textbf{TE} and \textbf{seTE} appear. I also see plenty of other variables, which correspond to the subgroups which were coded for this dataset.

\textbf{Here's a (shortened) table created for my data}

Author

TE

seTE

RoB

Control

intervention duration

intervention type

Call et al.

0.7091362

0.2608202

low

WLC

short

mindfulness

Cavanagh et al.

0.3548641

0.1963624

low

WLC

short

mindfulness

DanitzOrsillo

1.7911700

0.3455692

high

WLC

short

ACT

de Vibe et al.

0.1824552

0.1177874

low

no intervention

short

mindfulness

Frazier et al.

0.4218509

0.1448128

low

information only

short

PCI

Frogeli et al.

0.6300000

0.1960000

low

no intervention

short

ACT

Gallego et al.

0.7248838

0.2246641

high

no intervention

long

mindfulness

Hazlett-Stevens \& Oren

0.5286638

0.2104609

low

no intervention

long

mindfulness

Hintz et al.

0.2840000

0.1680000

low

information only

short

PCI

Kang et al.

1.2750682

0.3371997

low

no intervention

long

mindfulness

Kuhlmann et al.

0.1036082

0.1947275

low

no intervention

short

mindfulness

Lever Taylor et al.

0.3883906

0.2307689

low

WLC

long

mindfulness

Phang et al.

0.5407398

0.2443133

low

no intervention

short

mindfulness

Rasanen et al.

0.4261593

0.2579379

low

WLC

short

ACT

Ratanasiripong

0.5153969

0.3512737

high

no intervention

short

mindfulness

Shapiro et al.

1.4797260

0.3152817

low

WLC

long

mindfulness

SongLindquist

0.6125782

0.2266834

high

WLC

long

mindfulness

Warnecke et al.

0.6000000

0.2490000

low

information only

long

mindfulness

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{data-manipulation}{%
\section{Data manipulation}\label{data-manipulation}}

Now that we have the Meta-Analysis data in RStudio, let's do a \textbf{few manipulations with the data}. These functions might come in handy when were conducting analyses later on.

Going back to the output of the \texttt{str()} function, we see that this also gives us details on the type of column data we have stored in our data. There a different abbreviations signifying different types of data.

Abbreviation

Type

Description

num

Numerical

This is all data stored as numbers (e.g.~1.02)

chr

Character

This is all data stored as words

log

Logical

These are variables which are binary, meaning that they signify that a condition is either TRUE or FALSE

factor

Factor

Factors are stored as numbers, with each number signifying a different level of a variable. A possible factor of a variable might be 1 = low, 2 = medium, 3 = high

\hypertarget{convertfactors}{%
\subsection{Converting to factors}\label{convertfactors}}

Let's say we have the subgroup \textbf{Risk of Bias} (in which the Risk of Bias rating is coded), and want it to be a factor with two different levels: ``low'' and ``high''.

To do this, we need to the variable \texttt{ROB} to be a factor. However, this variable is currently stored as a character (\texttt{chr}). We can have a look at this variable by typing the name of our dataset, then adding the selector \texttt{\$} and then adding the variable we want to have a look at.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata}\OperatorTok{$}\NormalTok{ROB}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] "high" "low"  "high" "low"  "low"  "low"  "high" "low"  "low"  "low" 
## [11] "high" "low"  "low"  "low"  "high" "high" "high" "low"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(madata}\OperatorTok{$}\NormalTok{ROB)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  chr [1:18] "high" "low" "high" "low" "low" "low" "high" "low" "low" ...
\end{verbatim}

We can see now that \texttt{ROB}is indeed a \textbf{character} type variable, which contains only two words: ``low'' and ``high''. We want to convert this to a \textbf{factor} variable now, which has only two levels, low and high. To do this, we use the \texttt{factor()} function.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata}\OperatorTok{$}\NormalTok{ROB<-}\KeywordTok{factor}\NormalTok{(madata}\OperatorTok{$}\NormalTok{ROB)}
\NormalTok{madata}\OperatorTok{$}\NormalTok{ROB}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] high low  high low  low  low  high low  low  low  high low  low  low 
## [15] high high high low 
## Levels: high low
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(madata}\OperatorTok{$}\NormalTok{ROB)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  Factor w/ 2 levels "high","low": 1 2 1 2 2 2 1 2 2 2 ...
\end{verbatim}

We now see that the variable has been \textbf{converted to a factor with the levels ``high'' and ``low''}.

\hypertarget{converting-to-logicals}{%
\subsection{Converting to logicals}\label{converting-to-logicals}}

Now lets have a look at the \textbf{intervention type} subgroup variable. This variable is currently stores as a character \texttt{chr} too.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] "mindfulness" "mindfulness" "ACT"         "mindfulness" "PCI"        
##  [6] "ACT"         "mindfulness" "mindfulness" "PCI"         "mindfulness"
## [11] "mindfulness" "mindfulness" "mindfulness" "ACT"         "mindfulness"
## [16] "mindfulness" "mindfulness" "mindfulness"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  chr [1:18] "mindfulness" "mindfulness" "ACT" "mindfulness" "PCI" ...
\end{verbatim}

Let's say we want a variable which only contains information if a study way a mindfulness intervention or not. A logical is very well suited for this. To convert the data to logical, we use the \texttt{as.logical} function. We will create a new variable containing this information called \texttt{intervention.type.logical}. To tell R what to count as \texttt{TRUE} and what as \texttt{FALSE}, we have to define the specific intervention type using the \texttt{==} command.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{intervention.type.logical<-}\KeywordTok{as.logical}\NormalTok{(madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}\OperatorTok{==}\StringTok{"mindfulness"}\NormalTok{)}
\NormalTok{intervention.type.logical}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]  TRUE  TRUE FALSE  TRUE FALSE FALSE  TRUE  TRUE FALSE  TRUE  TRUE
## [12]  TRUE  TRUE FALSE  TRUE  TRUE  TRUE  TRUE
\end{verbatim}

We see that R has converted the character information into trues and falses for us. To check if this was done correctly, let's compare the original and the new variable

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{n<-}\KeywordTok{data.frame}\NormalTok{(intervention.type.logical,madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}\NormalTok{)}
\NormalTok{names<-}\KeywordTok{c}\NormalTok{(}\StringTok{"New"}\NormalTok{, }\StringTok{"Original"}\NormalTok{)}
\KeywordTok{colnames}\NormalTok{(n)<-names}
\KeywordTok{kable}\NormalTok{(n)}
\end{Highlighting}
\end{Shaded}

New

Original

TRUE

mindfulness

TRUE

mindfulness

FALSE

ACT

TRUE

mindfulness

FALSE

PCI

FALSE

ACT

TRUE

mindfulness

TRUE

mindfulness

FALSE

PCI

TRUE

mindfulness

TRUE

mindfulness

TRUE

mindfulness

TRUE

mindfulness

FALSE

ACT

TRUE

mindfulness

TRUE

mindfulness

TRUE

mindfulness

TRUE

mindfulness

\hypertarget{select}{%
\subsection{Selecting specific studies}\label{select}}

It may often come in handy to \textbf{select certain studies for further analyses}, or to \textbf{exclude some studies in further analyses} (e.g., if they are outliers).

To do this, we can use the \texttt{filter} function in the \texttt{dplyr}package, which is part of the \texttt{tidyverse} package we installed before.

So, let's load the package first.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(dplyr)}
\end{Highlighting}
\end{Shaded}

Let's say we want to do a Meta-Analysis with three studies in our dataset only. To do this, we need to create a new dataset containing only these studies using the \texttt{dplyr::filter()} function. The \texttt{dplyr::} part is necessary as there is more than one `\texttt{filter} function in R, and we want to use to use the one of the \texttt{dplyr}package.

Let's say we want to have the studies by \textbf{Cavanagh et al.}, \textbf{Frazier et al.} and \textbf{Phang et al.} stored in another dataset, so we can conduct analyses only for these studies.

The R code to store these three studies in a new dataset called \texttt{madata.new} looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata.new<-dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(madata,Author }\OperatorTok{%in%}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"Cavanagh et al."}\NormalTok{,}
                                               \StringTok{"Frazier et al."}\NormalTok{,}
                                               \StringTok{"Phang et al."}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Note that the \texttt{\%in\%}-Command tells the \texttt{filter} function to search for exactly the three cases we defined in the variable \texttt{Author}.
Now, let's have a look at the new data \texttt{madata.new} we just created.

Author

TE

seTE

RoB

Control

intervention duration

intervention type

population

type of students

prevention type

gender

mode of delivery

ROB streng

ROB superstreng

compensation

instruments

guidance

ROB

Cavanagh et al.

0.3548641

0.1963624

low

WLC

short

mindfulness

students

general

universal

mixed

online

low

high

none

PSS

self-guided

low

Frazier et al.

0.4218509

0.1448128

low

information only

short

PCI

students

psychology

universal

mixed

online

low

low

credit

PSS

reminders

low

Phang et al.

0.5407398

0.2443133

low

no intervention

short

mindfulness

students

medical studens

selective

mixed

group

low

low

none

PSS

f2f

low

Note that the function can also be used for any other type of data and variable. We can also use it to e.g., only select studies which were coded as being a mindfulness study

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata.new.mf<-dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(madata,}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`} \OperatorTok{%in%}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"mindfulness"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

We can also use the \texttt{dplyr::filter()} function to exclude studies from our dataset. To do this, we only have to add \texttt{!} in front of the variable we want to use for filtering.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata.new.excl<-dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(madata,}\OperatorTok{!}\NormalTok{Author }\OperatorTok{%in%}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"Cavanagh et al."}\NormalTok{,}
                                                     \StringTok{"Frazier et al."}\NormalTok{,}
                                                     \StringTok{"Phang et al."}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\hypertarget{changing-cell-values}{%
\subsection{Changing cell values}\label{changing-cell-values}}

Sometimes, even when preparing your data in EXCEL, you might want to \textbf{change values in RStudio once you have imported your data}.

To do this, we have to select a cell in our data frame in RStudio. This can be done by adding \texttt{{[}x,y{]}} to our dataset name, where \textbf{x} signifies the number of the \textbf{row} we want to select, and \textbf{y} signifies the number of the \textbf{column}.

To see how this works, let's select a variable using this command first:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata[}\DecValTok{6}\NormalTok{,}\DecValTok{1}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 1
##   Author        
##   <chr>         
## 1 Frogeli et al.
\end{verbatim}

We now see the \textbf{6th study} in our dataframe, and the value of this study for \textbf{Column 1 (the author name)} is displayed. Let's say we had a typo in this name and want to have it changed. In this case, we have to give this exact cell a new value.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata[}\DecValTok{6}\NormalTok{,}\DecValTok{1}\NormalTok{]<-}\StringTok{"Frogelli et al."}
\end{Highlighting}
\end{Shaded}

Let's check if the name has changed.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{madata[}\DecValTok{6}\NormalTok{,}\DecValTok{1}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 1
##   Author         
##   <chr>          
## 1 Frogelli et al.
\end{verbatim}

You can also use this function to change any other type of data, including numericals and logicals. Only for characters, you have to put the values you want to insert in \texttt{""}.

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{pool}{%
\chapter{Pooling Effect Sizes}\label{pool}}

\includegraphics{pooling.jpg}

Now, let's get to the core of every Meta-Analysis: \textbf{pooling your effect sizes} to get one overall effect size estimate of the studies.

\begin{rmdinfo}
When pooling effect sizes in Meta-Analysis, there are two approaches
which we can use: the \textbf{Fixed-Effect-Model}, or the
\textbf{Random-Effects-Model} {[}@borenstein2011{]}. There is an
extensive debate on which model fits best in which context
{[}@fleiss1993review{]}, with no clear consensus in sight. Although it
has been recommended to \textbf{only resort to the
Random-Effects-Pooling model} in clinical psychology and the health
sciences {[}@cuijpers2016meta{]}, we will describe how to conduct both
in R here.

Both of these models only require an \textbf{effect size}, and a
\textbf{dispersion (variance)} estimate for each study, of which the
inverse is taken. This is why the methods are often called
\textbf{generic inverse-variance methods}.
\end{rmdinfo}

We will describe in-depth how to conduct meta-analyses in R with \textbf{continuous variables} (such as effect sizes), as these are the most common ones in psychology and the health science field. Later on, we will present briefly how to do meta-analyses with \textbf{binary outcome} data too, which might be important if you're focusing on prevention trials.

For these meta-analyses, we'll use the \texttt{meta} package \citep{schwarzer2007meta}. In \protect\hyperlink{RStudio}{Section 2.1}, we showed how to install the package. Now, load the package from your library to proceed.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(meta)}
\end{Highlighting}
\end{Shaded}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{fixed}{%
\section{Fixed-Effects-Model}\label{fixed}}

\hypertarget{pre.calc}{%
\subsection{Pre-calculated effect size data}\label{pre.calc}}

\begin{rmdinfo}
\textbf{The idea behind the fixed-effects-model}

The fixed-effects-model assumes that all studies along with their effect
sizes stem from a single homogeneous population {[}@borenstein2011{]}.
To calculate the overall effect, we therefore average all effect sizes,
but give studies with greater precision a higher weight. In this case,
greater precision means that the study has a larger \textbf{N}, which
leads to a smaller \textbf{Standard Error} of its effect size estimate.

For this weighing, we use the \textbf{inverse of the variance}
\(1/\hat\sigma^2_k\) of each study \(k\). We then calculate a weighted
average of all studies, our fixed effect size estimator
\(\hat\theta_F\):
\end{rmdinfo}

\begin{equation}
\hat\theta_F = \frac{\sum\limits_{k=1}^K \hat\theta_k/ \hat\sigma^2_k}{\sum\limits_{k=1}^K 1/\hat\sigma^2_k}
\end{equation}

In \protect\hyperlink{excel_preparation}{Chapter 3.1}, we have described two ways your EXCEL spreadsheet for your meta-analysis data can look like:

\begin{itemize}
\tightlist
\item
  It can either be stored as the \textbf{raw data} (including the Mean, N, and SD of every study arm)
\item
  Or it only contains the \textbf{calculated effect sizes and the standard error (SE)}
\end{itemize}

The functions to pool the results with a fixed-effect-model \textbf{differ depending on which data format you used}, but not much. First, let's assume you already have a dataset with the \textbf{calucated effects and SE} for each study. In my case, this is my \texttt{madata} dataset.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(madata)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Classes 'tbl_df', 'tbl' and 'data.frame':    18 obs. of  17 variables:
##  $ Author               : chr  "Call et al." "Cavanagh et al." "DanitzOrsillo" "de Vibe et al." ...
##  $ TE                   : num  0.709 0.355 1.791 0.182 0.422 ...
##  $ seTE                 : num  0.261 0.196 0.346 0.118 0.145 ...
##  $ RoB                  : chr  "low" "low" "high" "low" ...
##  $ Control              : chr  "WLC" "WLC" "WLC" "no intervention" ...
##  $ intervention duration: chr  "short" "short" "short" "short" ...
##  $ intervention type    : chr  "mindfulness" "mindfulness" "ACT" "mindfulness" ...
##  $ population           : chr  "undergraduate students" "students" "undergraduate students" "undergraduate students" ...
##  $ type of students     : chr  "psychology" "general" "general" "general" ...
##  $ prevention type      : chr  "selective" "universal" "universal" "universal" ...
##  $ gender               : chr  "female" "mixed" "mixed" "mixed" ...
##  $ mode of delivery     : chr  "group" "online" "group" "group" ...
##  $ ROB streng           : chr  "high" "low" "high" "low" ...
##  $ ROB superstreng      : chr  "high" "high" "high" "low" ...
##  $ compensation         : chr  "none" "none" "voucher/money" "voucher/money" ...
##  $ instruments          : chr  "DASS" "PSS" "DASS" "other" ...
##  $ guidance             : chr  "f2f" "self-guided" "f2f" "f2f" ...
\end{verbatim}

This dataset has \textbf{continuous outcome data}. As our effect sizes are already calculated, we can use the \texttt{meta::metagen} function. For this function, we can specify loads of parameters, all of which you can accessed by typing \texttt{?metagen} in your console once the \texttt{meta} package is loaded.

\textbf{Here is a table with the most important parameters for our code:}

Parameter

Function

TE

This tells R to use the TE column to retrieve the effect sizes for each study

seTE

This tells R to use the seTE column to retrieve the standard error for each study

data=

After =, paste the name of your dataset here

studlab=paste()

This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)

comb.fixed=

Weather to use a fixed-effect-model

comb.random

Weather to use a random-effects-model

prediction=

Weather to print a prediction interval for the effect of future studies based on present evidence

sm=

The summary measure we want to calculate. We can either calculate the mean difference (MD) or Hedges' g/Cohen's d (SMD)

Let's code our first fixed-effects-model Meta-Analysis. We we will give the results of this analysis the simple name \texttt{m}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m<-}\KeywordTok{metagen}\NormalTok{(TE,}
\NormalTok{        seTE,}
        \DataTypeTok{data=}\NormalTok{madata,}
        \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),}
        \DataTypeTok{comb.fixed =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{comb.random =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)}
\NormalTok{m}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                           SMD            95%-CI %W(fixed)
## Call et al.            0.7091 [ 0.1979; 1.2203]       3.6
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]       6.3
## DanitzOrsillo          1.7912 [ 1.1139; 2.4685]       2.0
## de Vibe et al.         0.1825 [-0.0484; 0.4133]      17.5
## Frazier et al.         0.4219 [ 0.1380; 0.7057]      11.6
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]       6.3
## Gallego et al.         0.7249 [ 0.2846; 1.1652]       4.8
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]       5.5
## Hintz et al.           0.2840 [-0.0453; 0.6133]       8.6
## Kang et al.            1.2751 [ 0.6142; 1.9360]       2.1
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]       6.4
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]       4.6
## Phang et al.           0.5407 [ 0.0619; 1.0196]       4.1
## Rasanen et al.         0.4262 [-0.0794; 0.9317]       3.6
## Ratanasiripong         0.5154 [-0.1731; 1.2039]       2.0
## Shapiro et al.         1.4797 [ 0.8618; 2.0977]       2.4
## SongLindquist          0.6126 [ 0.1683; 1.0569]       4.7
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]       3.9
## 
## Number of studies combined: k = 18
## 
##                        SMD            95%-CI    z  p-value
## Fixed effect model  0.4805 [ 0.3840; 0.5771] 9.75 < 0.0001
## Prediction interval        [-0.0344; 1.1826]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.0752; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  45.50   17  0.0002
## 
## Details on meta-analytical method:
## - Inverse variance method
\end{verbatim}

We now see the results of our Meta-Analysis, including

\begin{itemize}
\tightlist
\item
  The \textbf{individual effect sizes} for each study, and their weight
\item
  The total \textbf{number of included studies} (k)
\item
  The \textbf{overall effect} (in our case, \emph{g} = 0.4805) and its confidence interval and p-value
\item
  Measures of \textbf{between-study heterogeneity}, such as \emph{tau\textsuperscript{2}} or \emph{I\textsuperscript{2}} and a \emph{Q}-test of heterogeneity
\end{itemize}

Using the \texttt{\$} command, we can also have a look at various outputs directly. For example

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m}\OperatorTok{$}\NormalTok{lower.I2}
\end{Highlighting}
\end{Shaded}

Gives us the lower bound of the 95\% confidence interval for \emph{I\textsuperscript{2}}

\begin{verbatim}
## [1] 0.3787897
\end{verbatim}

We can \textbf{save the results of the meta-analysis} to our working directory as a .txt-file using this command

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{sink}\NormalTok{(}\StringTok{"results.txt"}\NormalTok{)}
\KeywordTok{print}\NormalTok{(m)}
\KeywordTok{sink}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\hypertarget{fixed.raw}{%
\subsection{Raw effect size data}\label{fixed.raw}}

To conduct a fixed-effects-model Meta-Analysis from \textbf{raw data} (i.e, if your data has been prepared the way we describe in \protect\hyperlink{excel_preparation}{Chapter 3.1.1}), we have to use the \texttt{meta::metacont()} function instead. The structure of the code however, looks quite similar.

Parameter

Function

Ne

The number of participants (N) in the intervention group

Me

The Mean (M) of the intervention group

Se

The Standard Deviation (SD) of the intervention group

Nc

The number of participants (N) in the control group

Mc

The Mean (M) of the control group

Sc

The Standard Deviation (SD) of the control group

data=

After `=', paste the name of your dataset here

studlab=paste()

This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)

comb.fixed=

Weather to use a fixed-effects-model

comb.random

Weather to use a random-effects-model

prediction=

Weather to print a prediction interval for the effect of future studies based on present evidence

sm=

The summary measure we want to calculate. We can either calculate the mean difference (MD) or Hedges' g (SMD)

For this purpose, i will use my dataset \texttt{metacont}, which contains the raw data of all studies i want to snythesize

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(metacont)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Classes 'tbl_df', 'tbl' and 'data.frame':    6 obs. of  7 variables:
##  $ Author: chr  "Cavanagh" "Day" "Frazier" "Gaffney" ...
##  $ Ne    : num  50 64 90 30 77 60
##  $ Me    : num  4.5 18.3 12.5 2.34 15.21 ...
##  $ Se    : num  2.7 6.4 3.2 0.87 5.35 ...
##  $ Nc    : num  50 65 95 30 69 60
##  $ Mc    : num  5.6 20.2 15.5 3.13 20.13 ...
##  $ Sc    : num  2.6 7.6 4.4 1.23 7.43 ...
\end{verbatim}

Now, let's code the Meta-Analysis function, this time using the \texttt{meta::metacont} function, and my \texttt{metacont} dataset. I want to name my output \texttt{m.raw} now.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.raw<-}\KeywordTok{metacont}\NormalTok{(Ne,}
\NormalTok{                Me,}
\NormalTok{                Se,}
\NormalTok{                Nc,}
\NormalTok{                Mc,}
\NormalTok{                Sc,}
                \DataTypeTok{data=}\NormalTok{metacont,}
                \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),}
                \DataTypeTok{comb.fixed =} \OtherTok{TRUE}\NormalTok{,}
                \DataTypeTok{comb.random =} \OtherTok{FALSE}\NormalTok{,}
                \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,}
                \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)}
\NormalTok{m.raw}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              SMD             95%-CI %W(fixed)
## Cavanagh -0.4118 [-0.8081; -0.0155]      13.8
## Day      -0.2687 [-0.6154;  0.0781]      18.0
## Frazier  -0.7734 [-1.0725; -0.4743]      24.2
## Gaffney  -0.7303 [-1.2542; -0.2065]       7.9
## Greer    -0.7624 [-1.0992; -0.4256]      19.1
## Harrer   -0.1669 [-0.5254;  0.1916]      16.9
## 
## Number of studies combined: k = 6
## 
##                         SMD             95%-CI     z  p-value
## Fixed effect model  -0.5245 [-0.6718; -0.3773] -6.98 < 0.0001
## Prediction interval         [-1.1817;  0.1494]               
## 
## Quantifying heterogeneity:
## tau^2 = 0.0441; H = 1.51 [1.00; 2.38]; I^2 = 56.1% [0.0%; 82.3%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  11.39    5  0.0441
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Hedges' g (bias corrected standardised mean difference)
\end{verbatim}

\begin{rmdachtung}
As you can see, all the calculated effect sizes are \textbf{negative}
now, including the pooled effect. However, all studies report a positive
outcome, meaning that the symptoms in the intervention group (e.g., of
depression) were reduced. The negative orientation results from the fact
that in \textbf{most clinical trials, lower scores indicate better
outcomes} (e.g., less depression). It is no problem to report values
like this: in fact, it is conventional.

Some readers who are unfamiliar with meta-analysis, however,
\textbf{might be confused} by this, so may consider changing the
orientation of your values before you report them in your paper.
\end{rmdachtung}

We can \textbf{save the results of the meta-analysis} to our working directory as a .txt-file using this command

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{sink}\NormalTok{(}\StringTok{"results.txt"}\NormalTok{)}
\KeywordTok{print}\NormalTok{(m.raw)}
\KeywordTok{sink}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{random}{%
\section{Random-Effects-Model}\label{random}}

Previously, we showed how to perform a fixed-effect-model meta-analysis using the \texttt{meta:metagen} and \texttt{meta:metacont} functions.

However, we can only use the fixed-effect-model when we can assume that \textbf{all included studies come from the same population}. In practice this is hardly ever the case: interventions may vary in certain characteristics, the sample used in each study might be slightly different, or its methods. In this case, we cannot assume that all studies stem from one hypothesized ``population'' of studies.

Same is the case once we detect \textbf{statistical heterogeneity} in our fixed-effect-model meta-analysis, as indicated by \(I^{2}>0\).

So, it is very likely that you will actually use a random-effects-model for your meta-analysis. Thankfully, there's not much more we have to think about when conducting a random-effects-model meta-analysis in R instead of a fixed-effect-model meta-analysis.

\begin{rmdinfo}
\textbf{The Idea behind the Random-Effects-Model}

In the Random-Effects-Model, we want to account for our assumption that
the study effect estimates show more variance than when drawn from a
single population {[}@schwarzer2015meta{]}. The random-effects-model
works under the so-called \textbf{assumption of exchangeability}.

This means that in Random-Effects-Model Meta-Analyses, we not only
assume that effects of individual studies deviate from the true
intervention effect of all studies due to sampling error, but that there
is another source of variance introduced by the fact that the studies do
not stem from one single population, but are drawn from a ``universe''
of populations. We therefore assume that there is not only one true
effect size, but \textbf{a distribution of true effect sizes}. We
therefore want to estimate the mean of this distribution of true effect
sizes.

The fixed-effect-model assumes that when the observed effect size
\(\hat\theta_k\) of an individual study \(k\) deviates from the true
effect size \(\theta_F\), the only reason for this is that the estimate
is burdened by (sampling) error \(\epsilon_k\).

\[\hat\theta_k = \theta_F + \epsilon_k\]

While the random-effects-model assumes that, in addition, there is
\textbf{a second source of error} \(\zeta_k\).This second source of
error is introduced by the fact that even the true effect size
\(\theta_k\) of our study \(k\) is also only part of an over-arching
distribution of true effect sizes with the mean \(\mu\)
{[}@borenstein2011{]}.
\end{rmdinfo}

\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-53-1} \end{center}

\emph{An illustration of parameters of the random-effects-model}

\begin{rmdinfo}
The formula for the random-effects-model therefore looks like this:

\[\hat\theta_k = \mu + \epsilon_k + \zeta_k\]

When calculating a random-effects-model meta-analysis, where therefore
also have to take the error \(\zeta_k\) into account. To do this, we
have to \textbf{estimate the variance of the distribution of true effect
sizes}, which is denoted by \(\tau^{2}\), or
\emph{tau\textsuperscript{2}}. There are several estimators for
\(\tau^{2}\), all of which are implemented in \texttt{meta}. We will
give you more details about them in the next section.
\end{rmdinfo}

\begin{rmdachtung}
Even though it is \textbf{conventional} to use random-effects-model
meta-analyses in psychological outcome research, applying this model is
\textbf{not undisputed}. The random-effects-model pays \textbf{more
attention to small studies} when pooling the overall effect in a
meta-analysis {[}@schwarzer2015meta{]}. Yet, small studies in particular
are often fraught with \textbf{bias} (see
\protect\hyperlink{smallstudyeffects}{Chapter 8.1}). This is why some
have argued that the fixed-effects-model should be nearly always
preferred {[}@poole1999; @furukawa2003{]}.
\end{rmdachtung}

\hypertarget{tau2}{%
\subsection{\texorpdfstring{Estimators for \emph{tau\textsuperscript{2}} in the random-effects-model}{Estimators for tau2 in the random-effects-model}}\label{tau2}}

Operationally, conducting a random-effects-model meta-analysis in R is not so different from conducting a fixed-effects-model meta-analyis. Yet, we do have choose an estimator for \(\tau^{2}\). Here are the estimators implemented in \texttt{meta}, which we can choose using the \texttt{method.tau} variable in our meta-analysis code.

Code

Estimator

DL

DerSimonian-Laird

PM

Paule-Mandel

REML

Restricted Maximum-Likelihood

ML

Maximum-likelihood

HS

Hunter-Schmidt

SJ

Sidik-Jonkman

HE

Hedges

EB

Empirical Bayes

\begin{rmdinfo}
\textbf{Which estimator should i use?}

All of these estimators derive \(\tau^{2}\) using a slightly different
approach, leading to somewhat different pooled effect size estimates and
confidence intervals. If one of these approaches is more or less biased
often depends on the context, and parameters such as the number of
studies \(k\), the number of participants \(n\) in each study, how much
\(n\) varies from study to study, and how big \(\tau^{2}\) is.

An overview paper by Veroniki and colleagues {[}@veroniki2016methods{]}
provides an excellent summary on current evidence which estimator might
be more or less biased in which situation. The article is openly
accessible, and you can read it
\href{https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4950030/}{here}.

Especially in medical and psychological research, the by far most often
used estimator is the \textbf{DerSimonian-Laird estimator}
{[}@dersimonian1986meta{]}. Part of this widespread use might be
attributable to the fact that programs such as \emph{RevMan} or
\emph{Comprehensive Meta-Analysis} (older versions) only use this
estimator. It is also the default option in our \texttt{meta} package in
R. Simulation studies, however, have shown that the
\textbf{Maximum-Likelihood}, \textbf{Sidik-Jonkman}, and
\textbf{Empirical Bayes} estimators have better properties in estimating
the between-study variance
{[}@sidik2007comparison;@viechtbauer2005bias{]}.
\end{rmdinfo}

\begin{rmdinfo}
\textbf{The Hartung-Knapp-Sidik-Jonkman method}

Another criticism of the \textbf{DerSimonian-Laird} method is that when
estimating the variance of our pooled effect \(var(\hat\theta_F)\), this
method is very prone to producing false positives
{[}@inthout2014hartung{]}. This is especially the case when the
\textbf{number of studies} is small, and when there is substantial
\textbf{heterogeneity}
{[}@hartung1999alternative;@hartung2001refined;@hartung2001tests;@follmann1999valid;@makambi2004effect{]}.
Unfortunately, this is very often the case in when we do meta-analysis
in the medical field or in psychology. This is quite a problem, as we
don't want to find pooled effects to be statistically significant when
in fact they are not!

The \textbf{Hartung-Knapp-Sidik-Jonkman (HKSJ) method} was thus proposed
a way to produce more robust estimates of \(var(\hat\theta_F)\). It has
been shown that this method substantially outperforms the
DerSimonian-Laird method in many cases {[}@inthout2014hartung{]}. The
HKSJ method can also be very easily applied in R, while other programs
don't have this option yet. This is another big plus of doing
meta-analysis in R. The HKSJ usually leads to more \textbf{conservative}
results, indicated by wider confidence intervals.
\end{rmdinfo}

\begin{rmdachtung}
\textbf{Residual concerns with the Hartung-Knapp-Sidik-Jonkman method}

It should be noted, however, that the HKSJ method is not
uncontroversial. Some authors argue that other (standard) pooling models
should also be used \textbf{in addition} to the HKSJ as a
\textbf{sensitivity analysis} {[}@wiksten2016hartung{]}. Jackson and
colleagues {[}@jackson2017hartung{]} present four residual concerns with
this method, which you may take into account before selecting your
meta-analytic method. The paper can be read
\href{https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.7411}{here}.
\end{rmdachtung}

\hypertarget{random.precalc}{%
\subsection{Pre-calculated effect size data}\label{random.precalc}}

After all this input, you'll see that even random-effects-model meta-analyses are very easy to code in R. Compared to the fixed-effects-model \protect\hyperlink{fixed}{Chapter 4.1}, there's just three extra parameters we have to define. Especially, as we've described before, we have to tell R which \textbf{between-study-variance estimator} (\(\tau^{2}\)) we want to use, and if we want to use the \textbf{Knapp-Hartung-Sidik-Jonkman} adjustment.

\textbf{Here's a table of all parameters we have to define in our code to perform a random-effects-model meta-analysis with pre-calculated effect sizes}

Parameter

Function

TE

This tells R to use the TE column to retrieve the effect sizes for each study

seTE

This tells R to use the seTE column to retrieve the standard error for each study

data=

After =, paste the name of your dataset here

studlab=paste()

This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)

comb.fixed=

Weather to use a fixed-effects-model

comb.random=

Weather to use a random-effects-model. This has to be set to TRUE

method.tau=

Which estimator to use for the between-study variance

hakn=

Weather to use the Knapp-Hartung-Sidik-Jonkman method

prediction=

Weather to print a prediction interval for the effect of future studies based on present evidence

sm=

The summary measure we want to calculate. We can either calculate the mean difference (MD) or Hedges' g (SMD)

I will use my \texttt{madata} dataset again to do the meta-analysis. For illustrative purposes, let's use the Sidik-Jonkman estimator (``SJ'') and the HKSJ method. To do this analysis, make sure that \texttt{meta} as well as \texttt{metafor} are loaded in R.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(meta)}
\KeywordTok{library}\NormalTok{(metafor)}
\end{Highlighting}
\end{Shaded}

Now, let's code our random-effects-model meta-analysis. Remember, as our effect size data are precalculated, i'll use the \texttt{meta::metagen()} function.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj<-}\KeywordTok{metagen}\NormalTok{(TE,}
\NormalTok{        seTE,}
        \DataTypeTok{data=}\NormalTok{madata,}
        \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),}
        \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{,}
        \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)}
\NormalTok{m.hksj}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                           SMD            95%-CI %W(random)
## Call et al.            0.7091 [ 0.1979; 1.2203]        5.2
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]        6.1
## DanitzOrsillo          1.7912 [ 1.1139; 2.4685]        4.2
## de Vibe et al.         0.1825 [-0.0484; 0.4133]        7.1
## Frazier et al.         0.4219 [ 0.1380; 0.7057]        6.8
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]        6.1
## Gallego et al.         0.7249 [ 0.2846; 1.1652]        5.7
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]        5.9
## Hintz et al.           0.2840 [-0.0453; 0.6133]        6.5
## Kang et al.            1.2751 [ 0.6142; 1.9360]        4.3
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]        6.1
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]        5.6
## Phang et al.           0.5407 [ 0.0619; 1.0196]        5.4
## Rasanen et al.         0.4262 [-0.0794; 0.9317]        5.3
## Ratanasiripong         0.5154 [-0.1731; 1.2039]        4.1
## Shapiro et al.         1.4797 [ 0.8618; 2.0977]        4.5
## SongLindquist          0.6126 [ 0.1683; 1.0569]        5.7
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]        5.4
## 
## Number of studies combined: k = 18
## 
##                         SMD            95%-CI    t  p-value
## Random effects model 0.5935 [ 0.3891; 0.7979] 6.13 < 0.0001
## Prediction interval         [-0.2084; 1.3954]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  45.50   17  0.0002
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
\end{verbatim}

The output shows that our estimated effect is \(g=0.5935\), and the 95\% confidence interval stretches from \(g=0.39\) to \(0.80\) (rounded).

It also becomes clear that this effect is different (and larger) than the one we found in the fixed-effects-model meta-analysis in \protect\hyperlink{fixed}{Chapter 4.1} (\(g=0.48\)).

Let's compare this to the output using the \textbf{DerSimonian-Laird} estimator, and when setting \texttt{hakn=FALSE}. As this estimator is the \textbf{default}, i don't have to define \texttt{method.tau} this time.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.dl<-}\KeywordTok{metagen}\NormalTok{(TE,}
\NormalTok{        seTE,}
        \DataTypeTok{data=}\NormalTok{madata,}
        \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),}
        \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{hakn =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)}
\NormalTok{m.dl}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                           SMD            95%-CI %W(random)
## Call et al.            0.7091 [ 0.1979; 1.2203]        5.0
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]        6.3
## DanitzOrsillo          1.7912 [ 1.1139; 2.4685]        3.7
## de Vibe et al.         0.1825 [-0.0484; 0.4133]        8.0
## Frazier et al.         0.4219 [ 0.1380; 0.7057]        7.4
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]        6.3
## Gallego et al.         0.7249 [ 0.2846; 1.1652]        5.7
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]        6.0
## Hintz et al.           0.2840 [-0.0453; 0.6133]        6.9
## Kang et al.            1.2751 [ 0.6142; 1.9360]        3.8
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]        6.3
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]        5.6
## Phang et al.           0.5407 [ 0.0619; 1.0196]        5.3
## Rasanen et al.         0.4262 [-0.0794; 0.9317]        5.1
## Ratanasiripong         0.5154 [-0.1731; 1.2039]        3.6
## Shapiro et al.         1.4797 [ 0.8618; 2.0977]        4.1
## SongLindquist          0.6126 [ 0.1683; 1.0569]        5.7
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]        5.2
## 
## Number of studies combined: k = 18
## 
##                         SMD            95%-CI    z  p-value
## Random effects model 0.5741 [ 0.4082; 0.7399] 6.78 < 0.0001
## Prediction interval         [-0.0344; 1.1826]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.0752; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  45.50   17  0.0002
## 
## Details on meta-analytical method:
## - Inverse variance method
## - DerSimonian-Laird estimator for tau^2
\end{verbatim}

We see that the overall effect size estimate using this estimator is similar to the previous one (\(g=0.57\)), but the confidence intervals \textbf{is narrower because we did not adjust them} using the HKSJ method.

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-65-1.pdf}

\hypertarget{random.raw}{%
\subsection{Raw effect size data}\label{random.raw}}

I we use raw effect size data, such as the one stored in my \texttt{metacont} dataset, we can use the \texttt{meta::metacont} function again. The parameters stay the same as before.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj.raw<-}\KeywordTok{metacont}\NormalTok{(Ne,}
\NormalTok{        Me,}
\NormalTok{        Se,}
\NormalTok{        Nc,}
\NormalTok{        Mc,}
\NormalTok{        Sc,}
        \DataTypeTok{data=}\NormalTok{metacont,}
        \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),}
        \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{,}
        \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)}
\NormalTok{m.hksj.raw}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              SMD             95%-CI %W(random)
## Cavanagh -0.4118 [-0.8081; -0.0155]       15.7
## Day      -0.2687 [-0.6154;  0.0781]       17.7
## Frazier  -0.7734 [-1.0725; -0.4743]       19.7
## Gaffney  -0.7303 [-1.2542; -0.2065]       11.7
## Greer    -0.7624 [-1.0992; -0.4256]       18.1
## Harrer   -0.1669 [-0.5254;  0.1916]       17.2
## 
## Number of studies combined: k = 6
## 
##                          SMD             95%-CI     t p-value
## Random effects model -0.5161 [-0.8043; -0.2280] -4.60  0.0058
## Prediction interval          [-1.1947;  0.1625]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.0472; H = 1.51 [1.00; 2.38]; I^2 = 56.1% [0.0%; 82.3%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  11.39    5  0.0441
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
## - Hedges' g (bias corrected standardised mean difference)
\end{verbatim}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{binary}{%
\section{Meta-Analysis with binary outcomes}\label{binary}}

\includegraphics{compare.jpg}

\hypertarget{event-rate-data}{%
\subsection{Event rate data}\label{event-rate-data}}

In some cases, you will work with \textbf{binary outcome data} (e.g., dead/alive, Depressive Disorder/no Depressive Disorder) instead of continuous data. In such a case, you will probably be more interested in outcomes like the pooled \textbf{Odd's Ratio} or the \textbf{Relative Risk Reduction}.

Here, have two options again:

\begin{itemize}
\tightlist
\item
  \textbf{The effect sizes are already calculated}. In this case, we can use the \texttt{metagen} function as we did before (see \protect\hyperlink{fixed}{Chapter 4.1} and \protect\hyperlink{random}{Chapter 4.2}). The calculated effect \texttt{TE} then describes the Odds Ratio, or whatever binary outcome we calculated previously for our data.
\item
  \textbf{We only have the raw outcome data}. If this is the case, we will have to use the \texttt{meta::metabin} function instead. We'll show you how to do this now.
\end{itemize}

\textbf{For meta-analyses of binary outcomes, we need our data in the following format:}

Column

Description

Author

This signifies the column for the study label (i.e., the first author)

Ee

Number of events in the experimental treatment arm

Ne

Number of participants in the experimental treatment arm

Ec

Number of events in the control arm

Nc

Number of participants in the control arm

Subgroup

This is the label for one of your subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g.~population). In this column, each study should then be given a subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique

I'll use my dataset \texttt{binarydata}, which also has this format

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{load}\NormalTok{(}\StringTok{"binarydata.RData"}\NormalTok{)}
\KeywordTok{str}\NormalTok{(binarydata)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Classes 'tbl_df', 'tbl' and 'data.frame':    11 obs. of  5 variables:
##  $ Author: chr  "Alcorta-Fleischmann" "Craemer" "Eriksson" "Jones" ...
##  $ Ee    : num  2 18 6 3 0 8 12 1 7 17 ...
##  $ Ne    : num  279 1273 1858 297 300 ...
##  $ Ec    : num  1 17 5 6 1 9 12 10 8 21 ...
##  $ Nc    : num  70 1287 1852 314 295 ...
\end{verbatim}

The other parameters are like the ones we used in the meta-analyses with continuous outcome data, with two exceptions:

\begin{itemize}
\tightlist
\item
  \textbf{sm}: As we want to have a pooled effect for binary data, we have to choose another summary measure now. We can choose from \textbf{``OR''} (Odds Ratio), \textbf{``RR''} (Risk Ratio), or \textbf{RD} (Risk Difference), among other things.
\item
  \textbf{incr}. This lets us define if and how we want \textbf{conitinuity correction} to be performed. Such a correction is necessary in cases where one of the cells in your data is zero (e.g., because no one in the intervention arm died). This can be a frequent phenomenon in some contexts, and \textbf{distorts our effect size estimates}. By default, the \texttt{metabin} function adds the value \textbf{0.5} in all cells were N is zero \citep{gart1967bias}. This value can be changed using the \texttt{incr}-parameter (e.g., \texttt{incr=0.1}). If your trial arms are very uneven in terms of their total \(n\), we can also use the \textbf{treatment arm continuity correction} \citep{j2004add}. This can be done by using \texttt{incr="TACC"}.
\end{itemize}

\textbf{Here's the code for a meta-analysis with raw binary data}

I have decided to run a random-effect-model meta-analysis. I want the summary measure to be the Risk Ratio (RR).

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.bin<-}\KeywordTok{metabin}\NormalTok{(Ee,}
\NormalTok{        Ne,}
\NormalTok{        Ec,}
\NormalTok{        Nc,}
        \DataTypeTok{data=}\NormalTok{binarydata,}
        \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),}
        \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{,}
        \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{incr=}\FloatTok{0.1}\NormalTok{,}
        \DataTypeTok{sm=}\StringTok{"RR"}\NormalTok{)}
\NormalTok{m.bin}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                         RR            95%-CI %W(random)
## Alcorta-Fleischmann 0.5018 [0.0462;  5.4551]        3.0
## Craemer             1.0705 [0.5542;  2.0676]       14.6
## Eriksson            1.1961 [0.3657;  3.9124]        8.5
## Jones               0.5286 [0.1334;  2.0945]        7.0
## Knauer              0.0894 [0.0001; 57.7924]        0.5
## Kracauer            0.9076 [0.3512;  2.3453]       10.9
## La Sala             0.9394 [0.4233;  2.0847]       12.7
## Maheux              0.0998 [0.0128;  0.7768]        3.9
## Schmidthauer        0.7241 [0.2674;  1.9609]       10.3
## van der Zee         0.8434 [0.4543;  1.5656]       15.1
## Wang                0.5519 [0.2641;  1.1534]       13.5
## 
## Number of studies combined: k = 11
## 
##                          RR           95%-CI     t p-value
## Random effects model 0.7420 [0.5202; 1.0582] -1.87  0.0905
## Prediction interval         [0.2305; 2.3882]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.2417; H = 1.00 [1.00; 1.36]; I^2 = 0.0% [0.0%; 45.8%]
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  7.34   10  0.6929
## 
## Details on meta-analytical method:
## - Mantel-Haenszel method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
## - Continuity correction of 0.1 in studies with zero cell frequencies
\end{verbatim}

\textbf{L'Abbé Plots}

So-called \textbf{L'Abbé plots} \citep{labbe} are a good way to visualize data based on event rates. In a L'Abbé plot, the event rate of a study's intervention group is plotted against the event rate in the control group, and the \(N\) of the study is signified by the size of the bubble in the plot. Despite the simplicity in its principles, this plot allows us to check for three important aspects of our meta-analysis with binary outcomes:

\begin{itemize}
\tightlist
\item
  \textbf{The overall trend of our meta-analysis}. If we are expecting a type of intervention to have a protective effect (i.e., making an adverse outcome such as death or depression onset less likely) the studies should mostly lie in the bottom-right corner of the L'Abbé plot, because the control group event rate should be higher than the intervention group event rate. If there's no effect of the intervention compared to the control group, the event rates are identical and the study is shown on the diagonal of the L'Abbé plot.
\item
  \textbf{Heterogeneity of effect sizes}. The plot also allows us to eyeball for single studies or groups of studies which contribute to the heterogeneity of the effect we found. It could be the case, for example, that most studies lie in the bottom-right part of the plot as they report positive effects, while a few studies lie in the top-left sector indicated negative effects. Especially if such studies have a small precision (i.e., a small \(N\) of participants, indicated by small bubbles in the plot), they could have distorted our pooled effect and may contribute to the between-study heterogeneity.
\item
  \textbf{Heterogeneity of event rates}. It may also be the case that some of the heterogeneity in our meta-analysis was introduced by the fact that the event rates ``per se'' are higher or lower in some studies compared to the others. The L'Abbé plot provides as with this information, as studies with higher event rates will naturally tend towards the top-right corner of the plot.
\end{itemize}

The results of the \texttt{metabin} function can be easily used to generate L'Abbé plots using the \texttt{labbe.metabin} function included in the \texttt{meta} package. We can specify the following parameters:

Parameter

Description

x

This signifies our metabin meta-analysis output

bg

The background color of the studies

col

The line color of the studies

studlab

Wether the names of the studies should be printed in the plot (TRUE/FALSE)

col.fixed

The color of the dashed line symbolizing the pooled effect of the meta-analysis, if the fixed-effect-model was used

col.random

The color of the dashed line symbolizing the pooled effect of the meta-analysis, if the random-effects-model was used

For this example, i'll use the \texttt{m.bin} output i previously generated using the \texttt{metabin} function.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{labbe.metabin}\NormalTok{(}\DataTypeTok{x =}\NormalTok{ m.bin,}
              \DataTypeTok{bg =} \StringTok{"blue"}\NormalTok{,}
              \DataTypeTok{studlab =} \OtherTok{TRUE}\NormalTok{,}
              \DataTypeTok{col.random =} \StringTok{"red"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-72-1.pdf}

Works like a charm! We see that the \textbf{dashed red line} signifying the \textbf{pooled effect estimate} of my meta-analysis is running trough the bottom-right sector of my L'Abbé plot, meaning that the overall effect size is positive (i.e., that the intervention has a preventive effect).

However, it also becomes clear that \textbf{all studies} clearly follow this trend: we see that most studies lie tightly in the bottom-left corner of the plot, meaning that these studies had \textbf{small event rates} (i.e., the event in these studies was very rare irrespective of group assignment). We also see that two of our included studies don't fall into this pattern: \textbf{Schmidthauer} and \textbf{van der Zee}. Those two studies have higher event rates, and both favor the intervention group more clearly than the others did.

\hypertarget{incidence-rates}{%
\subsection{Incidence rates}\label{incidence-rates}}

The \protect\hyperlink{binary}{previous chapter} primarily dealt with raw event data. Such data usually does not contain any information on the \textbf{time span} during which events did or did not occur. Given that studies often have drastically different follow-up times (e.g., 8 weeks vs.~2 years), it often makes sense to also take the time interval during which events occured into account. In clinical epidemiology, \textbf{incidence rates} are often used to signify how many events occured within a \textbf{standardized timeframe} (e.g., one year). The corresponding effect size is the \textbf{incidence rate ratio} (IRR), which compares the incidence rate in the intervention group to the one in the control group.

To conduct meta-analyses using incidence rate data, so-called \textbf{person-time} data has to be collected or calculated by hand. What it basically needed to calculate person-time data is the \textbf{number of events} and the \textbf{timeframe} during which they occurred. You can find a general introduction into this topic \href{https://sph.unc.edu/files/2015/07/nciph_ERIC4.pdf}{here}, and this \href{https://www.cdc.gov/ophss/csels/dsepd/ss1978/lesson3/section2.html}{quick course} by the Centers for Disease Control and Prevention gives a hands-on introduction on how person-time data is calculated.

As an example, i will use my \texttt{IRR.data.RData} dataset, in which the person-time (in my case, person-year) is stored in the \texttt{time.e} and \texttt{time.c} column for the experimental and control group, respectively. The \texttt{event.e} and \texttt{event.c} column contains the number of \textbf{events} (in my case, depression onsets) for both groups.

\begin{verbatim}
## # A tibble: 8 x 5
##   Author                                    event.e time.e event.c time.c
##   <chr>                                       <dbl>  <dbl>   <dbl>  <dbl>
## 1 Stice et al., 2013 (b)                       6     362     16     356  
## 2 Stice et al. 2017a (clinician-led)           7      77.9    2.67   24.7
## 3 Stice et al. 2017a (peer-led)                3      78.5    2.67   24.7
## 4 Stice et al. 2017a (online)                 12      79.9    2.67   24.7
## 5 Stice et al. 2017b (non dissonance-based)    3     197      4.5   108. 
## 6 Stice et al. 2017b (dissonance-based)        3     197      4.5   108. 
## 7 Taylor et al., 2006                          8.35  224.    13.3   221. 
## 8 Taylor et al., 2016                         22     160     29     159
\end{verbatim}

To pool the data, we use the \texttt{metainc} function included in the \texttt{meta} package. We can set the following parameters:

Parameter

Function

event.e

The number of events in the intervention group

time.e

The person-time at risk in the intervention group

event.c

The number of events in the control group

time.c

The person-time at risk in the control group

data=

After =, paste the name of your dataset here

studlab=paste()

This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)

comb.fixed=

Weather to use a fixed-effects-model

comb.random=

Weather to use a random-effects-model. This has to be set to TRUE

method.tau=

Which estimator to use for the between-study variance

hakn=

Weather to use the Knapp-Hartung-Sidik-Jonkman method

prediction=

Weather to print a prediction interval for the effect of future studies based on present evidence

sm=

The summary measure we want to calculate. We want to calculate the Incidence Rate Ratio (IRR)

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{metainc}\NormalTok{(event.e, }
\NormalTok{        time.e, }
\NormalTok{        event.c, }
\NormalTok{        time.c, }
        \DataTypeTok{studlab =} \KeywordTok{paste}\NormalTok{(Author), }
        \DataTypeTok{data =}\NormalTok{ IRR.data, }
        \DataTypeTok{sm =} \StringTok{"IRR"}\NormalTok{,}
        \DataTypeTok{method.tau =} \StringTok{"DL"}\NormalTok{,}
        \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,}
        \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,}
        \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                                              IRR           95%-CI
## Stice et al., 2013 (b)                    0.3688 [0.1443; 0.9424]
## Stice et al. 2017a (clinician-led)        0.8312 [0.2030; 3.4038]
## Stice et al. 2017a (peer-led)             0.3536 [0.0680; 1.8393]
## Stice et al. 2017a (online)               1.3888 [0.3687; 5.2314]
## Stice et al. 2017b (non dissonance-based) 0.3638 [0.0844; 1.5678]
## Stice et al. 2017b (dissonance-based)     0.3638 [0.0844; 1.5678]
## Taylor et al., 2006                       0.6184 [0.2604; 1.4686]
## Taylor et al., 2016                       0.7539 [0.4332; 1.3121]
##                                           %W(random)
## Stice et al., 2013 (b)                          14.0
## Stice et al. 2017a (clinician-led)               6.2
## Stice et al. 2017a (peer-led)                    4.5
## Stice et al. 2017a (online)                      7.0
## Stice et al. 2017b (non dissonance-based)        5.8
## Stice et al. 2017b (dissonance-based)            5.8
## Taylor et al., 2006                             16.5
## Taylor et al., 2016                             40.2
## 
## Number of studies combined: k = 8
## 
##                         IRR           95%-CI     t p-value
## Random effects model 0.6157 [0.4349; 0.8716] -3.30  0.0131
## 
## Quantifying heterogeneity:
## tau^2 = 0; H = 1.00 [1.00; 1.44]; I^2 = 0.0% [0.0%; 51.8%]
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  4.71    7  0.6953
## 
## Details on meta-analytical method:
## - Mantel-Haenszel method
## - DerSimonian-Laird estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
\end{verbatim}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{forest-plots}{%
\chapter{Forest Plots}\label{forest-plots}}

\includegraphics{forest.jpg}

\begin{rmdinfo}
Now that we created the \textbf{output of our meta-analysis} using the
\texttt{metagen}, \texttt{metacont} or \texttt{metabin} functions in
\texttt{meta} (see \protect\hyperlink{fixed}{Chapter
4.1},\protect\hyperlink{random}{Chapter 4.2} and
\protect\hyperlink{binary}{Chapter 4.3}), it is time to present the data
in a more digestable way.

\textbf{Forest Plots} are an easy way to do this, and it is conventional
to report forest plots in meta-analysis publications.
\end{rmdinfo}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{generating-a-forest-plot}{%
\section{Generating a Forest Plot}\label{generating-a-forest-plot}}

To produce a forest plot, we use the meta-analysis output we just created (e.g., \texttt{m}, \texttt{m.raw}) und the \texttt{meta::forest()} function. I'll use my \texttt{m.hksj.raw} output from \protect\hyperlink{random.raw}{Chapter 4.2.3} to create the forest plot

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{forest}\NormalTok{(m.hksj.raw)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-78-1} \end{center}

Looks good so far. We see that the function plotted a forest plot with a \textbf{diamond} (i.e.~the overall effect and its confidence interval) and a \textbf{prediction interval}.

There are plenty of \textbf{other parameters} within the \texttt{meta::forest} function which we can use to modify the forest plot.

Type

Parameter

Description

General

sortvar

A sorting variable. For example, you can sort the forest plot by effect size using `TE', or by author name using `Author'

General

studlab

This tells the function which variable should be printed as the study label. The standard is `Author'

General

comb.fixed

Whether fixed effect estimate should be plotted. (TRUE/FALSE)

General

comb.random

Whether random effects estimate should be plotted. (TRUE/FALSE)

General

overall

Whether overall summaries should be plotted. This argument is useful in a meta-analysis with subgroups if summaries should only be plotted on group level.

General

text.fixed

A character string used in the plot to label the pooled fixed effect estimate. Has to be put in "" (e.g. ``Overall effect'')

General

text.random

A character string used in the plot to label the pooled random effect estimate. Has to be put in "" (e.g. ``Overall effect'')

General

col.fixed

Line colour (pooled fixed effect estimate). E.g. ``red'', ``blue'', or hex color code (``\#2e8aff'')

General

col.random

Line colour (random fixed effect estimate). E.g. ``red'', ``blue'', or hex color code (``\#2e8aff'')

General

prediction

Whether a prediction interval should be printed.

General

text.predict

A character string used in the plot to label the prediction interval. E.g. ``Prediction Interval''

General

subgroup

A logical indicating whether subgroup results should be shown in forest plot. This argument is useful in a meta-analysis with subgroups if summaries should not be plotted on group level. (TRUE/FALSE)

General

print.subgroup.labels

A logical indicating whether subgroup label should be printed. (TRUE/FALSE)

General

study.results

Whether results for individual studies should be shown in the figure (useful to only plot subgroup results). (TRUE/FALSE)

General

xlab

A label for the x-axis on the bottom. Put in "".

General

smlab

A label for the summary measure on top. Put in "".

General

xlim

The x limits (min,max) of the plot, or the character ``s'' to produce symmetric forest plots. This is particularly revelant when your results deviate substantially from zero, or if you also want to have outliers depicted. (e.g.~xlim=c(0,1.5) for effects from 0 to 1.5).

General

ref

The reference value to be plotted as a line in the forest plot. This is interesting if you want to compare effects to common thresholds or results from previous analyses (e.g.~ref=0.5)

General

leftcols

Here you can specify all variables which should be printed on the left side of your plot. The variables have to be part of you meta-analysis output, so check with name\_of\_your\_output\$ which variables can be displayed. E.g. leftcols=c(``TE'',``seTE'').

General

rightcols

Same as leftcols, but for the right side of your plot

General

leftlabs

This specifies how the columns on the left side should be named. Always provide all labels (e.g.~leftlabs=c(``Author'',``Effect size'',``Standard error''))

General

rightlabs

Same as leftlabs, but for the left side of you plot

General

print.I2

Whether to print the value of the I-squared statistic.

General

print.I2.ci

Whether to print the confidence interval of the I-squared statistic.

General

squaresize

A numeric used to increase or decrease the size of squares in the forest plot. (E.g., squaresize = 1.2)

Color

col.study

The colour for individual study results and confidence limits. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.inside

The colour for individual study results and confidence limits if confidence limits are completely within squares. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.square

The colour for squares reflecting study's weight in the meta-analysis. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.square.lines

The colour for the outer lines of squares reflecting study's weight in the meta-analysis. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.diamond

The colour of diamonds representing the results for fixed effect and random effects models. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.diamond.fixed

The colour of diamonds for fixed effect estimates. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.diamond.random

The colour of diamonds for random effects estimates. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.diamond.lines

The colour of the outer lines of diamonds representing the results for fixed effect and random effects models. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.diamond.lines.fixed

The colour of the outer lines of diamond for fixed effect estimate. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.diamond.lines.random

The colour of the outer lines of diamond for random effects estimate. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.inside.fixed

The colour for result of fixed effect meta-analysis if confidence limit lies completely within square. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.inside.random

The colour for result of random effects meta-analysis if confidence limit lies completely within square. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.predict

Background colour of prediction interval. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.predict.lines

Colour of outer lines of prediction interval. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.label.right

The colour for label on right side of null effect. E.g. `red', `blue', or hex color code (`\#2e8aff')

Color

col.label.left

The colour for label on left side of null effect. E.g. `red', `blue', or hex color code (`\#2e8aff')

Digits

digits

Minimal number of significant digits for treatment effects (TE)

Digits

digits.se

Minimal number of significant digits for standard errors

Digits

digits.zval

Minimal number of significant digits for z- or t-statistic for test of overall effect

Digits

digits.tau2

Minimal number of significant digits for between-study variance

Digits

digits.pval

Minimal number of significant digits for p-value of overall treatment effect

Digits

digits.pval.Q

Minimal number of significant digits for p-value of heterogeneity test

Digits

digits.Q

Minimal number of significant digits for heterogeneity statistic Q

Digits

digits.I2

Minimal number of significant digits for I-squared statistic

Digits

digits.weight

Minimal number of significant digits for weights

Digits

digits.mean

Minimal number of significant digits for the mean

Digits

digits.sd

Minimal number of significant digits for the standard deviations

This is again just an overview. For all settings, type \texttt{?meta::forest} in your \textbf{console} to see more.

Let's play around with the function a little now:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{forest}\NormalTok{(m.hksj.raw,}
       \DataTypeTok{sortvar=}\NormalTok{TE,}
       \DataTypeTok{xlim =} \KeywordTok{c}\NormalTok{(}\OperatorTok{-}\FloatTok{1.5}\NormalTok{,}\FloatTok{0.5}\NormalTok{),}
       \DataTypeTok{rightlabs =} \KeywordTok{c}\NormalTok{(}\StringTok{"g"}\NormalTok{,}\StringTok{"95% CI"}\NormalTok{,}\StringTok{"weight"}\NormalTok{),}
       \DataTypeTok{leftlabs =} \KeywordTok{c}\NormalTok{(}\StringTok{"Author"}\NormalTok{, }\StringTok{"N"}\NormalTok{,}\StringTok{"Mean"}\NormalTok{,}\StringTok{"SD"}\NormalTok{,}\StringTok{"N"}\NormalTok{,}\StringTok{"Mean"}\NormalTok{,}\StringTok{"SD"}\NormalTok{),}
       \DataTypeTok{lab.e =} \StringTok{"Intervention"}\NormalTok{,}
       \DataTypeTok{pooled.totals =} \OtherTok{FALSE}\NormalTok{,}
       \DataTypeTok{smlab =} \StringTok{""}\NormalTok{,}
       \DataTypeTok{text.random =} \StringTok{"Overall effect"}\NormalTok{,}
       \DataTypeTok{print.tau2 =} \OtherTok{FALSE}\NormalTok{,}
       \DataTypeTok{col.diamond =} \StringTok{"blue"}\NormalTok{,}
       \DataTypeTok{col.diamond.lines =} \StringTok{"black"}\NormalTok{,}
       \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,}
       \DataTypeTok{print.I2.ci =} \OtherTok{TRUE}\NormalTok{,}
       \DataTypeTok{digits.sd =} \DecValTok{2}
\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-80-1} \end{center}

Looks good so far! For special \textbf{layout types}, proceed to \protect\hyperlink{layouttypes}{Chapter 5.2} now.

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{layouttypes}{%
\section{Layout types}\label{layouttypes}}

The \texttt{meta::forest} function also has two \textbf{Layouts} preinstalled which we can use. Those layouts can be accessed with the \texttt{layout=} parameter.

\begin{itemize}
\tightlist
\item
  \textbf{``RevMan5''}. This layout is used for Cochrane reviews and generated by \emph{Review Manager 5}.
\item
  \textbf{``JAMA''}. This layout gives you a forest plot according to the guidelines of the \emph{Journal of the American Medical Association} as output (see details \href{https://jamanetwork.com/journals/jama/pages/instructions-for-authors}{here}).
\end{itemize}

The \textbf{RevMan} layout looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{forest}\NormalTok{(m.hksj.raw,}
       \DataTypeTok{layout =} \StringTok{"RevMan5"}\NormalTok{,}
       \DataTypeTok{digits.sd =} \DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-81-1} \end{center}

The \textbf{JAMA} layout looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{forest}\NormalTok{(m.hksj.raw,}
       \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,}
       \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,}
       \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,}
       \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-82-1} \end{center}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{saving-the-forest-plots}{%
\section{Saving the forest plots}\label{saving-the-forest-plots}}

Let's say i want to save the JAMA version of my Forest Plot now. To do this, i have to reuse the code with which i plotted my forest plot, and put it between \texttt{pdf(file=\textquotesingle{}name\_of\_the\_pdf\_i\_want\_to\_create.pdf\textquotesingle{})} and \texttt{dev.off}, both in separate lines. This saves the plot into a PDF in my Working Directory.

This way, i can export the plot in different formats (you can find more details on the saving options \protect\hyperlink{saving}{here}).

\textbf{PDF}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{pdf}\NormalTok{(}\DataTypeTok{file=}\StringTok{'forestplot.pdf'}\NormalTok{) }
\NormalTok{forest.jama<-}\KeywordTok{forest}\NormalTok{(m.hksj.raw,}
       \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,}
       \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,}
       \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,}
       \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))}
\KeywordTok{dev.off}\NormalTok{() }
\end{Highlighting}
\end{Shaded}

\textbf{PNG}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{png}\NormalTok{(}\DataTypeTok{file=}\StringTok{'forestplot.png'}\NormalTok{) }
\NormalTok{forest.jama<-}\KeywordTok{forest}\NormalTok{(m.hksj.raw,}
       \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,}
       \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,}
       \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,}
       \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))}
\KeywordTok{dev.off}\NormalTok{() }
\end{Highlighting}
\end{Shaded}

\textbf{Scalable Vector Graphic}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{svg}\NormalTok{(}\DataTypeTok{file=}\StringTok{'forestplot.svg'}\NormalTok{) }
\NormalTok{forest.jama<-}\KeywordTok{forest}\NormalTok{(m.hksj.raw,}
       \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,}
       \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,}
       \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,}
       \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))}
\KeywordTok{dev.off}\NormalTok{() }
\end{Highlighting}
\end{Shaded}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{heterogeneity}{%
\chapter{Between-study Heterogeneity}\label{heterogeneity}}

\includegraphics{schiffchen.jpg}

By now, we have already shown you how to pool effect sizes in a meta-analysis. In meta-analytic pooling, we aim to \textbf{synthesize the effects of many different studies into one single effect}. However, this makes only sense if we aren't comparing \textbf{Apples and Oranges}. For example, it could be the case that while the overall effect we calculate in the meta-analysis is \textbf{small}, there are still a few studies which report \textbf{very high} effect sizes. Such information is lost in the aggregate effect, but it is very important to know if all studies, or interventions, yield small effect sizes, or if there are exceptions.

It could also be the case that even some very \textbf{extreme effect sizes} were included in the meta-analysis, so-called \textbf{outliers}. Such outliers might have even distorted our overall effect, and it is important to know how our overall effect would have looked without them.

The extent to which effect sizes vary within a meta-analysis is called \textbf{heterogeneity}. It is very important to assess heterogeneity in meta-analyses, as high heterogeneity could be caused by the fact that there are actually two or more \textbf{subgroups} of studies present in the data, which have a different true effect. Such information could be very valuable for \textbf{research}, because this might allow us to find certain interventions or populations for which effects are lower or higher.

From a statistical standpoint, high heterogeneity is also problematic. Very high heterogeneity could mean that the studies have nothing in common, and that there is no \textbf{``real'' true effect behind our data}, meaning that it makes no sense to report the pooled effect at all \citep{borenstein2011}.

\begin{rmdinfo}
\textbf{The idea behind heterogeneity}

Rücker and colleagues {[}@rucker2008undue{]} name three types of
heterogeneity in meta-analyses:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \emph{Clinical baseline heterogeneity}. These are differences between
  sample characteristics between the studies. For example, while one
  study might have included rather old people into their study, another
  might have recruited study participants who were mostly quite young.
\item
  \emph{Statistical heterogeneity}. This is the statistical
  heterogeneity we find in our collected effect size data. Such
  heterogeneity migh be either important from a clinical standpoint
  (e.g., when we don't know if a treatment is very or only marginally
  effective because the effects vary much from study to study), or from
  statistical standpoint (because it dilutes the confidence we have in
  our pooled effect)
\item
  \emph{Other sources of heterogeneity}, such as design-related
  heterogeneity.
\end{enumerate}

Point 1. and 3. may be controlled for to some extent by restricting the
scope of our search for studies to certain well-defined intervention
types, populations, and outcomes.

Point 2., on the other hand, has to be assessed once we conducted the
pooling of studies. This is what this chapter focuses on.
\end{rmdinfo}

\begin{rmdinfo}
\textbf{Heterogeneity Measures}

There are \textbf{three types of heterogeneity measures} which are
commonly used to assess the degree of heterogeneity. In the following
examples, \(k\) denotes the individual study, \(K\) denotes all studies
in our meta-analysis, \(\hat \theta_k\) is the estimated effect of \(k\)
with a variance of \(\hat \sigma^{2}_k\), and \(w_k\) is the individual
\textbf{weight} of the study (i.e., its \emph{inverse variance}:
\(w_k = \frac{1}{\hat \sigma^{2}_k}\); see infobox in
\protect\hyperlink{fixed}{Chapter 4.1.1} for more details).

\textbf{1. Cochran's \emph{Q} }

Cochran's \emph{Q}-statistic is the \textbf{difference between the
observed effect sizes and the fixed-effect model estimate} of the effect
size, which is then \textbf{squared, weighted and summed}.

\[ Q = \sum\limits_{k=1}^K w_k (\hat\theta_k  - \frac{\sum\limits_{k=1}^K w_k \hat\theta_k}{\sum\limits_{k=1}^K w_k})^{2}\]

\textbf{2. Higgin's \& Thompson's \emph{I}\textsuperscript{2} }

\(I^{2}\) {[}@higgins2002quantifying{]} is the \textbf{percentage of
variability} in the effect sizes which is not caused by sampling error.
It is derived from \(Q\):

\[I^{2} = max \left\{0, \frac{Q-(K-1)}{Q}  \right\}\]

\textbf{3. Tau-squared}

\(\tau^{2}\) is the between-study variance in our meta-analysis. As we
show in \protect\hyperlink{tau2}{Chapter 4.2.1}, there are various
proposed ways to calculate \(\tau^{2}\)
\end{rmdinfo}

\begin{rmdachtung}
\textbf{Which measure should i use?}

Generally, when we assess and report heterogeneity in a meta-analysis,
we need a measure which is \textbf{robust, and not to easily influenced
by statistical power}.

\textbf{Cochran's \emph{Q} } increases both when the \textbf{number of
studies} (\(k\)) increases, and when the \textbf{precision} (i.e., the
sample size \(N\) of a study) increases. Therefore, \(Q\) and weather it
is \textbf{significant} highly depends on the size of your
meta-analysis, and thus its statistical power. We should therefore not
only rely on \(Q\) when assessing heterogeneity.

\textbf{I\textsuperscript{2}} on the other hand, is not sensitive to
changes in the number of studies in the analyses. \(I^2\) is therefore
used extensively in medical and psychological research, especially since
there is a \textbf{``rule of thumb''} to interpret it
{[}@higgins2003measuring{]}:

\begin{itemize}
\tightlist
\item
  \emph{I}\textsuperscript{2} = 25\%: \textbf{low heterogeneity}
\item
  \emph{I}\textsuperscript{2} = 50\%: \textbf{moderate heterogeneity}
\item
  \emph{I}\textsuperscript{2} = 75\%: \textbf{substantial heterogeneity}
\end{itemize}

Despite its common use in the literature, \(I^2\) not always an adequate
measure for heterogeneity either, because it still heavily depends on
the \textbf{precision} of the included studies {[}@rucker2008undue;
@borenstein2017basics{]}. As said before, \(I^{2}\) is simply the amount
of variability \textbf{not caused by sampling error}. If our studies
become increasingly large, this sampling error tends to \textbf{zero},
while at the same time, \(I^{2}\) tends to 100\% simply because the
single studies have greater \(N\). Only relying on \(I^2\) is therefore
not a good option either.

\textbf{Tau-squared}, on the other hand, is \textbf{insensitive} to the
number of studies, \textbf{and} the precision. Yet, it is often hard to
interpret how relevant our tau-squared is from a practical standpoint.

\textbf{Prediction intervals} (like the ones we automatically calculated
in \protect\hyperlink{pool}{Chapter 4}) are a good way to overcome this
limitation {[}@inthout2016plea{]}, as they take our between-study
variance into account. Prediction intervals give us a range for which we
can \textbf{expect the effects of future studies to fall} based on
\textbf{our present evidence in the meta-analysis}. If our prediction
interval, for example, lies completely on the positive side favoring the
intervention, we can be quite confident to say that \textbf{despite
varying effects, the intervention might be at least in some way
beneficial in all contexts we studied in the future}. If the confidence
interval includes \textbf{zero}, we can be less sure about this,
although it should be noted that \textbf{broad prediction intervals are
quite common, especially in medicine and psychology}.
\end{rmdachtung}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{assessing-the-heterogeneity-of-your-pooled-effect-size}{%
\section{Assessing the heterogeneity of your pooled effect size}\label{assessing-the-heterogeneity-of-your-pooled-effect-size}}

Thankfully, once you've already pooled your effects in meta-analysis using the \texttt{metagen()}, \texttt{metabin()}, or \texttt{metacont} function, it is very easy and straightforward to retrieve the \textbf{three most common heterogeneity measures} that we described before.

In \protect\hyperlink{random.precalc}{Chapter 4.2.2}, we already showed you how to conduct a \textbf{random-effect-model meta-analysis}. In this example, we stored our \textbf{results} in the object \texttt{m.hksj}, which we will use again here.

One way to get heterogeneity measures of my meta-analysis is to \textbf{print} the meta-analysis (in my case, \texttt{m.hksj}) output again.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{print}\NormalTok{(m.hksj)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                           SMD            95%-CI %W(random)
## Call et al.            0.7091 [ 0.1979; 1.2203]        5.2
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]        6.1
## DanitzOrsillo          1.7912 [ 1.1139; 2.4685]        4.2
## de Vibe et al.         0.1825 [-0.0484; 0.4133]        7.1
## Frazier et al.         0.4219 [ 0.1380; 0.7057]        6.8
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]        6.1
## Gallego et al.         0.7249 [ 0.2846; 1.1652]        5.7
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]        5.9
## Hintz et al.           0.2840 [-0.0453; 0.6133]        6.5
## Kang et al.            1.2751 [ 0.6142; 1.9360]        4.3
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]        6.1
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]        5.6
## Phang et al.           0.5407 [ 0.0619; 1.0196]        5.4
## Rasanen et al.         0.4262 [-0.0794; 0.9317]        5.3
## Ratanasiripong         0.5154 [-0.1731; 1.2039]        4.1
## Shapiro et al.         1.4797 [ 0.8618; 2.0977]        4.5
## SongLindquist          0.6126 [ 0.1683; 1.0569]        5.7
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]        5.4
## 
## Number of studies combined: k = 18
## 
##                         SMD            95%-CI    t  p-value
## Random effects model 0.5935 [ 0.3891; 0.7979] 6.13 < 0.0001
## Prediction interval         [-0.2084; 1.3954]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  45.50   17  0.0002
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
\end{verbatim}

We see that this output \textbf{already provides us with all three heterogeneity measures} (and even one more, \emph{H}, which we will not cover here).

\begin{itemize}
\item
  \(\tau^{2}\), as we can see from the \texttt{tau\^{}2} output, is \textbf{0.1337}.
\item
  \(I^{2}\) is printed next to \texttt{I\^{}2}, and has the value \textbf{62.6\%}, and a 95\% confidence interval rangin from 37.9\% to 77.5\%.
\item
  The value of \(Q\) is displayed next to \texttt{Q} under \texttt{Test\ of\ heterogeneity:}.
  As we can see, the value is \textbf{45.50}. In our case, this is highly significant (\(p=0.0002\); see \texttt{p-value}).
\item
  The \textbf{prediction interval} can be found next to \texttt{Prediction\ interval}. As we can see, the 95\% interval ranges from \textbf{g=-0.2084} to \textbf{1.3954}.
\end{itemize}

How can we interpret the values of this example analysis? Well, all three of our indicators suggest that \textbf{moderate to substantial heterogeneity is present in our data}. Given the \textbf{broad prediction interval}, which stretches well below zero, we also cannot be overly confident that the positive effect we found for our interventions is robust in every context. It might be very well possible that the intervention does not yield positive effects in some future scenarios; even a small negative effect might be possible based on the evidence the meta-analysis gives us. Very high effect sizes, on the other hand, are possible too.

\textbf{When the measures are not displayed in my output}

Depending on how you changed the settings of the \texttt{metagen}, \texttt{metabin}, or \texttt{metacont}, it is possible that some of the measures are not displayed in your output. That's not a big deal, because all measures are stored in the object, no matter if they are immediately displayed or not.

To directly access one of the measures, we can to use \texttt{\$} again (see \protect\hyperlink{convertfactors}{Chapter 3.3.1}). We use this \textbf{in combination with our meta-analysis output object} to define which measure we want to see.

Code

Measure

\$Q

Cochran's Q

\$pval.Q

The p-value for Cochran's Q

\$I2

I-squared

\$lower.I2

The lower bound of the I-squared 95\%CI

\$upper.I2

The upper bound of the I-squared 95\%CI

\$tau\^{}2

Tau-squared

\$lower.predict

The lower bound of the 95\% prediction interval

\$upper.predict

The upper bound of the 95\% prediction interval

Here are a few exmaples for my \texttt{m.hksj} object. As you'll see, the output is \textbf{identical} to the one before.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{Q}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 45.50257
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{I2}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.6263947
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{tau}\OperatorTok{^}\DecValTok{2}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.1337024
\end{verbatim}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{detecting-outliers-influential-cases}{%
\section{Detecting outliers \& influential cases}\label{detecting-outliers-influential-cases}}

As mentioned before, \textbf{between-study heterogeneity} can also be caused by one more studies with \textbf{extreme effect sizes} which don't quite \textbf{fit in}. Especially when the \textbf{quality of these studies is low}, or the \textbf{studies are very small}, this may \textbf{distort} our pooled effect estimate, and it's a good idea to have a look on the \textbf{pooled effect again once we remove such outliers from the analysis}.

On the other hand, we also want to know \textbf{if the pooled effect estimate we found is robust}, meaning that the effect does not depend heavily on \textbf{one single study}. Therefore, we also want to know \textbf{whether there are studies which heavily push the effect of our analysis into some direction}. Such studies are called \textbf{influential cases}, and we'll devote some time to this topic in the \protect\hyperlink{influenceanalyses}{second part} of this chapter.

\begin{rmdachtung}
It should be noted that they are \textbf{many methods} to spot
\textbf{outliers and influential cases}, and the methods described here
are not comprehensive. If you want to read more about the underpinnings
of this topic, we can recommend the paper by Wolfgang Viechtbauer and
Mike Cheung {[}@viechtbauer2010outlier{]}.
\end{rmdachtung}

\hypertarget{outliers}{%
\subsection{Searching for extreme effect sizes (outliers)}\label{outliers}}

A common method to detect outliers directly is to define a study as an outlier if the \textbf{study's confidence interval does not overlap with the confidence interval}. This means that we define a study as an outlier when it's effect size estimate is \textbf{so extreme that we have high certainty that the study cannot be part of the ``population'' of effect size we determined when pooling our results} (i.e., the individual study differs significantly from the overall effect).

To detect such outliers in our dataset, the \texttt{filter} function in the \texttt{dplyr} package we introduced in \protect\hyperlink{filter}{Chapter 3.3.3} comes in handy again.

Using this function, we can search for all studies:

\begin{itemize}
\tightlist
\item
  for which the \textbf{upper bound of the 95\% confidence interval is lower than the lower bound of the pooled effect confidence interval} (i.e., extremely small effects)
\item
  for which the \textbf{lower bound of the 95\% confidence interval is higher than the higher bound of the pooled effect confidence interval} (i.e., extremely large effects)
\end{itemize}

Here, i'll use my \texttt{m.hksj} meta-analysis output from \protect\hyperlink{random.precalc}{Chapter 4.2.2} again. Let's see what the \textbf{upper and lower bound of my pooled effect confidence interval} is. As i performed a \textbf{random-effect meta-analysis in this example}, i will use the value stored under \texttt{\$lower.random} and \texttt{\$upper.random}. If you performed a \textbf{fixed-effect meta-analysis}, the objects would be \texttt{\$lower.fixed} and \texttt{\$upper.fixed}, respectively.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{lower.random}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.389147
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{upper.random}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.7979231
\end{verbatim}

Here we go. I now see that my \textbf{pooled effect confidence interval} stretches from \(g = 0.389\) to \(g = 0.798\). We can use these values to filter out outliers now.

To filter out outliers \textbf{automatically}, we have prepared two \textbf{functions} for you, \texttt{spot.outliers.random} and \texttt{spot.outliers.fixed}. Both need the \texttt{dplyr} package (see \protect\hyperlink{select}{Chapter 3.3.3}) to function, so we need to need to have this package \textbf{installed} and \textbf{loaded into our library}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(dplyr)}
\end{Highlighting}
\end{Shaded}

The function we'll use in the case of my \texttt{m.hksj} dataset is \texttt{spot.outliers.random}, because we conducted a \textbf{random-effect meta-analysis to get this output object}. R doesn't know this function yet, so we have to let R learn it copying and pasting the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spot.outliers.random<-}\ControlFlowTok{function}\NormalTok{(data)\{}
\NormalTok{data<-data}
\NormalTok{Author<-data}\OperatorTok{$}\NormalTok{studlab}
\NormalTok{lowerci<-data}\OperatorTok{$}\NormalTok{lower}
\NormalTok{upperci<-data}\OperatorTok{$}\NormalTok{upper}
\NormalTok{m.outliers<-}\KeywordTok{data.frame}\NormalTok{(Author,lowerci,upperci)}
\NormalTok{te.lower<-data}\OperatorTok{$}\NormalTok{lower.random}
\NormalTok{te.upper<-data}\OperatorTok{$}\NormalTok{upper.random}
\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,upperci }\OperatorTok{<}\StringTok{ }\NormalTok{te.lower)}
\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,lowerci }\OperatorTok{>}\StringTok{ }\NormalTok{te.upper)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Now, the function is ready to be used. The only thing we have to tell the \texttt{spot.outliers.random} function is the \textbf{meta-analysis output} that we want to check for outliers, which is defined by \texttt{data}. In my case, this is \texttt{m.hksj}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{spot.outliers.random}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj)}
\end{Highlighting}
\end{Shaded}

This is the output we get from the function:

\begin{verbatim}
##           Author   lowerci  upperci
## 1  DanitzOrsillo 1.1138668 2.468473
## 2 Shapiro et al. 0.8617853 2.097667
\end{verbatim}

We see that the function has detected \textbf{two outliers}. Looking at the \texttt{lowerci} value, the lower bound of the two study's confidence intervals, we see that both have extremely high positive effects, because the lower bounds are both much higher than the \textbf{higher bound of the confidence interval of our pooled effect}, which was \(g = 0.798\).

Thus, we can conduct a \textbf{sensitivity analysis} in which we \textbf{exclude these two outliers}. We can do this with the \texttt{update.meta} function in \texttt{meta}. This creates an update of our meta-analysis output \texttt{m.hksj} without the outliers.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj.outliers<-}\KeywordTok{update.meta}\NormalTok{(m.hksj,}
                             \DataTypeTok{subset =}\NormalTok{ Author }\OperatorTok{!=}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"DanitzOrsillo"}\NormalTok{,}
                                                  \StringTok{"Shapiro et al."}\NormalTok{))}
\NormalTok{m.hksj.outliers}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                           SMD            95%-CI %W(random)
## Call et al.            0.7091 [ 0.1979; 1.2203]        5.0
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]        6.9
## de Vibe et al.         0.1825 [-0.0484; 0.4133]       10.4
## Frazier et al.         0.4219 [ 0.1380; 0.7057]        9.1
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]        7.0
## Gallego et al.         0.7249 [ 0.2846; 1.1652]        6.0
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]        6.4
## Hintz et al.           0.2840 [-0.0453; 0.6133]        8.1
## Kang et al.            1.2751 [ 0.6142; 1.9360]        3.5
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]        7.0
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]        5.8
## Phang et al.           0.5407 [ 0.0619; 1.0196]        5.4
## Rasanen et al.         0.4262 [-0.0794; 0.9317]        5.1
## Ratanasiripong         0.5154 [-0.1731; 1.2039]        3.3
## SongLindquist          0.6126 [ 0.1683; 1.0569]        5.9
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]        5.3
## 
## Number of studies combined: k = 16
## 
##                         SMD           95%-CI    t  p-value
## Random effects model 0.4708 [0.3406; 0.6010] 7.71 < 0.0001
## Prediction interval         [0.0426; 0.8989]              
## 
## Quantifying heterogeneity:
## tau^2 = 0.0361; H = 1.15 [1.00; 1.56]; I^2 = 24.8% [0.0%; 58.7%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  19.95   15  0.1739
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
\end{verbatim}

The entire procedure works the same if you \textbf{conducted a fixed-effect meta-analysis}. However, you need to copy and paste the code for the \texttt{spot.outliers.fixed} function then, which can be found \textbf{below}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spot.outliers.fixed<-}\ControlFlowTok{function}\NormalTok{(data)\{}
\NormalTok{data<-data}
\NormalTok{Author<-data}\OperatorTok{$}\NormalTok{studlab}
\NormalTok{lowerci<-data}\OperatorTok{$}\NormalTok{lower}
\NormalTok{upperci<-data}\OperatorTok{$}\NormalTok{upper}
\NormalTok{m.outliers<-}\KeywordTok{data.frame}\NormalTok{(Author,lowerci,upperci)}
\NormalTok{te.lower<-data}\OperatorTok{$}\NormalTok{lower.fixed}
\NormalTok{te.upper<-data}\OperatorTok{$}\NormalTok{upper.fixed}
\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,upperci }\OperatorTok{<}\StringTok{ }\NormalTok{te.lower)}
\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,lowerci }\OperatorTok{>}\StringTok{ }\NormalTok{te.upper)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{influenceanalyses}{%
\section{Influence Analyses}\label{influenceanalyses}}

We have now showed you how you can detect and remove \textbf{extreme effect sizes} (outliers) in your meta-analysis.

As we've mentioned before in \href{}{Chapter}, however, it is not only statistical outliers which may cause concerns regarding the robustness of our pooled effect. It is also possible that \textbf{some studies in a meta-analysis exert a very high influence on our overall results}. For example, it could be the case that we find that an overall effect is not significant, when in fact, a highly significant effect is consistently found once we remove one particular study in our analysis. Such information is \textbf{highly important once we want to communicate the results of our meta-analysis to the public}.

Here, we present techniques which dig a little deeper than simple outlier removal. To some extent, they are based on the \textbf{Leave-One-Out}-method, in which we \textbf{recalculate the results of our meta-analysis} \(K-1\) times, each times leaving out one study. This way, we can more easily detect \textbf{studies which influence the overall estimate of our meta-analysis the most}, and lets us better assess if this \textbf{influence may distort our pooled effect} \citep{viechtbauer2010outlier}. Thus, such analyses are called \textbf{Influence Analyses}.

We have created the \textbf{function} \texttt{influence.analysis} for you through which influences can be conducted all in one. For this function to work, you need to have the \texttt{meta} and \texttt{metafor} packages installed and loaded in your library.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{influence.analysis<-}\ControlFlowTok{function}\NormalTok{(data,method.tau,hakn)\{}
  
\NormalTok{  influence.data<-data}
\NormalTok{  TE<-data}\OperatorTok{$}\NormalTok{TE}
\NormalTok{  seTE<-data}\OperatorTok{$}\NormalTok{seTE}
\NormalTok{  method.tau<-method.tau}
\NormalTok{  hakn<-hakn}
  
\ControlFlowTok{if}\NormalTok{(hakn }\OperatorTok{==}\StringTok{ }\OtherTok{TRUE}\NormalTok{)\{}
\NormalTok{  res <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{TE, }\DataTypeTok{sei=}\NormalTok{seTE, }\DataTypeTok{measure=}\StringTok{"ZCOR"}\NormalTok{, }
           \DataTypeTok{data=}\NormalTok{influence.data, }
           \DataTypeTok{method =} \KeywordTok{paste}\NormalTok{(method.tau),}
           \DataTypeTok{test=}\StringTok{"knha"}\NormalTok{)}
\NormalTok{  res}
\NormalTok{  inf <-}\StringTok{ }\KeywordTok{influence}\NormalTok{(res)}
\NormalTok{  influence.data<-}\KeywordTok{metainf}\NormalTok{(data)}
\NormalTok{  influence.data}\OperatorTok{$}\NormalTok{I2<-}\KeywordTok{format}\NormalTok{(}\KeywordTok{round}\NormalTok{(influence.data}\OperatorTok{$}\NormalTok{I2,}\DecValTok{2}\NormalTok{),}\DataTypeTok{nsmall=}\DecValTok{2}\NormalTok{)}
  \KeywordTok{plot}\NormalTok{(inf)}
  \KeywordTok{baujat}\NormalTok{(data)}
  \KeywordTok{forest}\NormalTok{(influence.data,}
       \DataTypeTok{sortvar=}\NormalTok{I2,}
       \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),}
       \DataTypeTok{smlab =} \StringTok{"Sorted by I-squared"}\NormalTok{)}
  \KeywordTok{forest}\NormalTok{(influence.data,}
       \DataTypeTok{sortvar=}\NormalTok{TE,}
       \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),}
       \DataTypeTok{smlab =} \StringTok{"Sorted by Effect size"}\NormalTok{)}

\NormalTok{\} }\ControlFlowTok{else}\NormalTok{ \{}
  
\NormalTok{  res <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{TE, }\DataTypeTok{sei=}\NormalTok{seTE, }\DataTypeTok{measure=}\StringTok{"ZCOR"}\NormalTok{, }
           \DataTypeTok{data=}\NormalTok{influence.data, }
           \DataTypeTok{method =} \KeywordTok{paste}\NormalTok{(method.tau))}
\NormalTok{  res}
\NormalTok{  inf <-}\StringTok{ }\KeywordTok{influence}\NormalTok{(res)}
\NormalTok{  influence.data<-}\KeywordTok{metainf}\NormalTok{(data)}
\NormalTok{  influence.data}\OperatorTok{$}\NormalTok{I2<-}\KeywordTok{format}\NormalTok{(}\KeywordTok{round}\NormalTok{(influence.data}\OperatorTok{$}\NormalTok{I2,}\DecValTok{2}\NormalTok{),}\DataTypeTok{nsmall=}\DecValTok{2}\NormalTok{)}
  \KeywordTok{plot}\NormalTok{(inf)}
  \KeywordTok{baujat}\NormalTok{(data)}
  \KeywordTok{forest}\NormalTok{(influence.data,}
       \DataTypeTok{sortvar=}\NormalTok{I2,}
       \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),}
       \DataTypeTok{smlab =} \StringTok{"Sorted by I-squared"}\NormalTok{)}
  \KeywordTok{forest}\NormalTok{(influence.data,}
       \DataTypeTok{sortvar=}\NormalTok{TE,}
       \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),}
       \DataTypeTok{smlab =} \StringTok{"Sorted by Effect size"}\NormalTok{)}
\NormalTok{\}\}  }
\end{Highlighting}
\end{Shaded}

The \texttt{influence.analysis} function has \textbf{three parameters} which we have to define in the function.

Code

Description

data

The output object from our meta-analysis. In my case, this is `data=m.hksj'

method.tau

The method we used to estimate tau-squared (see Chapter 4.2.1). If you haven't set the estimator `method.tau' in your analysis, use `DL' because the DerSimonian-Laird estimator is the default in meta

hakn

Weather we used the Knapp-Hartung-Sidik-Jonkman adjustments. If yes, use hakn=TRUE. If not, use hakn=FALSE

This is how the function code looks for my \texttt{m.hksj} data:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{influence.analysis}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj,}\DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{, }\DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Now, let's have a look at the output.

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-107-1} 

}

\caption{Influence Analyses}\label{fig:unnamed-chunk-107}
\end{figure}
\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-108-1} 

}

\caption{Baujat Plot}\label{fig:unnamed-chunk-108}
\end{figure}

\begin{verbatim}
## Warning in forest.meta(influence.data.metainf, sortvar = I2, rightcols =
## c("TE", : Argument 'sortvar' ignored for objects created with metacum() or
## metainf().
\end{verbatim}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-109-1} 

}

\caption{Leave-One-Out-Analyses}\label{fig:unnamed-chunk-1091}
\end{figure}

\begin{verbatim}
## Warning in forest.meta(influence.data.metainf, sortvar = TE, rightcols =
## c("TE", : Argument 'sortvar' ignored for objects created with metacum() or
## metainf().
\end{verbatim}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-109-2} 

}

\caption{Leave-One-Out-Analyses}\label{fig:unnamed-chunk-1092}
\end{figure}

As you can see, the \texttt{influence.analysis} function gives us various types of \textbf{plots} as output. Let's interpret them one by one.

\textbf{Influence Analyses}

In the first analysis, you can see different influence measures, for which we can see \textbf{graphs including each individual study of our meta-analysis}. This type of \textbf{influence analysis} has been proposed by Viechtbauer and Cheung \citep{viechtbauer2010outlier}. We'll discuss the most important ones here:

\begin{itemize}
\tightlist
\item
  \textbf{dffits}: The DIFFITS value of a study indicates in standard deviations how much the predicted pooled effect changes after excluding this study
\item
  \textbf{cook.d}: The \textbf{Cook's distance} resembles the \textbf{Mahalanobis distance} you may know from outlier detection in conventional multivariate statistics. It is the distance between the value once the study is included compared to when it is excluded
\item
  \textbf{cov.r}: The \textbf{covariance ratio} is the determinant of the variance-covariance matrix of the parameter estimates when the study is removed, divided by the determinant of the variance-covariance matrix of the parameter estimates when the full dataset is considered. Importantly, values of cov.r \textless{} 1 indicate that removing the study will lead to a more precise effect size estimation (i.e., less heterogeneity).
\end{itemize}

Usually, however, you don't have to dig this deep into the calculations of the individual measures. As a rule of thumb, \textbf{influential cases} are studies with \textbf{very extreme values in the graphs}. Viechtbauer and Cheung have also proposed cut-offs when to define a a study as an influential case, for example (with \(p\) being the number of model coefficients and \(k\) the number of studies):

\[ DFFITS > 3\times\sqrt{\frac{p}{k-p}}\]
\[ hat > 3\times\frac{p}{k}\]

If a case was determined being \textbf{an influential case using these cut-offs}, its value will be displayed in \textbf{red} (in our example, this is the case for study number 3).

\begin{rmdachtung}
Please note, as Viechtbauer \& Cheung emphasize, that \textbf{these
cut-offs are set on somewhat arbitrary thresholds}. Therefore, you
should never only have a look on the color of the study, but the general
structure of the graph, and interpret results in context.

In our example, we see that while only Study 3 is defined as an
influential case, there are \textbf{actually two spiked in most plots},
while the other studies all quite have the same value. Given this
structure, we could also decide to define \textbf{Study 16} as an
influential case too, because its values are very extreme too.
\end{rmdachtung}

Let's have a look what the \textbf{3rd and 16th study} in our \texttt{m.hksj} meta-analysis output were.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{studlab[}\KeywordTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{,}\DecValTok{16}\NormalTok{)]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "DanitzOrsillo"  "Shapiro et al."
\end{verbatim}

This is an interesting finding, as we \textbf{detected the same studies when only looking at statistical outliers}. This further corroborates that these two studies could maybe have distorted our pooled effect estimate, and \textbf{might cause parts of the between-group heterogeneity we found in our meta-analysis}.

\textbf{Baujat Plot}

The Baujat Plot \citep{baujat2002graphical} is a diagnostic plot to detect studies \textbf{overly contributing to the heterogeneity of a meta-analysis}. The plot shows the contribution of each study to the overall heterogeneity as measured by Cochran's \emph{Q} on the \textbf{horizontal axis}, and its \textbf{influence on the pooled effect size} on the vertical axis. As we want to assess heterogeneity and studies contributing to it, all studies \textbf{on the right side of the plot are important to look at}, as this means that they cause much of the heterogeneity we observe. \textbf{This is even more important when a study contributes much to the overall heterogeneity, while at the same time being not very influential concerning the overall pooled effect} (e.g., because the study had a very small sample size). Therefore, \textbf{all studies on the right side of the Baujat plot}, especially in the \textbf{lower part}, are important for us.

As you might have already recognized, the only two \textbf{studies we find in those regions of the plot are the two studies we already detected before} (Danitz \& Orsillo, Shapiro et al.). These studies don't have a large impact on the overall results (presumably because they are very small), but they do \textbf{add substantially to the heterogeneity we found in the meta-analysis}.

\textbf{Leave-One-Out Analyses}

In these to forest plots, we see the \textbf{pooled effect recalculated, with one study omitted each time}. There are two plots, which provide the same data, but are ordered by different values.

The \textbf{first plot is ordered by heterogeneity (low to high), as measured by \emph{I}\textsuperscript{2} }. We see in the plot that the lowest \emph{I}\textsuperscript{2} heterogeneity is reached (as we've seen before) by omitting the studies \textbf{Danitz \& Orsillo} and \textbf{Shapiro et al.}. This again corroborates our finding that these two studies were the main ``culprits'' for the between-study heterogeneity we found in the meta-analysis.

The \textbf{second plot is ordered by effect size (low to high)}. Here, we see how the overall effect estimate changes with one study removed. Again, as the two outlying studies have very high effect sizes, we find that the overall effect is smallest when they are removed.

All in all, the results of our \textbf{outlier and influence analysis} in this example point in the \textbf{same direction}. The two studies are probably \textbf{outliers which may distort the effect size estimate}, as well as its \textbf{precision}. We should therefore also conduct and report a \textbf{sensitivity analysis in which these studies are excluded}.

\begin{rmdachtung}
\textbf{The influence analysis function for fixed-effect-model
meta-analyses}

The \texttt{influence.analysis} function we presented above can only be
used for \textbf{random-effect meta-analyses}. If you want to perform
influence analyses for meta-analyses in \textbf{which you pooled the
effects with a fixed-effect model}, you will have to use the
\texttt{influence.analysis.fixed} function, which can be found
\href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/influence_analysis_function_for_fixed_effect_model.R}{here}.

To use this function, \textbf{you only have to set the parameter}
\texttt{data}, as \texttt{method.tau} and \texttt{hakn} only apply to
random-effect-models.
\end{rmdachtung}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{subgroup}{%
\chapter{Subgroup Analyses}\label{subgroup}}

\includegraphics{subgroup.jpg}

In \protect\hyperlink{heterogeneity}{Chapter 6}, we discussed in depth why \textbf{between-study heterogeneity} is such an important issue in interpreting the results of our meta-analysis, and how we can \textbf{explore sources of heterogeneity} using \protect\hyperlink{outliers}{outlier} and \protect\hyperlink{influenceanalyses}{influence analyses}.

Another source of between-study heterogeneity making our effect size estimate less precise could be that \textbf{there are slight differences in the study design or intervention components between the studies}. For example, in a meta-analysis on the effects of \textbf{cognitive behavioral therapy} (CBT) for \textbf{depression} in \textbf{university students}, it could be the case that some studies delivered the intervention in a \textbf{group setting}, while others delivered the therapy to each student \textbf{individually}. In the same example, it is also possible that studies used different \textbf{criteria} to determine if a student suffers from \textbf{depression} (e.g.~they either used the \emph{ICD-10} or the \emph{DSM-5} diagnostic manual).

Many other differences of this sort are possible, and it seems plausible that such study differences may also be associated with differences in the overall effect.

In \textbf{subgroup analyses}, we therefore have a look at different \textbf{subgroups within the studies of our meta-analysis} and try to determine of the \textbf{differ between these subgroups}.

\begin{rmdinfo}
\textbf{The idea behind subgroup analyses}

Basically, a every subgroup analysis consists of \textbf{two parts}: (1)
\textbf{pooling the effect of each subgroup}, and (2) \textbf{comparing
the effects of the subgroups} {[}@borenstein2013meta{]}.

\textbf{1. Pooling the effect of each subgroup}

This point it rather straightforward, as the same criteria as the ones
for a \textbf{simple meta-analysis without subgroups} (see
\protect\hyperlink{pool}{Chapter 4} and
\protect\hyperlink{random}{Chapter 4.2}) apply here.

\begin{itemize}
\tightlist
\item
  If you assume that \textbf{all studies in subgroup} stem from the same
  population, and all have \textbf{one shared true effect}, you may use
  the \textbf{fixed-effect-model}. As we mention in
  \protect\hyperlink{pool}{Chapter 4}, many \textbf{doubt} that this
  assumption is ever \textbf{true in psychological} and \textbf{medical
  research}, even when we partition our studies into subgroups.
\item
  The alternative, therefore, is to use a \textbf{random-effect-model}
  which assumes that the studies within a subgroup are drawn from a
  \textbf{universe} of populations follwing its own distribution, for
  which we want to estimate the \textbf{mean}.
\end{itemize}

\textbf{2. Comparing the effects of the subgroups}

After we calculated the pooled effect for each subgroup, \textbf{we can
compare the size of the effects of each subgroup}. However, to know if
this difference is in fact singnificant and/or meaningful, we have to
calculate the \textbf{Standard Error of the differences between subgroup
effect sizes} \(SE_{diff}\), to calculate \textbf{confidence intervals}
and conduct \textbf{significance tests}. There are \textbf{two ways to
calculate} \(SE_{diff}\), and both based on different assumptions.

\begin{itemize}
\tightlist
\item
  \textbf{Fixed-effects (plural) model}: The fixed-effects-model for
  subgroup comparisons is appropriate when \textbf{we are only
  interested in the subgroups at hand} {[}@borenstein2013meta{]}. This
  is the case when \textbf{the subgroups we chose to examine} were not
  randomly ``chosen'', but represent fixed levels of a characteristic we
  want to examine. Gender is such a characteristic, as its two subgroups
  \textbf{female} and \textbf{male} were not randomly chosen, but are
  the two subgroups that gender (in its classical conception) has. Same
  does also apply, for example, if we were to examine if studies in
  patients with \textbf{clinical depression} versus \textbf{subclinical
  depression} yield different effects. Borenstein and Higgins
  {[}@@borenstein2013meta{]} argue that the \textbf{fixed-effects
  (plural) model} may be the \textbf{only plausible model} for most
  analysis in \textbf{medical research, prevention, and other fields}.
\end{itemize}

As this model assumes that \textbf{no further sampling error is
introduced at the subgroup level} (because subgroups were not randomly
sampled, but are fixed), \(SE_{diff}\) only depends on the
\emph{variance within the subgroups} \(A\) and \(B\), \(V_A\) and
\(V_B\).

\[V_{Diff}=V_A + V_B\]

The fixed-effects (plural) model can be used to test differences in the
pooled effects between subgroups, while the pooling \textbf{within the
subgroups is still conducted using a random-effects-model}. Such a
combination is sometimes called a \textbf{mixed-effects-model}. We'll
show you how to use this model in R in the
\protect\hyperlink{mixed}{next chapter}.

\begin{itemize}
\tightlist
\item
  \textbf{Random-effects-model}: The random-effects-model for
  between-subgroup-effects is appropriate when the \textbf{subgroups we
  use were randomly sampled from a population of subgroups}. Such an
  example would be if we were interested if the effect of an
  intervention \textbf{varies by region} by looking at studies from 5
  different countries (e.g., Netherlands, USA, Australia, China,
  Argentina). These variable ``region'' has many different potential
  subgroups (countries), from which we randomly selected five means that
  this has introduced a \textbf{new sampling error}, for which we have
  to control for using the \textbf{random-effects-model} for
  between-subgroup-comparisons.
\end{itemize}

The (simplified) formula for the estimation of \(V_{Diff}\) using this
model therefore looks like this:

\[V_{Diff}=V_A + V_B + \frac{\hat T^2_G}{m} \]

Where \(\hat T^2_G\) is the \textbf{estimated variance between the
subgroups}, and \(m\) is the \textbf{number of subgroups}.
\end{rmdinfo}

\begin{rmdachtung}
Be aware that subgroup analyses should \textbf{always be based on an
informed, \emph{a priori} decision} which subgroup differences within
the study might be \textbf{practically relevant}, and would lead to
information gain on relevant \textbf{research questions} in your field
of research. It is also \textbf{good practice} to specify your subgroup
analyses \textbf{before you do the analysis}, and list them in
\textbf{the registration of your analysis}.

It is also important to keep in mind that \textbf{the capabilites of
subgroup analyses to detect meaningful differences between studies is
often limited}. Subgroup analyses also need \textbf{sufficient power},
so it makes no sense to compare two or more subgroups when your entire
number of studies in the meta-analysis is smaller than \(k=10\)
{[}@higgins2004controlling{]}.
\end{rmdachtung}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{mixed}{%
\section{Subgroup Analyses using the Mixed-Effects-Model}\label{mixed}}

To conduct subgroup analyses using the \textbf{Mixed-Effects-Model} (random-effects-model within subgroups, fixed-effects-model between subgroups), you can use the \texttt{subgroup.analysis.mixed.effects} function we prepared for you. To use the function, \texttt{meta} and \texttt{metafor} need to be installed and loaded in your library.

As the code for the function is pretty long, we \textbf{don't display it here}. To access the function, use this \href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/subgroup_analyses_mixed_effects_function.R}{link}. Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code from the website \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

For the \texttt{subgroup.analysis.mixed.effects} function, the following parameters have to be set:

Code

Description

m

The output of you meta-analysis. In my case, this is `m.hksj'

subgroups

The variable in our dataset in which we coded which study belongs to which subgroup. Note that we also tell the function in which dataset this variable is stored. In my case, this was the `madata' dataset i used to get the meta-analysis output `m.hksj'. The dataset and the subgroup variable have to be connected with \$ (e.g.~madata\$Control). It is important that the subgroup variable is defined for each effect size we used to calculate the initial meta-analysis

exclude

This controls if one or more subgroups should be excluded from the subgroup analysis. To exclude one or more subgroups, we have to provide the function with the name of the subgroup as it appears in our dataframe (in quotation marks; e.g., exclude=`psychology'). One can also exclude more than one subgroup at the same time. To do this, one has to concatenate the names of the subgroups we want to exclude (e.g.~exclude = c(`psychology',`general')). If you do not want to exclude subgroups (as is advised), you can leave out this parameter.

In my \texttt{madata} dataset, which i used previously to generate my meta-analysis output \texttt{m.hksj}, i stored the subgroup variable \texttt{Control}. This variable specifies \textbf{which control group type was employed in which study}. There are \textbf{three subgroups}: \texttt{WLC} (waitlist control), \texttt{no\ intervention} and \texttt{information\ only}.

The function to do a subgroup analysis using the mixed-effects-model with these parameters looks like this.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{subgroup.analysis.mixed.effects}\NormalTok{(}\DataTypeTok{m=}\NormalTok{m.hksj,}
                                \DataTypeTok{subgroups=}\NormalTok{madata}\OperatorTok{$}\NormalTok{Control)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Subgroup Results:
## --------------
##           Subgroup k        TE       seTE  LLCI  ULCI            p
## 1 information only 3 0.4048358 0.07875304 0.250 0.559 2.739011e-07
## 2  no intervention 8 0.5107720 0.11945084 0.277 0.745 1.902797e-05
## 3              WLC 7 0.7836032 0.20993337 0.372 1.195 1.894924e-04
##           Q   I2 I2.lower I2.upper
## 1  1.144426 0.00     0.00     0.82
## 2 16.704467 0.58     0.08     0.81
## 3 22.167163 0.73     0.42     0.87
## 
## Test for subgroup differences (mixed/fixed-effects (plural) model):
## --------------
##                       Q df         p
## Between groups 3.031562  2 0.2196365
## 
## - Total number of studies included in subgroup analysis:  18
## - Tau estimator used for within-group pooling:  SJ
\end{verbatim}

The results of the subgroup analysis are displayed under \texttt{Subgroup\ Results}. We also see that, while the \textbf{pooled effects of the subgroups differ quite substantially} (\emph{g} = 0.41-0.78), this difference is \textbf{not statistically significant}.

This can be seen under \texttt{Test\ for\ subgroup\ differences\ (mixed/fixed-effects\ (plural)\ model)} in the \texttt{Between\ groups} row. We can see that \(Q=3.03\) and \(p=0.2196\). This information can be reported in our meta-analysis paper.

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{subgroup-analyses-using-the-random-effects-model}{%
\section{Subgroup Analyses using the Random-Effects-Model}\label{subgroup-analyses-using-the-random-effects-model}}

Now, let's assume i want to \textbf{know if intervention effects in my meta-analysis differ by region}. I use a \textbf{random-effects-model} and the selected coutries Argentina, Australia, China, and the Netherlands.

Again, i use the \texttt{m.hksj} meta-analysis output object. I can perform a random-effects-model for between-subgroup-differences using the \texttt{update.meta} function. For this function, we have to \textbf{set two parameters}.

Code

Description

byvar

Here, we specify the variable in which the subgroup of each study is stored

comb.random

Weather we want to use a random-effects-model for between-subgroup-differences. In this case, we have to set comb.random = TRUE

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{region.subgroup<-}\KeywordTok{update.meta}\NormalTok{(m.hksj, }
                             \DataTypeTok{byvar=}\NormalTok{region, }
                             \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{, }
                             \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{)}
\NormalTok{region.subgroup}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                                          95%-CI %W(random)      region
## Call et al.            0.7091 [ 0.1979; 1.2203]        5.2 Netherlands
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]        6.1 Netherlands
## DanitzOrsillo          1.7912 [ 1.1139; 2.4685]        4.2 Netherlands
## de Vibe et al.         0.1825 [-0.0484; 0.4133]        7.1         USA
## Frazier et al.         0.4219 [ 0.1380; 0.7057]        6.8         USA
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]        6.1         USA
## Gallego et al.         0.7249 [ 0.2846; 1.1652]        5.7         USA
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]        5.9   Argentina
## Hintz et al.           0.2840 [-0.0453; 0.6133]        6.5   Argentina
## Kang et al.            1.2751 [ 0.6142; 1.9360]        4.3   Argentina
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]        6.1   Australia
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]        5.6   Australia
## Phang et al.           0.5407 [ 0.0619; 1.0196]        5.4   Australia
## Rasanen et al.         0.4262 [-0.0794; 0.9317]        5.3       China
## Ratanasiripong         0.5154 [-0.1731; 1.2039]        4.1       China
## Shapiro et al.         1.4797 [ 0.8618; 2.0977]        4.5       China
## SongLindquist          0.6126 [ 0.1683; 1.0569]        5.7       China
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]        5.4       China
## 
## Number of studies combined: k = 18
## 
##                                       95%-CI    t  p-value
## Random effects model 0.5935 [0.3891; 0.7979] 6.13 < 0.0001
## 
## Quantifying heterogeneity:
## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%]
## 
## Quantifying residual heterogeneity:
## H = 1.68 [1.27; 2.24]; I^2 = 64.7% [37.6%; 80.0%]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  45.50   17  0.0002
## 
## Results for subgroups (random effects model):
##                        k                   95%-CI     Q  tau^2   I^2
## region = Netherlands   3 0.9142 [-0.9150; 2.7433] 13.06 0.4508 84.7%
## region = USA           4 0.4456 [ 0.0600; 0.8312]  6.87 0.0357 56.3%
## region = Argentina     3 0.6371 [-0.5837; 1.8580]  6.95 0.1826 71.2%
## region = Australia     3 0.3194 [-0.2427; 0.8815]  2.13 0.0204  6.1%
## region = China         5 0.7098 [ 0.2018; 1.2177]  7.81 0.1110 48.8%
## 
## Test for subgroup differences (random effects model):
##                     Q d.f. p-value
## Between groups   4.52    4  0.3405
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
\end{verbatim}

Here, we get the \textbf{pooled effect for each subgroup} (country). Under \texttt{Test\ for\ subgroup\ differences\ (random\ effects\ model)}, we can see the \textbf{test for subgroup differences using the random-effects-model}, which is \textbf{not significant} (\(Q=4.52\),\(p=0.3405\)). This means that we did not find differences in the overall effect between different regions, represented by the country in which the study was conducted.

\begin{rmdachtung}
\textbf{Using a fixed-effect-model for within-subgroup-pooling and a
fixed-effects-model for between-subgroup-differences}

To use a fixed-effect-model in combination with a fixed-effects-model,
we can also use the \texttt{update.meta} function again. The procedure
is the same as the one we described before, but we have to set
\texttt{comb.random} as \texttt{FALSE} and \texttt{comb.fixed} as
\texttt{TRUE}.
\end{rmdachtung}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{metareg}{%
\chapter{Meta-Regression}\label{metareg}}

\includegraphics{regressionbild.jpg}

Conceptually, \textbf{Meta-Regression} does not differ much from a \textbf{subgroup analysis}. In fact, subgroup analyses with more than two groups are nothing more than a meta-regression with categorial covariates. However, meta-regression does also allow us to use \textbf{continuous data} as covariates and check weather values of this variable are associated with effect size.

\begin{rmdinfo}
\textbf{The idea behind meta-regression}

You may have already performed regressions in regular data where
participants or patients are the \textbf{unit of analysis}. In typical
Meta-Analyses, we do not have the individual data for each participant
available, but only the \textbf{aggregated effects}, which is why we
have to perform meta-regressions with covariates at the \textbf{study
level}. This also means that while we conduct analyses on participant
samples much larger than usual in single studies, it is still very
likely that \textbf{we don't have enough studies for a meta-regression
to be sensible}. In \protect\hyperlink{subgroup}{Chapter 7}, we told you
that subgroup analyses make no sense when \emph{k}\textless{}10. For
\textbf{meta-regression}, Borenstein and colleages {[}@borenstein2011{]}
recommend that \textbf{each covariate should at least contain ten
studies}, although this should not be seen as clear rule.

In a conventional regression, we want to estimate a parameter \(y\)
using a covariate \(x_i\) with \(n\) regression coefficients \(\beta\).
A standard regression equation therefore looks like this:

\[y=\beta_0 + \beta_1x_1 + ...+\beta_nx_n\]

In a meta-regression, we want to estimate the \textbf{effect size}
\(\theta\) for different values of the covariate(s), so our regression
looks like this:

\[\hat \theta_k = \theta + \beta_1x_{1k} + ... + \beta_nx_{nk} + \epsilon_k + \zeta_k\]

You might have seen that when estimating the effect size \(\theta_k\) of
a study \(k\) in our regression model, there are two \textbf{extra terms
in the equation}, \(\epsilon_k\) and \(\zeta_k\). The same terms can
also be found in the equation for the random-effects-model in
\protect\hyperlink{random}{Chapter 4.2}. The two terms signify two types
of \textbf{independent errors} which cause our regression prediction to
be \textbf{imperfect}. The first one, \(\epsilon_k\), is the sampling
error through which the effect size of the study deviates from its
``true'' effect. The second one, \(\zeta_k\), denotes that even the true
effect size of the study is only sampled from \textbf{an overarching
distribution of effect sizes} (see the Chapter on the
\protect\hyperlink{random}{Random-Effects-Model} for more details). In a
\textbf{fixed-effect-model}, we assume that all studies actually share
the \textbf{same true effect size} and that the \textbf{between-study
heterogeneity} \(\tau^2 = 0\). In this case, we do not consider
\(\zeta_k\) in our equation, but only \(\epsilon_k\).

As the equation above has includes \textbf{fixed effects} (the \(\beta\)
coefficients) as well as \textbf{random effects} (\(\zeta_k\)), the
model used in meta-regression is often called \textbf{a
mixed-effects-model}. Mathematically, this model is identical to the
\textbf{mixed-effects-model} we described in
\protect\hyperlink{subgroup}{Chapter 7} where we explained how
\textbf{subgroup analyses} work.

Indeed \textbf{subgroup analyses with more than two subgroups} are
nothing else than a \textbf{meta-regression} with a \textbf{categorical
predictor}. For meta-regression, these subgroups are then
\textbf{dummy-coded}, e.g.

\[ D_k =  \{\begin{array}{c}0:ACT \\1:CBT \end{array}\]

\[\hat \theta_k = \theta + \beta x_{k} + D_k \gamma + \epsilon_k + \zeta_k\]

In this case, we assume the same \textbf{regression line}, which is
simply ``shifted'' \textbf{up or down for the different subgroups}
\(D_k\).
\end{rmdinfo}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-125-1} 

}

\caption{Visualisation of a Meta-Regression with dummy-coded categorial predictors}\label{fig:unnamed-chunk-125}
\end{figure}

\begin{rmdinfo}
\textbf{Assessing the fit of a regression model}

To evaluate the \textbf{statistical significance of a predictor}, we a
\textbf{t-test} of its \(\beta\)-weight is performed.

\[ t=\frac{\beta}{SE_{\beta}}\]

Which provides a \(p\)-value telling us if a variable significantly
predicts effect size differences in our regression model.

If we fit a regression model, our aim is to find a model \textbf{which
explains as much as possible of the current variability in effect sizes}
we find in our data.

In conventional regression, \(R^2\) is commonly used to quantify the
\textbf{goodness of fit} of our model in percent (0-100\%). As this
measure is commonly used, and many researchers know how to to interpret
it, we can also calculate a \(R^2\) analog for meta-regression using
this formula:

\[R_2=\frac{\hat\tau^2_{REM}-\hat\tau^2_{MEM}}{\hat\tau^2_{REM}}\]

Where \(\hat\tau^2_{REM}\) is the estimated total heterogenetiy based on
the random-effects-model and \(\hat\tau^2_{REM}\) the total
heterogeneity of our mixed-effects regression model.
\end{rmdinfo}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{calculating-meta-regressions-in-r}{%
\section{Calculating meta-regressions in R}\label{calculating-meta-regressions-in-r}}

Meta-regressions can be conducted in R using the \texttt{metareg} function in \texttt{meta}. To show the similarity between \texttt{subgroup} analysis and \texttt{meta-regression} with categorical predictors, i'll first conduct a meta-regression with my variable ``Control'' as predictor again.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{metareg}\NormalTok{(m.hksj,Control)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Mixed-Effects Model (k = 18; tau^2 estimator: SJ)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.1343 (SE = 0.0536)
## tau (square root of estimated tau^2 value):             0.3665
## I^2 (residual heterogeneity / unaccounted variability): 73.92%
## H^2 (unaccounted variability / sampling variability):   3.84
## R^2 (amount of heterogeneity accounted for):            0.00%
## 
## Test for Residual Heterogeneity: 
## QE(df = 15) = 40.0161, p-val = 0.0005
## 
## Test of Moderators (coefficient(s) 2:3): 
## F(df1 = 2, df2 = 15) = 0.9467, p-val = 0.4100
## 
## Model Results:
## 
##                         estimate      se    tval    pval    ci.lb   ci.ub
## intrcpt                   0.4252  0.2250  1.8899  0.0782  -0.0543  0.9048
## Controlno intervention    0.1003  0.2678  0.3744  0.7134  -0.4706  0.6711
## ControlWLC                0.3380  0.2765  1.2224  0.2404  -0.2514  0.9274
##                          
## intrcpt                 .
## Controlno intervention   
## ControlWLC               
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

We see in the output that the \texttt{metareg} function uses the values of ``Control'' (i.e, the three different types of control groups) as a \textbf{moderator}. It takes \textbf{``information only''} as a dummy-coded \emph{reference group}, and \textbf{``no intervention''} and \textbf{``WLC''} as dummy-coded \textbf{predictors}. Under \texttt{Test\ of\ Moderators}, we can see that control groups are not significantly associated with effect size differences \(F_{2,15}=0.947\), \(p=0.41\). Our regression model does not explain any of the variability in our effect size data (\(R^2=0\%\)).

Below \texttt{Model\ Results}, we can also see the \(\beta\)-values (\texttt{estimate}) of both predictors, and their significance level \texttt{pval}. As we can see, both predictors were not significant.

\textbf{Continuous variables}

Let's assume i want to check if the \textbf{publication year} is associated with effect size. I have stored the variable \texttt{pub\_year}, containing the publication year of every study in my dataset, and conducted the meta-analysis with it. I stored my meta-analysis output in the \texttt{m.pubyear} output.

\textbf{Now, i can use this predictor in a meta-regression.}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{output.metareg<-}\KeywordTok{metareg}\NormalTok{(m.pubyear,pub_year)}
\NormalTok{output.metareg}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Mixed-Effects Model (k = 18; tau^2 estimator: DL)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.0831 (SE = 0.0488)
## tau (square root of estimated tau^2 value):             0.2883
## I^2 (residual heterogeneity / unaccounted variability): 64.69%
## H^2 (unaccounted variability / sampling variability):   2.83
## R^2 (amount of heterogeneity accounted for):            0.00%
## 
## Test for Residual Heterogeneity: 
## QE(df = 16) = 45.3076, p-val = 0.0001
## 
## Test of Moderators (coefficient(s) 2): 
## QM(df = 1) = 0.0054, p-val = 0.9412
## 
## Model Results:
## 
##           estimate       se     zval    pval     ci.lb    ci.ub   
## intrcpt    -1.4580  27.6151  -0.0528  0.9579  -55.5825  52.6666   
## pub_year    0.0010   0.0137   0.0737  0.9412   -0.0259   0.0280   
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

As you can see from the output, \texttt{pub\_year} was now included as a \textbf{predictor}, but it is not significantly associated with the effect size (\(p=0.9412\)).

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{plotting-regressions}{%
\section{Plotting regressions}\label{plotting-regressions}}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-130-1} 

}

\caption{A finished bubble plot}\label{fig:unnamed-chunk-130}
\end{figure}

To plot our meta-regression output, we can use the \texttt{bubble} function in \texttt{meta}. Here a few parameters we can specify for this function.

Parameter

Function

xlim

The x-axis limit of the plot. Must be specified as, e.g., `xlim=c(0,1)'

ylim

The y-axis limit of the plot. Must be specified as, e.g., `ylim=c(0,1)'

xlab

The label for the x axis

ylab

The label for the y axis

col

The color of the individual studies

lwd

The line width of the regression line

col.line

The color of the regression line

studlab

If the labels for each study should be printed within the plot (TRUE/FALSE)

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{bubble}\NormalTok{(output.metareg,}
       \DataTypeTok{xlab =} \StringTok{"Publication Year"}\NormalTok{,}
       \DataTypeTok{col.line =} \StringTok{"blue"}\NormalTok{,}
       \DataTypeTok{studlab =} \OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\hypertarget{multiple-meta-regression}{%
\section{Multiple Meta-Regression}\label{multiple-meta-regression}}

\includegraphics{mvreg.jpg}

In the beginning of \protect\hyperlink{metareg}{Chapter 8}, we already introduced the basic idea behind meta-regression. We showed that, in general, a meta-regression has the following form:

\[\hat \theta_k = \theta + \beta_1x_{1k} + ... + \beta_nx_{nk} + \epsilon_k + \zeta_k\]

Where \(\hat \theta_k\) is the estimate of our effect size of a study \(k\) and \(\beta\) are the regression coefficients. There are two parameters in this formula which cause our effect size estimate to deviate from the ``true'' overall effect size: \(\epsilon_k\), the sampling error, and \(\zeta_k\), which denotes that the true effect size of the study is only sampled from an \textbf{overarching distribution of effect sizes}. As we explained before, if \(\zeta_k>0\), the model we describe with this formula is a \textbf{random-effects meta-regression model}, also known as a \emph{mixed-effects model}.

\begin{rmdinfo}
\textbf{The idea behind multiple meta-regression}

Previously, we only considered the scenario in which we use \textbf{one
predictor} \(\beta_1x_1\) in our meta-regression (e.g., we want to check
if the effect size a study reports depends on the year it was
published). But imagine another scenario. Let's assume that we have made
the experience in our previous research that the effect sizes we find in
a research field for a certain study depends on how prestigious the
journal in which the study was published is (e.g., higher effects are
reported in journals with a high reputation). On the other hand, it also
seems quite logical that journals with a good reputation publish studies
of high quality, which may also be associated with higher effect sizes.
So to check if journal reputation is indeed associated with higher
effect sizes, we have to make sure that this relationship is not
\textbf{confounded} by the fact that prestigious journals publish
studies of higher quality, which may be the true reason we find this
relationship between journal reputation and effect size. This means we
have to \textbf{control} for study quality (e.g., rated on a scale from
0 to 10) when we examine the relationship between journal prestige
(e.g., operationalized as the
\href{https://en.wikipedia.org/wiki/Impact_factor}{impact factor} of a
journal) and effect size.

This, and many other possible scenarios can be dealt with using
\textbf{multiple meta-regression}. In multiple meta-regression we use
several predictors (variables) to predict (differences in) effect sizes.
When we look back at the \textbf{general meta-regression} formula we
defined before, we actually see that the formula already provides us
with this feature through the \(\beta_nx_{nk}\) part. Here, the
parameter \(n\) denotes that we can include \(n\) more
predictors/variables into our meta-regression, making it a multiple
meta-regression.

While, statistically speaking, the \(n\) parameter gives us the
possibility to include as many predictors into our multiple
meta-regression model as we wish to, things are a little more tricky in
reality. In the following, we will discuss a few important pitfalls in
multiple meta-regression and how we can build multiple meta-regression
models which are robust and trustworthy. But first, let's cover another
important feature of multiple meta-regression: \textbf{interactions}.
\end{rmdinfo}

\begin{rmdinfo}
\textbf{Interactions}

So far, in our multiple meta-regression model, we only considered the
case where we have multiple predictor variables \(x_1,x_2, ... x_n\),
and along with their predictor estimates \(\beta_n\), \textbf{add them
together} to calculate our estimate of the true effect size
\(\hat \theta_k\) for each study \(k\). In multiple meta-regression
models, however, we can not only model such \textbf{additive
relationships}. We can also model so-called \textbf{interactions}.
Interactions mean that the \textbf{relationship} between one
\textbf{predictor variable} (e.g., \(x_1\)) and the \textbf{estimated
effect size} is different for different values of another predictor
variable (e.g. \(x_2\)).

Imagine a scenario where we want to model two predictors and their
relationship to the effect size: the \textbf{publication year} (\(x_1\))
of a study and the \textbf{quality} (\(x_2\)) of a study, which we rate
like this:

\[\begin{equation}
  x_2 = \left \{\begin{array}{ll}
      0: bad
      \\1: moderate
      \\2: good
    \end{array}
  \right.
\end{equation}\]

As we described before, we can now imagine a meta-regression model in
which we combine these two predictors \(x_1\) and \(x_2\) and assume an
additive relationship. We can do this by simply adding them:

\[\hat \theta_k = \theta + \beta_1x_{1k} + \beta_2x_{2k} + \epsilon_k + \zeta_k\]

Let's assume that, overall, higher publication year (\(x_1\)) is
associated with higher effect sizes (i.e., reported effect sizes have
risen over the years). We could now ask ourselves if this positive
relationship \textbf{varies} depending on the quality of the studies
(\(x_2\)). For example, maybe this rise in effect sizes was strongest
for high-quality studies, while effect sizes stayed mostly the same over
the years for studies of lower quality? We can visualize our assumed
relationship between effect size (\(\hat \theta_k\)), publication year
(\(x_1\)) and study quality (\(x_2\)) the following way:

\includegraphics{vis2.png}

To examine such questions, we can add an \textbf{interaction term} to
our meta-regression model. This interaction term lets predictions of
\(x_1\) vary for different values of \(x_2\) (and vice versa). We can
denote this additional interactional relationship in our model by
introducing a third predictor, \(\beta_3\), which captures this
interaction \(x_{1k}x_{2k}\) we want to test in our model:

\[\hat \theta_k = \theta + \beta_1x_{1k} + \beta_2x_{2k} + \beta_3x_{1k}x_{2k}+ \epsilon_k + \zeta_k\]
\end{rmdinfo}

\hypertarget{pitfalls}{%
\subsection{Common pitfalls of multiple meta-regression models}\label{pitfalls}}

As we have mentioned before, multiple meta-regression, while \textbf{very useful when applied properly}, comes with \textbf{certain caveats} we have to know and consider when fitting a model. Indeed, some argue that (multiple) meta-regression is \textbf{often improperly used and interpreted in practice}, leading to a low validity of many meta-regression models \citep{higgins2004controlling}. Thus, there are some points we have to keep in mind when fitting multiple meta-regression models, which we will describe in the following.

\begin{rmdachtung}
\textbf{Overfitting: seeing a signal when there is none}

To better understand the risks of (multiple) meta-regression models, we
have to understand the concept of \textbf{overfitting}. Overfitting
occurs when we build a statistical model which fits the data \textbf{too
closely}. In essence, this means that we build a statistical model which
can predict the data at hand very well, but performs bad at predicting
future data it has never seen before. This happens if our model assumes
that some variation in our data \textbf{stems from a true ``signal'' in
our data, when in fact we only model random noise}
{[}@iniesta2016machine{]}. As a result, our statistical model produces
\textbf{false positive results}: it sees relationships where there are
none.

\begin{figure}
\centering
\includegraphics{overfitting.png}
\caption{Illustration of an overfitted model vs.~model with a good fit}
\end{figure}

Regression methods, which usually utilize minimization or maximization
procedures such as Ordinary Least Squares or Maximum Likelihood
estimatation, can be prone to overfitting {[}@gigerenzer2004mindless;
@gigerenzer2008rationality{]}. Unfortunately, the risk of building a
\textbf{non-robust model, which produces false-positive results}, is
\textbf{even higher} once we go from conventional regression to
\textbf{meta-regression} {[}@higgins2004controlling{]}. There are
several reasons for this:

\begin{enumerate}
\def\arabic{enumi}.{\arabic{enumi}.}
\tightlist
\item
  In Meta-Regression, our \textbf{sample of data is mostly small}, as we
  can only use the synthesized data of all analyzed studies \(k\).
\item
  As our meta-analysis aims to be a \textbf{comprehensive overview of
  all evidence}, we have no \textbf{additional data} on which we can
  ``test'' how well our regression model can predict high or low effect
  sizes.
\item
  In meta-regressions, we have to deal with the potential presence of
  effect size heterogeneity (\protect\hyperlink{heterogeneity}{see
  Chapter 6}). Imagine a case in which we have to studies with different
  effect sizes and non-overlapping confidence intervals. Every variable
  which has different values for the different studies might be a
  potential explanation for effect size difference we find, while it
  seems straightforward that \textbf{most of such explanations are
  spurious} {[}@higgins2004controlling{]}.
\item
  Meta-regression as such, and multiple meta-regression in particular,
  make it very easy to \textbf{``play around'' with predictors}. We can
  test numerous meta-regression models, include many more predictors or
  remove them in an attempt to explain the heterogeneity in our data.
  Such an approach is of course tempting, and often found in practice,
  because we, as meta-analysts, want to find a significant model which
  explains why effect sizes differ {[}@higgins2002statistical{]}.
  However, such behavior has been shown to massively \textbf{increase
  the risk of spurious findings} {[}@higgins2004controlling{]}, because
  we can change parts of our model indefinitely until we find a
  significant model, which is then very likely to be overfitted (i.e.,
  it mostly models statistical noise).
\end{enumerate}

\textbf{Some guidelines have been proposed to avoid an excessive false
positive rate when building meta-regression models:}

\begin{itemize}
\tightlist
\item
  Minimize the number of investigated predictors. In multiple
  meta-regression, this translates to the concept of \textbf{parsimony},
  or simplicity: when evaluating the fit of a meta-regression model, we
  prefer models which achieve a good fit with less predictors.
  Estimators such as the \emph{Akaike} and \emph{Bayesian information
  criterion} can help with such decisions. We will get to this topic
  again below.
\item
  Predictor selection should be based on \textbf{predefined scientific
  or theoretical questions} we want to answer in our meta-regression.
\item
  When the number of studies is low (which is very likely to be the
  case), and we want to compute the significance of a predictor, we
  should use the \protect\hyperlink{random}{Knapp-Hartung adjustment} to
  obtain more reliable estimates {[}@higgins2002statistical{]}.
\item
  We can use \textbf{permutation} to assess the robustness of our model
  in resampled data. We will describe the details of this method later.
\end{itemize}
\end{rmdachtung}

\begin{rmdachtung}
\textbf{Multicollinearity}

Multicollinearity means that one or more predictor in our regression
model can be (linearly) \textbf{predicted from another predictor} in our
model with relatively high accuracy {[}@mansfield1982detecting{]}. This
basically means that we have two or more predictors in our model which
are \textbf{highly correlated}. Most of the dangers of multicollinearity
are associated with the problem of \textbf{overfitting} which we
described above. High collinearity can cause our predictor coefficient
estimates \(b\) to behave erratically, and change considerably with
minor changes in our data. It also limits the size of the explained
variance by the model, in our case the \(R^2\) analog.

\textbf{Multicollinearity in meta-regression is common}
{[}@berlin1992advantages{]}. Although multiple regression can handle
lower degrees of collinearity, we should \textbf{check} and, if
necessary, \textbf{control for very highly correlated predictors}. There
is no consolidated yes-no-rule for the presence of multicollinearity. A
crude, but often effective way is to check for very high correlations
(i.e., \(r\geq0.8\)) before fitting the model. Multicollinearity can
then be reduced by either (1) removing one of the close-to-redundant
predictors, or (2) trying to combine the predictors into one single
predictor.
\end{rmdachtung}

\hypertarget{model-building-methods}{%
\subsection{Model building methods}\label{model-building-methods}}

When building a multiple regression model, there are \textbf{different approaches through which we can select and insert predictors} into our model. Here, we discuss the most important ones along with their strengths and weaknesses:

\begin{itemize}
\tightlist
\item
  \textbf{Forced entry.} In forced entry methods, all relevant predictors are forced into the regression model simultaneously. In most functions in R, this is the default setting. Although this is a generally recommended procedure \citep{field2012discovering}, keep in mind that all predictors to use in forced entry should still be based on a predefined, theory-led decision.
\item
  \textbf{Hierarchical.} Hierchical multiple regression means including predictors into our regression model stepwise, but based on a clearly defined scientific rationale. First, only predictors which have been associated with effect size differences in previous research are included in the order of their importance. After this step, novel predictors can be added to explore if these variables explain heterogeneity which has not yet been captured by the known predictors.
\item
  \textbf{Stepwise.} Stepwise methods mean that variables/predictors are added to the model one after another. At first glance, this sounds a lot like hierarchical regression, but there is a crucial difference: stepwise regression methods select predictors based on a \emph{statistical} criterion. In a procedure called \textbf{forward selection}, the variable explaining the largest amount of variability in the data is used as the first predictor. This processes is then repeated for the remaining variables, each time selecting the variable which explains most of the \emph{remaining} (unexplained) variability in the data as the next predictor, which is then added to the model. Furthermore, there is a procedure called \textbf{backward selection}, in which all variables are used as predictors in the model first, and then removed successively based on a predefined statistical criterion. There is an extensive literature discouraging the usage of stepwise methods \citep{whittingham2006we, chatfield1995model}, and if we recall the common \protect\hyperlink{pitfalls}{pitfalls} of multiple regression models we presented above, it becomes clear that these methods have high risk of producing spurious findings. Nevertheless, stepwise methods are still frequently used in practice, making it important to know that these procedures exist. If we use stepwise methods ourselves, however, it is advised to primarily do so in an exploratory fashion and keep the limitations of this procedure in mind.
\item
  \textbf{Multimodel inference.} Multi-model methods differ from stepwise methods as they do not try to successively build the ``best'' one model explaining most of the variance. Instead, in this procedure, \emph{all} possible combinations of a predifined selection of predictors are modeled, and evaluated using a criterion such as Akaike's Information Criterion, which rewards simpler models. This enables a full eximination of all possible models, and how they perform. A common finding using this procedure is that there are many different kinds of predictor combinations within a model which lead to a good fit. In multimodel inference, the estimated coefficients of predictors can then be synthesized across all possible models to infer how important certain predictors are overall.
\end{itemize}

\hypertarget{using-metafor-to-compute-multiple-meta-regressions}{%
\subsection{\texorpdfstring{Using \texttt{metafor} to compute Multiple Meta-Regressions}{Using metafor to compute Multiple Meta-Regressions}}\label{using-metafor-to-compute-multiple-meta-regressions}}

After all this input, it is now \textbf{time to fit our first multiple regression models}. To do this, we will not use the \texttt{meta} package we have used so far. Instead, we will use the \texttt{metafor} package \citep{viechtbauer2010conducting}. This package provides a vast array of advanced functionalities for meta-analysis, along with great \href{http://www.metafor-project.org/doku.php/}{documentation}. We will also use this package in a later chapter where we focus on \textbf{Multilevel Meta-Analysis}, so it is worth getting accustomed to the inner workings of this package. So, to begin, let's make sure we have \texttt{metafor} installed and loaded in our library.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(metafor)}
\end{Highlighting}
\end{Shaded}

For our multiple meta-regression examples, i will use my \texttt{mvreg.data} dataset. This is a ``toy'' dataset, which we simulated for illustrative purposes. \textbf{Let's have a look at the structure of my data:}

\begin{verbatim}
## 'data.frame':    36 obs. of  6 variables:
##  $ yi        : num  0.0944 0.0998 0.1693 0.1751 0.273 ...
##  $ sei       : num  0.196 0.192 0.119 0.116 0.165 ...
##  $ reputation: num  -11 0 -11 4 -10 -9 -8 -8 -8 0 ...
##  $ quality   : num  6 9 5 9 2 10 6 3 10 3 ...
##  $ pubyear   : num  -0.855 -0.753 -0.66 -0.563 -0.431 ...
##  $ continent : chr  "North America" "Europe" "North America" "Europe" ...
\end{verbatim}

We see that there are 6 variables in our dataset. The \texttt{yi} and \texttt{sei} columns store the \textbf{effect size} and \textbf{standard error} of a particular study. Thus, these columns correspond to the \texttt{TE} and \texttt{seTE} columns we used before. We have named these variables this way because this is the standard notation that \texttt{metafor} uses: \texttt{yi} corresponds to the effect size \(y_i\) we want to predict in our meta-regression, while \texttt{sei} is \(SE_i\), the standard error. To designate the variance of an effect size, \texttt{metafor} uses \texttt{vi}, or \(v_i\) in mathematical notation, which we do not need here because \texttt{yi} and \texttt{sei} contain all the information we need.

The other four variables we have in our dataset are potential predictors for our meta-regression. We want to check if \texttt{reputation}, the (mean-centered) impact score of the journal the study was published in, \texttt{quality}, the quality of the study rated from 0 to 10, \texttt{pubyear}, the (standardized) publication year, and \texttt{continent}, the continent in which the study was performed, are associated with different effect sizes.

For \texttt{continent}, note that we store information as a predictor with 2 labels: \texttt{Europe} and \texttt{North\ America}, meaning that this predictor is a \textbf{dummy variable}. Always remember that such dummy variables have to be converted from a \texttt{chr} to a factor vector before we can proceed.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mvreg.data}\OperatorTok{$}\NormalTok{continent =}\StringTok{ }\KeywordTok{factor}\NormalTok{(mvreg.data}\OperatorTok{$}\NormalTok{continent)}
\end{Highlighting}
\end{Shaded}

\hypertarget{checking-for-multicollinearity}{%
\subsubsection{Checking for multicollinearity}\label{checking-for-multicollinearity}}

As we mentioned before, we have to check for multicollinearity of our predictors to make sure that our meta-regression model coefficient estimates are robust. A quick way to check for high intercorrelation is to calculate a \textbf{intercorrelation matrix} for all continuous variables with the following code:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{cor}\NormalTok{(mvreg.data[,}\DecValTok{3}\OperatorTok{:}\DecValTok{5}\NormalTok{])}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##            reputation    quality    pubyear
## reputation  1.0000000  0.3015694  0.3346594
## quality     0.3015694  1.0000000 -0.1551123
## pubyear     0.3346594 -0.1551123  1.0000000
\end{verbatim}

The \texttt{PerformanceAnalytics} package provides the \texttt{chart.Correlation} function which we can use to visualize the intercorrelations. Make sure to install the \texttt{PerformanceAnalytics} package first, and then use this code:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(PerformanceAnalytics)}
\KeywordTok{chart.Correlation}\NormalTok{(mvreg.data[,}\DecValTok{3}\OperatorTok{:}\DecValTok{5}\NormalTok{])}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-142-1.pdf}

We see that our variables are indeed correlated, but probably not to a degree that would warrant excluding one of them.

\hypertarget{fitting-a-meta-regression-model-without-interaction-terms}{%
\subsubsection{Fitting a meta-regression model without interaction terms}\label{fitting-a-meta-regression-model-without-interaction-terms}}

Now, let's fit our first meta-regression model, for now without assuming any interactions between predictors. Previously, we wanted to explore if a high \textbf{reputation} of the journal a study was published predicts higher effect sizes. Let's also assume we already know very well from previous literature that the \textbf{quality} of a study is also associated with differences in effects. We can now perform a hierarchical regression where we first include our known predictor \texttt{quality}, and then check if \texttt{reputation} explains heterogeneity beyond that while taking \texttt{quality} into account.

To do this, we use the \texttt{rma} function in \texttt{metafor} to perform a \textbf{random-effects meta-regression}. This function has countless parameters, but we will only concentrate on the following (to see all parameters, type \texttt{?rma} into your console):

Parameter

Function

yi

The column of our dataset in which the effect sizes for each study are stored.

sei

The column of our dataset in which the standard error of the effect sizes for each study is stored.

data

The dataset with our meta-analysis data.

method

The estimator for tau-squared we want to use. For meta-regression models, it is advised to use the
maximum likelihood (`ML') or restricted maximum likelihood (`REML') estimator.

mods

This parameter defines our meta-regression model. First, we specify our model with `\textasciitilde{}' (a tilde). Then, we add the predictors we want to include, separating them with `+' (e.g., `variable1 + variable2'). Interactions between two variables are denoted by 'variable1*variable2'.

test

The test we want to apply for our regression coefficients. We can choose from `z' (default) and `knha' (Knapp-Hartung method).

First, let's perform a meta-regression using only \texttt{quality} as the predictor along with Knapp-Hartung adjustments. \textbf{We save the results as \texttt{model1}, and then inspect the output}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model1 <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{yi, }
              \DataTypeTok{sei=}\NormalTok{sei, }
              \DataTypeTok{data=}\NormalTok{mvreg.data, }
              \DataTypeTok{method =} \StringTok{"ML"}\NormalTok{, }
              \DataTypeTok{mods =} \OperatorTok{~}\StringTok{ }\NormalTok{quality, }
              \DataTypeTok{test=}\StringTok{"knha"}\NormalTok{)}
\NormalTok{model1}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Mixed-Effects Model (k = 36; tau^2 estimator: ML)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.0667 (SE = 0.0275)
## tau (square root of estimated tau^2 value):             0.2583
## I^2 (residual heterogeneity / unaccounted variability): 60.04%
## H^2 (unaccounted variability / sampling variability):   2.50
## R^2 (amount of heterogeneity accounted for):            7.37%
## 
## Test for Residual Heterogeneity: 
## QE(df = 34) = 88.6130, p-val < .0001
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 34) = 3.5330, p-val = 0.0688
## 
## Model Results:
## 
##          estimate      se    tval    pval    ci.lb   ci.ub   
## intrcpt    0.3429  0.1354  2.5318  0.0161   0.0677  0.6181  *
## quality    0.0356  0.0189  1.8796  0.0688  -0.0029  0.0740  .
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

From the output, we can see the results for our predictor \texttt{quality} under \texttt{Model\ Results} and the values for our regression intercept (\texttt{intrcpt}). We see that the \(p\) value for our predictor is non-significant \(p=0.0688\), but only marginally so. Under \texttt{Test\ of\ Moderators\ (coefficient(s)\ 2)}, we can see the overall test results for our regression model (\(F_{1,34}=3.68, p=0.0688\)). Because we only included one predictor, the \(p\)-value reported there is identical to the one we saw before. In total, our model explains \(R^2=2.23\%\) of the heterogeneity in our data, which we can see next to the \texttt{R\^{}2\ (amount\ of\ heterogeneity\ accounted\ for)} line in our output.

\textbf{Now, let's check if we can improve our model by including \texttt{reputation} as predictor. We add \texttt{+\ reputation} in our argument to \texttt{mods} and save the output to \texttt{model2}.}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model2 <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{yi, }
              \DataTypeTok{sei=}\NormalTok{sei, }
              \DataTypeTok{data=}\NormalTok{mvreg.data, }
              \DataTypeTok{method =} \StringTok{"ML"}\NormalTok{, }
              \DataTypeTok{mods =} \OperatorTok{~}\StringTok{ }\NormalTok{quality }\OperatorTok{+}\StringTok{ }\NormalTok{reputation, }
              \DataTypeTok{test=}\StringTok{"knha"}\NormalTok{)}
\NormalTok{model2}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Mixed-Effects Model (k = 36; tau^2 estimator: ML)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.0238 (SE = 0.0161)
## tau (square root of estimated tau^2 value):             0.1543
## I^2 (residual heterogeneity / unaccounted variability): 34.62%
## H^2 (unaccounted variability / sampling variability):   1.53
## R^2 (amount of heterogeneity accounted for):            66.95%
## 
## Test for Residual Heterogeneity: 
## QE(df = 33) = 58.3042, p-val = 0.0042
## 
## Test of Moderators (coefficient(s) 2:3): 
## F(df1 = 2, df2 = 33) = 12.2476, p-val = 0.0001
## 
## Model Results:
## 
##             estimate      se    tval    pval    ci.lb   ci.ub     
## intrcpt       0.5005  0.1090  4.5927  <.0001   0.2788  0.7222  ***
## quality       0.0110  0.0151  0.7312  0.4698  -0.0197  0.0417     
## reputation    0.0343  0.0075  4.5435  <.0001   0.0189  0.0496  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

We now see that a new line appears under \texttt{Model\ Results} displaying the results for our \texttt{reputation} predictor. We see that a coefficient of \(b=0.0343\) was calculated for \texttt{reputation}. We also see that this predictor is highly significant (\(p<0.0001\)) and positively associated with the effect sizes. We also see that the meta-regression model itself is significant (\(F_{2,33}=12.25, p<0.0001\)), explaining \(R^2=66.95\%\) of the heterogeneity. This means that journal reputation is \textbf{associated with higher effect sizes}, even when controlling for study quality.

\textbf{But is \texttt{model2} indeed better than our first model, \texttt{model1}? To assess this, we can use the \texttt{anova} function, feeding it with the two models we want to compare.}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{anova}\NormalTok{(model1, model2)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##         df     AIC     BIC    AICc   logLik     LRT   pval      QE  tau^2
## Full     4 19.4816 25.8157 20.7720  -5.7408                58.3042 0.0238
## Reduced  3 36.9808 41.7314 37.7308 -15.4904 19.4992 <.0001 88.6130 0.0667
##            R^2
## Full          
## Reduced 64.32%
\end{verbatim}

This function performs a model test and provides us with several statistics to assess if \texttt{model2} has a better fit than \texttt{model1}. Here we compare our \texttt{full} model, \texttt{model2}, which includes both \texttt{quality} and \texttt{reputation}, with our \texttt{Reduced} model, which only uses \texttt{quality} as a predictor.

The \texttt{anova} function performs a \textbf{Likelihood Ratio Test}, the results of which we can see in the \texttt{LRT} column. The test is highly significant (\(\chi^2_1=19.50, p<0.001\)), which means that that our full model indeed provides a better fit. Another important statistic is reported in the \texttt{AICc} column. This provides us with the \emph{Akaike's Information Criterion}, corrected for small samples. As we mentioned before, AICc penalizes complex models with more predictors to avoid overfitting. It is important to note that \textbf{lower values of AIC(c) mean that a model performs better}. Interestingly, in our output, we see that the full model (\(AICc=20.77\)) has a better AIC value than our reduced model (\(AICc=37.73\)), despite having more predictors. All of this suggests that our multiple regression \textbf{does indeed provide a good fit} to our data.

\hypertarget{modeling-interaction-terms}{%
\subsubsection{Modeling interaction terms}\label{modeling-interaction-terms}}

Let's say we want to model an \textbf{interaction hypothesis} with our additional predictors \texttt{pubyear} (publication year) and \texttt{continent}, where we assume that the relationship between publication year and effect size differs for Europe and North America. To model this in our \texttt{rma} function, we have to \textbf{connect our predictors} with \texttt{*} in the \texttt{mods} parameter. Because we do not want to compare the models directly using the \texttt{anova} function, we specify the \(\tau^2\) estimator to be \texttt{"REML"} (restricted maximum likelihood) this time:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{interaction.model <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{yi,}
                         \DataTypeTok{sei=}\NormalTok{sei, }
                         \DataTypeTok{data=}\NormalTok{mvreg.data, }
                         \DataTypeTok{method =} \StringTok{"REML"}\NormalTok{, }
                         \DataTypeTok{mods =} \OperatorTok{~}\StringTok{ }\NormalTok{pubyear}\OperatorTok{*}\NormalTok{continent, }
                         \DataTypeTok{test=}\StringTok{"knha"}\NormalTok{)}
\NormalTok{interaction.model}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Mixed-Effects Model (k = 36; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of residual heterogeneity):     0 (SE = 0.0098)
## tau (square root of estimated tau^2 value):             0
## I^2 (residual heterogeneity / unaccounted variability): 0.00%
## H^2 (unaccounted variability / sampling variability):   1.00
## R^2 (amount of heterogeneity accounted for):            100.00%
## 
## Test for Residual Heterogeneity: 
## QE(df = 32) = 24.8408, p-val = 0.8124
## 
## Test of Moderators (coefficient(s) 2:4): 
## F(df1 = 3, df2 = 32) = 28.7778, p-val < .0001
## 
## Model Results:
## 
##                                 estimate      se    tval    pval    ci.lb
## intrcpt                           0.3892  0.0421  9.2472  <.0001   0.3035
## pubyear                           0.1683  0.0834  2.0184  0.0520  -0.0015
## continentNorth America            0.3986  0.0658  6.0539  <.0001   0.2645
## pubyear:continentNorth America    0.6323  0.1271  4.9754  <.0001   0.3734
##                                  ci.ub     
## intrcpt                         0.4750  ***
## pubyear                         0.3380    .
## continentNorth America          0.5327  ***
## pubyear:continentNorth America  0.8911  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

The last line, \texttt{pubyear:continentNorth\ America}, is the coefficient for our interaction term. Note that \texttt{metafor} automatically \textbf{includes not only the interaction term}, but also both \texttt{pubyear} and \texttt{contintent} as \textbf{``normal'' lower-order predictors} (as one should do). Also note that, as \texttt{continent} is a factor, \texttt{rma} detected that this is a \textbf{dummy predictor}, and used our category \texttt{Europe} as the \(D=0\) dummy against which the \texttt{North\ America} category is compared. We see that our interaction term \texttt{pubyear:continentNorth\ America} has a positive coefficient (\(b=0.6323\)), and that it is highly significant (\(p<0.0001\)), meaning that assumed interaction effect might in fact exist: there is an increase in effect sizes in recent years, but it is stronger for studies conducted in North America. We also see that model we fitted explains \(R^2=100\%\) of our heterogeneity. This is because our data was simulated for illustrative purposes. In practice, you will hardly ever explain all of the heterogeneity in your data (in fact, one should rather be concerned if one finds such results in real-life data, as this might mean we have overfitted our model).

\hypertarget{testing-the-robustness-of-our-model-through-a-permutation-test}{%
\subsubsection{Testing the robustness of our model through a permutation test}\label{testing-the-robustness-of-our-model-through-a-permutation-test}}

\begin{rmdinfo}
\textbf{The idea behind permutation}

Permutation is a \textbf{mathematical operation} in which we take a
\textbf{set} containing numbers or objects, and iteratively draw
elements from this set to put them in a sequential order. When we
already have a ordered set of numbers, this equals a process in which we
\textbf{rearrange}, or \textbf{shuffle}, the order of our data. As an
example, imagine we have a set \(S\) containing \textbf{three numbers}:
\(S=\{1,2,3 \}\). One possible permutation of this set is \((2,1,3)\);
another is \((3,2,1)\). We see that the permuted results both contain
\textbf{all three numbers from before}, but in a different order.

Permutation can also be used to perform \textbf{permutation tests},
which is a specific form of \textbf{resampling methods}. Broadly
speaking, resampling methods are used to validate the
\textbf{robustness} of a statistical model by providing it with
(slightly) different data sampled from the same data source or
generative process {[}@good2013permutation{]}. This is a way to better
\textbf{assess if the coeffients in our model indeed capture a true
pattern underlying our data}, or if we overfitted our model, thereby
falsely assuming patterns in our data when they are in fact statistical
noise.

Permutation tests do not require us to have a spare ``test'' dataset on
which we can evaluate how our meta-regression actually performs in
predicting effect sizes. For this reason, among others, permutation
tests have been \textbf{recommended} to assess the robustness of our
meta-regression models {[}@higgins2004controlling{]}.

We will not go too much into the \textbf{details of how a permutation
test is performed for meta-regression} models, but the most important
part is that we \textbf{re-calculate} the \textbf{p-values} of our
overall meta-regression model and its coefficients based on the test
statitics obtained across all possible, or many randomly selected,
permutations of our original dataset (or, to be more precise, our
meta-regression model matrix \(X_{ij}\)). The crucial indicator here is
\textbf{how often the test statistic we obtain from in our permuted data
is equal or greater than our original test statistic}. For example, if
our test statistic is greater or equal to the original test statistic in
50 of 1000 permuted datasets, we get a p-value of \(p=0.05\)
{[}@viechtbauer2015comparison{]}.
\end{rmdinfo}

To \textbf{perform a permutation test} on our meta-regression model, we can use \texttt{metafors} in-built \texttt{permutest} function. As an example, i will now recalculate the results of my \texttt{model2} model we fitted above. We only have to supply the \texttt{permutest} function with the \texttt{rma} object, but \textbf{be aware that the permutation test can be computationally intensive, especially for large datasets, so the function might need some time to run}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{permutest}\NormalTok{(model2)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Test of Moderators (coefficient(s) 2:3): 
## F(df1 = 2, df2 = 33) = 12.2476, p-val* = 0.0010
## 
## Model Results:
## 
##             estimate      se    tval   pval*    ci.lb   ci.ub     
## intrcpt       0.5005  0.1090  4.5927  0.2040   0.2788  0.7222     
## quality       0.0110  0.0151  0.7312  0.4580  -0.0197  0.0417     
## reputation    0.0343  0.0075  4.5435  0.0010   0.0189  0.0496  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

We again see our \textbf{familiar output} including the \textbf{results for all predictors}. Looking at the \texttt{pval*} column, we see that our p-value for the \texttt{reputation} predictor has decreased from \(p<0.0001\) to \(p^*=0.001\). The p-value of our overall model has decreased from \(p=0.0001\) to \(p^*=0.001\). However, as both of these p-values are still highly significant, this indicates that our model might indeed capture a real pattern underlying our data. It has been \textbf{recommended} to always use this \textbf{permutation test on our meta-regression model before we report a meta-regression model} to be significant in our research \citep{higgins2004controlling}.

\begin{rmdachtung}
\textbf{Permutation tests in small datasets}

Please note that when the \textbf{number of studies} \(k\) included in
our model is \textbf{small}, conventionally used tresholds for
statistical significance (i.e., p \textless{} 0.05) \textbf{cannot be
reached}. For meta-regression models, a permutation test using
\texttt{permutest} will only be able to reach statistical significance
if \(k \geq 5\). More details on the \texttt{permutest} function can be
found
\href{https://www.rdocumentation.org/packages/metafor/versions/1.9-9/topics/permutest}{here}.
\end{rmdachtung}

\hypertarget{multimodel-inference}{%
\subsubsection{Multimodel Inference}\label{multimodel-inference}}

Previously, we already described that one can also try to \textbf{model all possible predictor combinations} in a procedure called \textbf{multimodel inference} to examine which possible predictor combinations provide the best fit, and which predictors are the most important ones overall.

To perform multimodel inference, we have prepared a function for you, \texttt{mreg.multimodel.inference}. The full code for this function can be found \href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/mreg.multimodel.inference.R}{here}. Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code from the website \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}. For the \texttt{mreg.multimodel.inference} function, the following parameters need to be specified:

Parameter

Function

TE

The name of the variable containing the calculated effect sizes for each study.

seTE

The name of the variable containing the calculated standard error of each effect size.

data

Our dataset containing all effect size and predictor data.

predictors

The predictors in our dataset we want to consider. We have to supply them to the function as predictors=c(`variable1', `variable2', \ldots{}).

method

The tau-squared estimator we want to use. We can choose from `Maximum-Likelihood ('ML') and Restricted Maximum Likelihood (`REML'; default).

test

The test we want to use to evaluate the significance of a predictor.'knha' (Knapp-Hartung) is used as the default, but we can also set this to `z'.

level

This specifies if we want to also model all possible interaction terms (level=2) or not (level=1). Default is level=1.

The \texttt{mreg.multimodel.inference} requires the \texttt{glmulti} package to be installed on you computer, so let's make sure first to install this package and then load it into our library.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(glmulti)}
\end{Highlighting}
\end{Shaded}

\begin{rmdachtung}
\textbf{Problems with rJava}

\texttt{glmulti} requires the installation of \texttt{rJava} as a
dependency. Usually, this is done automatically when installing
\texttt{glmulti}. However, \texttt{rJava} installation often
\textbf{does not work properly}, and \texttt{rJava} may cease to
function after updates are installed. If this happens on your computer,
this
\href{https://stackoverflow.com/questions/6492361/problem-loading-rjava}{thread},
or a quick Google search may help you to fix this problem.
\end{rmdachtung}

\textbf{Now, let's perform multimodelling for all possible predictor combinations using all predictors in the \texttt{mvreg.data} dataset, excluding all interaction terms}. Be aware that this can take some time, especially if the number of predictors is large.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{mreg.multimodel.inference}\NormalTok{(}\DataTypeTok{TE =} \StringTok{"yi"}\NormalTok{, }
                          \DataTypeTok{seTE =} \StringTok{"sei"}\NormalTok{,}
                          \DataTypeTok{data =}\NormalTok{ mvreg.data,}
                          \DataTypeTok{predictors =} \KeywordTok{c}\NormalTok{(}\StringTok{"pubyear"}\NormalTok{, }\StringTok{"quality"}\NormalTok{, }\StringTok{"reputation"}\NormalTok{, }\StringTok{"continent"}\NormalTok{), }
                          \DataTypeTok{level=}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Initialization...
## TASK: Exhaustive screening of candidate set.
## Fitting...
## Completed.
## 
## 
## Multimodel Inference: Final Results
## --------------------------
## 
##  - Number of fitted models: 16
##  - Full formula: TE ~ pubyear+quality+reputation+continent
##  - Coefficient significance test: knha
##  - Modeled interaction level: 1
##  - Evaluation criterion: aicc 
## 
## 
## Best 5 Models
## --------------------------
## 
## 
##                                       model     aicc   weights
## 1 TE ~ 1 + continent + pubyear + reputation 31.29497 0.1641659
## 2             TE ~ 1 + pubyear + reputation 31.39575 0.1560991
## 3    TE ~ 1 + continent + pubyear + quality 31.82622 0.1258708
## 4              TE ~ 1 + continent + pubyear 32.03358 0.1134738
## 5   TE ~ 1 + pubyear + quality + reputation 32.19910 0.1044612
## 
## 
## Multimodel Inference Coefficients
## --------------------------
## 
## 
##              Estimate Uncond. variance Nb models Importance
## quality    0.01538909     0.0003906956         8  0.4243529
## continent1 0.15426094     0.0244447268         8  0.5396121
## reputation 0.01792199     0.0002024595         8  0.6497092
## pubyear    0.35634006     0.0255989257         8  0.8431014
## intrcpt    0.42573480     0.0202096069        16  1.0000000
##            +/- (alpha=0.05)
## quality          0.03874069
## continent1       0.30643664
## reputation       0.02788799
## pubyear          0.31358766
## intrcpt          0.27862945
\end{verbatim}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-156-1.pdf}

\textbf{Let's go through the output one after another:}

\begin{itemize}
\tightlist
\item
  \texttt{Multimodel\ Inference:\ Final\ Results}. This part of the output provides us with details about the fitted models. We see that the total number of \(2^4=16\) possible models have been fitted. We also see that the function used the corrected AIC (\texttt{aicc}) to compare the models.
\item
  \texttt{Best\ 5\ Models}. Displayed here are the \textbf{five models with the lowest AICc}, sorted from low to high. We see that \texttt{TE\ \textasciitilde{}\ 1\ +\ pubyear\ +\ reputation\ +\ continent} shows the best fit (\(AICc=31.30\)). But we also see that the other models including other predictors which are presented here come very close to this value. Thus, it is hard to say which model really is the ``best'' model. Nevertheless, we see that all of the best five models contain the predictor \texttt{pubyear}, suggesting that this variable might be particularly important.
\item
  \texttt{Multimodel\ Inference\ Coefficients}. Here, we can see the \textbf{coefficients of all predictors} we used for our models \textbf{aggregated} over all models in which they appear. We see that the coefficient \texttt{Estimate} is largest for \texttt{pubyear} (\(b=0.356\)), which corroborates our finding from before. Approximate confidence intervals can be obtained by subtracting and adding the value stored in \texttt{+/-\ (alpha=0.05)} from/to \texttt{Estimate}.
\item
  \textbf{Model-averaged importance of terms plot}. In the plot, we see the averaged importance of each predictor across all models displayed as a bar chart. We again see that \texttt{pubyear} is the most important predictor, followed by \texttt{reputation}, \texttt{continent}, and \texttt{quality}.
\end{itemize}

\begin{rmdinfo}
This example should make clear that Multimodel Inference can be a useful
way to obtain a \textbf{comprehensive look on which predictors are more
or less important} for predicting differences in effect sizes. Despite
avoiding some of the problems of stepwise regression methods, it should
be noted that this method should still be rather seen as
\textbf{exploratory}, and may be used when we have no prior knowledge on
how our predictors are related to effect sizes in the research field we
meta-analyze. If you decide to build a meta-regression model using the
information from multimodel inference, it is \textbf{essential to report
that you have used this procedure before building the model}, because
your model will then not be based on an apriori hypothesis, but on
statistical properties.
\end{rmdinfo}

\begin{rmdinfo}
The output of the \texttt{mreg.multimodel.inference} function displayed
after running the function is \textbf{only a small part of all results
returned by the function}. If we want to access all results of the
function, we can save the results to an object (e.g.
\texttt{results\ \textless{}-\ mreg.multimodel.inference(...)}), and
then open them using the `dollar' operator. The following results are
returned, among other things:

\begin{itemize}
\tightlist
\item
  \texttt{all.models}: the AICc statistics for all fitted models,
  ordered from best to worst.
\item
  \texttt{top5.models}: the top 5 models by AICc we already saw in the
  output of the function before.
\item
  \texttt{multimodel.coef}: the averaged multimodel coefficient
  estimates we already saw before.
\item
  \texttt{model.details}: a list containing the detailed \texttt{rma}
  output for all fitted models. We can access a specific model by its
  number the following way: \texttt{results\$model.details{[}{[}1{]}{]}}
  (for the first model).
\end{itemize}
\end{rmdinfo}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{publication-bias}{%
\chapter{Publication Bias}\label{publication-bias}}

\includegraphics{publicationbias.jpg}

In the last chapters, we have showed you how to \textbf{pool effects} in meta-analysis, choose the \textbf{right pooling model}, assess the \textbf{heterogeneity} of your effect estimate, and determine sources of heterogeneity through \textbf{outlier, influence, and subgroup analyses}.

Nevertheless, even the most thoroughly conducted meta-analysis can only work with the \textbf{study material at hand}. An issue commonly discussed in research, however, is the \textbf{file-drawer} or \textbf{publication bias} problem, which states that a study with high effect sizes \textbf{is more likely to be published} than a study with a low effect size \citep{rothstein2006publication}.

Such \textbf{missing studies} with low effect sizes, it is assumed, thus never get published and therefore cannot be integrated into our meta-analysis. This leads to \textbf{publication bias}, as the pooled effect we estimate in our meta-analysis might be higher than the \textbf{true effect size} because we did not consider the missing studies with lower effects due to the simple fact that they were never published.

Although this practice is gradually changing \citep{nelson2018psychology}, weather a study is published or not heavily depends on the \textbf{statistical significance} (\(p<0.05\)) of its results \citep{dickersin2005publication}. For any sample size, a result is more likely to become \textbf{statistically significant} if its \textbf{effect size is high}. This is particularly true for \textbf{small studies}, for which very large effect sizes are needed to attain a statisitcally significant result.

In the \protect\hyperlink{smallstudyeffects}{following chapter}, we will describe the \textbf{idea behind statistical models for publication bias} in further depth. We termed these concepts and methods \textbf{small-study effect methods}, as it is small studies that they mostly focus on. These methods assume that publication bias is primarily driven by \textbf{effect size} and because researchers \textbf{immediately put every study in the file drawer once the results are not significant}.

Recently, it has been argued that these \textbf{assumptions may not be true}, and that publication bias is mostly caused by \textbf{significance levels} and \textbf{p-hacking} \citep{simonsohn2014p}. An alternative method called \textbf{P-Curve} has therefore been suggested to examine \textbf{publication bias}. We will present this method in the \protect\hyperlink{pcurve}{last chapter} of this section.

\begin{rmdinfo}
\textbf{Which method should i use for my meta-analysis?}

While recent research suggests that the conventional \textbf{small-study
effect methods} may have substantial \textbf{limitations}, and that
\textbf{p-Curve} may be able to estimate the true effect with less bias
{[}@simonsohn2014p;@simonsohn2015better;@simonsohn2014pb{]}, please note
that both methods are based on different \textbf{theoretical
assumptions} about the origin of publication bias. As we cannot
ultimately decide which assumption is the \textbf{``true''} one in
specific research fields, and, in practice \textbf{the true effect is
unkown when doing meta-analysis}, we argue that you may use \textbf{both
methods} and compare results as \textbf{sensitivity analyses}
{[}@harrer2019internet{]}.

\textbf{P-curve} was developed with \textbf{full-blown experimental
psychological research in mind}, in which researchers often have
\textbf{high degrees of ``researcher freedom''} {[}@simmons2011false{]}
in deleting outliers and performing statistical test on their data.

We argue that this looks slightly different for \textbf{clinical
psychology} and the medical field, where researchers conduct
\textbf{randomized controlled trials} whith a clear \textbf{primary
outcome}: the difference between the control and the intervention group
after the treatment. While it is also true for \textbf{medicine and
clinical psychology that statistical significance plays an important
role}, the \textbf{effect size} of an intervention is often of greater
interest, as \textbf{treatments are often compared in terms of their
treatment effects} in this field. Furthermore, best practice for
randomized controlled trials is to perform \textbf{intention-to-treat}
analyses, in which all collected data in a trial has to be considered,
giving researchers less space to ``play around'' with their data and
perform p-hacking. While we certainly do not want to insinuate that
\textbf{outcome research in clinical psychology} is free from p-hacking
and bad data analysis practices, this should be seen as a
\textbf{caveat} that the assumptions of the small-study effects methods
may be more adequate for clinical psychology than other fields within
psychology, especially when **the risk of bias for each study is also
taken into account*. Facing this uncertainty, we think that conducting
both analyses and reporting them in our research paper may be the most
adequate approach until meta-scientific research gives us more certainty
about which \textbf{assumption actually best reflects the field of
clinical psychology}.
\end{rmdinfo}

\hypertarget{smallstudyeffects}{%
\section{Small-study effect methods}\label{smallstudyeffects}}

The \textbf{small-study effect methods} we present here have been conventional for many years. Thus various methods to assess and control for publication bias have been developed, but we will only focus on the most important ones here.

\begin{rmdinfo}
\textbf{The model behind small-study effects methods}

According to Borenstein et al. {[}@borenstein2011{]}. The model behind
the most common small-study effects methods has these core
\textbf{assumptions}:

\begin{enumerate}
\defenumi{enumi}.{enumi{enumi}.}
\tightlist
\item
  Because they involve large commitment of ressources and time,
  \textbf{large studies are likely to get published}, weather the
  results are significant or not
\item
  Moderately sized studies are at \textbf{greater risk of missing}, but
  with a moderate sample size even moderately sized effects are likely
  to become significant, which means that only some studies will be
  missing
\item
  Small studies are \textbf{at greatest risk} for being non-significant,
  and thus being missing. Only small studies with a very large effect
  size become significant, and will be found in the published
  literature.
\end{enumerate}

In accordance with these assumptions, the methods we present here
particularly focus \textbf{on small studies with small effect sizes, and
wheather they are missing}.
\end{rmdinfo}

\hypertarget{funnel-plots}{%
\subsection{Funnel plots}\label{funnel-plots}}

The best way to visualize weather \textbf{small studies with small effect sizes are missing} is through \textbf{funnel plots}.

We can generate a funnel plot for our \texttt{m.hksj} meta-analysis output using the \texttt{funnel()} function in \texttt{meta}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{funnel}\NormalTok{(m.hksj,}\DataTypeTok{xlab =} \StringTok{"Hedges' g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-162-1.pdf}

The \textbf{funnel plot} basically consists of a \textbf{funnel} and two \textbf{axes}: the y-axis showing the \textbf{Standard Error} \(SE\) of each study, with larger studies (which thus have a smaller \(SE\)) plotted \textbf{on top of the y-axis}; and the x-axis showing the \textbf{effect size} of each study.

Given our assumptions, and in the case when there is \textbf{no publication bias}, all studies would lie \textbf{symmetrically around our pooled effect size (the striped line)} within the form of the funnel. When \textbf{publication bias is present}, we would assume that the funnel would look asymmetrical, because only the small studies with a large effect size very published, \textbf{while small studies without a significant, large effect would be missing}.

We see from the plot that in the case of our meta-anlysis \texttt{m.hksj}, the latter is probably true. We see that the plot is highly asymmetrical, with exactly the small studies with low effect size missing in the bottom-left corner of our plot.

We can also display the name of each study using the \texttt{studlab} parameter.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{funnel}\NormalTok{(m.hksj,}\DataTypeTok{xlab =} \StringTok{"g"}\NormalTok{,}\DataTypeTok{studlab =} \OtherTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-163-1.pdf}

Here, we see that asymmetry is primarily driven by \textbf{three studies with high effects, but a small study sample} in the bottom right corner. Interestingly, two of these studies are the ones we also detected in our \protect\hyperlink{outliers}{outlier} and \protect\hyperlink{influenceanalyses}{influence} analyses.

An even better way to inspect the funnel plot is through \textbf{contour-enhanced funnel plots}, which help to distinguish publication bias from other forms of asymmetry \citep{peters2008contour}. Contour-enhanced funnels include colors signifying the significance level into which the effects size of each study falls. We can plot such funnels using this code:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{funnel}\NormalTok{(m.hksj, }\DataTypeTok{xlab=}\StringTok{"Hedges' g"}\NormalTok{, }
       \DataTypeTok{contour =} \KeywordTok{c}\NormalTok{(.}\DecValTok{95}\NormalTok{,.}\DecValTok{975}\NormalTok{,.}\DecValTok{99}\NormalTok{),}
       \DataTypeTok{col.contour=}\KeywordTok{c}\NormalTok{(}\StringTok{"darkblue"}\NormalTok{,}\StringTok{"blue"}\NormalTok{,}\StringTok{"lightblue"}\NormalTok{))}\OperatorTok{+}
\KeywordTok{legend}\NormalTok{(}\FloatTok{1.4}\NormalTok{, }\DecValTok{0}\NormalTok{, }\KeywordTok{c}\NormalTok{(}\StringTok{"p < 0.05"}\NormalTok{, }\StringTok{"p<0.025"}\NormalTok{, }\StringTok{"< 0.01"}\NormalTok{),}\DataTypeTok{bty =} \StringTok{"n"}\NormalTok{,}
       \DataTypeTok{fill=}\KeywordTok{c}\NormalTok{(}\StringTok{"darkblue"}\NormalTok{,}\StringTok{"blue"}\NormalTok{,}\StringTok{"lightblue"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-165-1.pdf}

We can see in the plot that while \textbf{some studies have statistically significant effect sizes} (blue background), other do not (white background). We also see a trend that while the moderately sized studies are partly significant and non-significant, with slightly more significant studies, the asymmetry is much larger for small studies. This gives us a hint that publication bias might indeed be present in our analysis.

\hypertarget{testing-for-funnel-plot-asymmetry-using-eggers-test}{%
\subsection{Testing for funnel plot asymmetry using Egger's test}\label{testing-for-funnel-plot-asymmetry-using-eggers-test}}

\textbf{Egger's test of the intercept} \citep{egger1997bias} quantifies the funnel plot asymmetry and performs a statistical test.

We have prepared a function called \texttt{eggers.test} for you, which can be found below. The function is a wrapper for the \texttt{metabias} function in \texttt{meta}.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{eggers.test<-}\ControlFlowTok{function}\NormalTok{(data)\{}

\NormalTok{  data<-data}
\NormalTok{  eggers<-}\KeywordTok{metabias}\NormalTok{(data)}
\NormalTok{  intercept<-}\KeywordTok{as.numeric}\NormalTok{(eggers}\OperatorTok{$}\NormalTok{estimate[}\DecValTok{1}\NormalTok{])}
\NormalTok{  intercept<-}\KeywordTok{round}\NormalTok{(intercept,}\DataTypeTok{digits=}\DecValTok{3}\NormalTok{)}
\NormalTok{  se.intercept<-eggers}\OperatorTok{$}\NormalTok{estimate[}\DecValTok{2}\NormalTok{]}
\NormalTok{  lower.intercept<-}\KeywordTok{as.numeric}\NormalTok{(intercept}\FloatTok{-1.96}\OperatorTok{*}\NormalTok{se.intercept)}
\NormalTok{  lower.intercept<-}\KeywordTok{round}\NormalTok{(lower.intercept,}\DataTypeTok{digits =} \DecValTok{2}\NormalTok{)}
\NormalTok{  higher.intercept<-}\KeywordTok{as.numeric}\NormalTok{(intercept}\FloatTok{+1.96}\OperatorTok{*}\NormalTok{se.intercept)}
\NormalTok{  higher.intercept<-}\KeywordTok{round}\NormalTok{(higher.intercept,}\DataTypeTok{digits =} \DecValTok{2}\NormalTok{)}
\NormalTok{  ci.intercept<-}\KeywordTok{paste}\NormalTok{(lower.intercept,}\StringTok{"-"}\NormalTok{,higher.intercept)}
\NormalTok{  ci.intercept<-}\KeywordTok{gsub}\NormalTok{(}\StringTok{" "}\NormalTok{, }\StringTok{""}\NormalTok{, ci.intercept, }\DataTypeTok{fixed =} \OtherTok{TRUE}\NormalTok{)}
\NormalTok{  intercept.pval<-}\KeywordTok{as.numeric}\NormalTok{(eggers}\OperatorTok{$}\NormalTok{p.value)}
\NormalTok{  intercept.pval<-}\KeywordTok{round}\NormalTok{(intercept.pval,}\DataTypeTok{digits=}\DecValTok{5}\NormalTok{)}
\NormalTok{  eggers.output<-}\KeywordTok{data.frame}\NormalTok{(intercept,ci.intercept, intercept.pval)}
  \KeywordTok{names}\NormalTok{(eggers.output)<-}\KeywordTok{c}\NormalTok{(}\StringTok{"intercept"}\NormalTok{,}\StringTok{"95%CI"}\NormalTok{,}\StringTok{"p-value"}\NormalTok{)}
\NormalTok{  title<-}\StringTok{"Results of Egger's test of the intercept"}
  
\KeywordTok{print}\NormalTok{(title)}
\KeywordTok{print}\NormalTok{(eggers.output)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Now we can use the \texttt{eggers.test} function. We only have to specify our meta-analysis output \texttt{m.hksj} as the \texttt{data} the function should use.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{eggers.test}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Results of Egger's test of the intercept"
##   intercept     95%CI p-value
## 1     4.111 2.39-5.83 0.00025
\end{verbatim}

The function returns the \texttt{intercept} along with its confidence interval. We can see that the \texttt{p-value} of Egger's test is \textbf{significant} (\(p<0.05\)), which means that there is substanital asymmetry in the Funnel plot. This asymmetry could have been caused by publication bias.

\hypertarget{dant}{%
\subsection{Duval \& Tweedie's trim-and-fill procedure}\label{dant}}

\textbf{Duval \& Tweedie's trim-and-fill procedure} \citep{duval2000trim} is also based the funnel plot and its symmetry/asymmetry. When \textbf{Egger's test is significant}, we can use this method to estimate what \textbf{the actaul effect size would be had the ``missing'' small studies been published}. The procedure \textbf{imputes} missing studies into the funnel plot until symmetry is reached again.

\begin{rmdinfo}
\textbf{The trim-and-fill procedure includes the following five steps}
{[}@schwarzer2015meta{]}:

\begin{enumerate}
\defenumi{enumi}.{enumi{enumi}.}
\tightlist
\item
  Estimating the number of studies in the outlying (right) part of the
  funnel plot
\item
  Removing (trimming) these effect sizes and pooling the results with
  the remaining effect sizes
\item
  This pooled effect is then taken as the center of all effect sizes
\item
  For each trimmed/removed study, a additional study is imputed,
  mirroring the effect of the study on the left side of the funnel plot
\item
  Pooling the results with the imputed studies and the trimmed studies
  included
\end{enumerate}
\end{rmdinfo}

The \textbf{trim-and-fill-procedure} can be performed using the \texttt{trimfill} function in \texttt{meta}, and specifying our meta-analysis output.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{trimfill}\NormalTok{(m.hksj)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                             SMD             95%-CI %W(random)
## Call et al.              0.7091 [ 0.1979;  1.2203]        3.8
## Cavanagh et al.          0.3549 [-0.0300;  0.7397]        4.1
## DanitzOrsillo            1.7912 [ 1.1139;  2.4685]        3.3
## de Vibe et al.           0.1825 [-0.0484;  0.4133]        4.4
## Frazier et al.           0.4219 [ 0.1380;  0.7057]        4.3
## Frogeli et al.           0.6300 [ 0.2458;  1.0142]        4.1
## Gallego et al.           0.7249 [ 0.2846;  1.1652]        4.0
## Hazlett-Stevens & Oren   0.5287 [ 0.1162;  0.9412]        4.0
## Hintz et al.             0.2840 [-0.0453;  0.6133]        4.2
## Kang et al.              1.2751 [ 0.6142;  1.9360]        3.4
## Kuhlmann et al.          0.1036 [-0.2781;  0.4853]        4.1
## Lever Taylor et al.      0.3884 [-0.0639;  0.8407]        3.9
## Phang et al.             0.5407 [ 0.0619;  1.0196]        3.9
## Rasanen et al.           0.4262 [-0.0794;  0.9317]        3.8
## Ratanasiripong           0.5154 [-0.1731;  1.2039]        3.3
## Shapiro et al.           1.4797 [ 0.8618;  2.0977]        3.5
## SongLindquist            0.6126 [ 0.1683;  1.0569]        4.0
## Warnecke et al.          0.6000 [ 0.1120;  1.0880]        3.9
## Filled: Warnecke et al.  0.0520 [-0.4360;  0.5401]        3.9
## Filled: SongLindquist    0.0395 [-0.4048;  0.4837]        4.0
## Filled: Frogeli et al.   0.0220 [-0.3621;  0.4062]        4.1
## Filled: Call et al.     -0.0571 [-0.5683;  0.4541]        3.8
## Filled: Gallego et al.  -0.0729 [-0.5132;  0.3675]        4.0
## Filled: Kang et al.     -0.6230 [-1.2839;  0.0379]        3.4
## Filled: Shapiro et al.  -0.8277 [-1.4456; -0.2098]        3.5
## Filled: DanitzOrsillo   -1.1391 [-1.8164; -0.4618]        3.3
## 
## Number of studies combined: k = 26 (with 8 added studies)
## 
##                         SMD            95%-CI    t p-value
## Random effects model 0.3431 [ 0.0994; 0.5868] 2.90  0.0077
## Prediction interval         [-0.8463; 1.5326]             
## 
## Quantifying heterogeneity:
## tau^2 = 0.3181; H = 2.05 [1.70; 2.47]; I^2 = 76.2% [65.4%; 83.7%]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  105.15   25 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
## - Trim-and-fill method to adjust for funnel plot asymmetry
\end{verbatim}

We see that the procedure identified and trimmed \textbf{eight studies} \texttt{(with\ 8\ added\ studies)}). The overall effect estimated by the procedure is \(g = 0.34\).

Let's compare this to our initial results.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{TE.random}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.593535
\end{verbatim}

The initial pooled effect size was \(g = 0.59\), which is substantially larger than the bias-corrected effect. In our case, if we assume that \textbf{publication bias} was a problem in the analyses, the \textbf{trim-and-fill procedure} lets us assume that our initial results were \textbf{overestimated} due to publication bias, and the ``true'' effect when controlling for selective publication might be \(g = 0.34\) rather than \(g = 0.59\).

If we store the results of the \texttt{trimfill} function in an \textbf{object}, we can also create \textbf{funnel plots including the imputed studies}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{m.hksj.trimfill<-}\KeywordTok{trimfill}\NormalTok{(m.hksj)}
\KeywordTok{funnel}\NormalTok{(m.hksj.trimfill,}\DataTypeTok{xlab =} \StringTok{"Hedges' g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-171-1.pdf}

\hypertarget{pcurve}{%
\section{P-Curve}\label{pcurve}}

In the last chapter, we showed you how you can apply \textbf{Egger's test of the intercept}, \textbf{Duval \& Tweedie's trim and fill procedure}, and inspect \textbf{Funnel plots} in R.

As we have mentioned before, recent research has shown \textbf{that the assumptions of the small-effect study methods may be inaccurate in many cases}. The \textbf{Duval \& Tweedie trim-and-fill procedure} in particular has been shown to be prone to providing \textbf{inaccurate effect size estimates} \citep{simonsohn2014pb}.

\textbf{P-curve Analysis} has been proposed as an alternative way to assess publication bias and estimate the true effect behind our collected data.

P-Curve assumes that publication bias is not primarily generated because researchers \textbf{do not publish non-significant results}, but because the \textbf{``play'' around with their data (e.g., selectively removing outliers, choosing different outcomes, controlling for different variables) until a non-significant finding becomes significant}. This (bad) practice is called \textbf{p-hacking}, and has been shown to be extremely frequent among researchers \citep{head2015extent}.

\begin{rmdinfo}
\textbf{The idea behind P-Curve}
\end{rmdinfo}

\hypertarget{preparing-rstudio-and-our-data}{%
\subsection{Preparing RStudio and our data}\label{preparing-rstudio-and-our-data}}

To use \textbf{P-curve}, we need our data in the same format as the one we used for the \texttt{metacont} function in \protect\hyperlink{excel_preparation}{Chapter 3.1.1}. We need to specify \texttt{Me}, \texttt{Se}, \texttt{Ne}, \texttt{Mc}, \texttt{Sc}, and \texttt{Nc}. My \texttt{metacont} data already has this format.

Author

Ne

Me

Se

Nc

Mc

Sc

Cavanagh

50

4.5

2.7

50

5.6

2.6

Day

64

18.3

6.4

65

20.2

7.6

Frazier

90

12.5

3.2

95

15.5

4.4

Gaffney

30

2.34

0.87

30

3.13

1.234

Greer

77

15.212

5.35

69

20.13

7.43

Harrer

60

3.153

1.256

60

3.4213

1.878

To use p-curve, we also have to install and load \textbf{four packages}. The \texttt{esc} package \citep{esc}, the \texttt{compute.es} package \citep{del2013compute}, the \texttt{stringr} package, and the \texttt{poibin} package \citep{hong2011poibin}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(compute.es)}
\KeywordTok{library}\NormalTok{(esc)}
\KeywordTok{library}\NormalTok{(stringr)}
\KeywordTok{library}\NormalTok{(poibin)}
\end{Highlighting}
\end{Shaded}

To \textbf{prepare our data} which we stored in the format described above, we need to use the \texttt{pcurve\_dataprep} function, which we prepared for you.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{pcurve_dataprep<-}\ControlFlowTok{function}\NormalTok{(data)\{}
\NormalTok{data<-data}
\NormalTok{Me<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Me)}
\NormalTok{Se<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Se)}
\NormalTok{Ne<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Ne)}
\NormalTok{Mc<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Mc)}
\NormalTok{Sc<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Sc)}
\NormalTok{Nc<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Nc)}

\NormalTok{esc<-}\KeywordTok{esc_mean_sd}\NormalTok{(}\DataTypeTok{grp1m=}\NormalTok{Me, }
                 \DataTypeTok{grp1sd=}\NormalTok{Se, }
                 \DataTypeTok{grp1n=}\NormalTok{Ne, }
                 \DataTypeTok{grp2m=}\NormalTok{Mc, }
                 \DataTypeTok{grp2sd=}\NormalTok{Sc, }
                 \DataTypeTok{grp2n=}\NormalTok{Nc, }
                 \DataTypeTok{es.type =} \StringTok{"d"}\NormalTok{)}

\NormalTok{output<-}\KeywordTok{des}\NormalTok{(}\DataTypeTok{d=}\NormalTok{esc}\OperatorTok{$}\NormalTok{es,}\DataTypeTok{n.1=}\NormalTok{Ne,}\DataTypeTok{n.2=}\NormalTok{Nc, }\DataTypeTok{verbose =} \OtherTok{FALSE}\NormalTok{)}
\NormalTok{output}\OperatorTok{$}\NormalTok{r<-}\KeywordTok{abs}\NormalTok{(output}\OperatorTok{$}\NormalTok{r)}
\NormalTok{tot<-}\KeywordTok{data.frame}\NormalTok{(}\KeywordTok{paste}\NormalTok{(}\StringTok{"r("}\NormalTok{,output}\OperatorTok{$}\NormalTok{N.total}\DecValTok{-2}\NormalTok{,}\StringTok{")="}\NormalTok{,output}\OperatorTok{$}\NormalTok{r))}
\KeywordTok{colnames}\NormalTok{(tot)<-}\KeywordTok{c}\NormalTok{(}\StringTok{"output"}\NormalTok{)}
\NormalTok{tot}\OperatorTok{$}\NormalTok{output<-}\KeywordTok{gsub}\NormalTok{(}\StringTok{" "}\NormalTok{, }\StringTok{""}\NormalTok{, tot}\OperatorTok{$}\NormalTok{output, }\DataTypeTok{fixed =} \OtherTok{TRUE}\NormalTok{)}
\NormalTok{totoutput<-}\KeywordTok{as.character}\NormalTok{(tot}\OperatorTok{$}\NormalTok{output)}
\KeywordTok{print}\NormalTok{(tot, }\DataTypeTok{row.names =} \OtherTok{FALSE}\NormalTok{)}
\KeywordTok{write}\NormalTok{(totoutput,}\DataTypeTok{ncolumns=}\DecValTok{1}\NormalTok{, }\DataTypeTok{file=}\StringTok{"input.txt"}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

To use the \texttt{pcurve\_dataprep} function, we simply have to specify our dataset. In my case, this is \texttt{metacont}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{pcurve_dataprep}\NormalTok{(}\DataTypeTok{data=}\NormalTok{metacont)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##       output
##    r(98)=0.2
##  r(127)=0.13
##  r(183)=0.36
##   r(58)=0.35
##  r(144)=0.36
##  r(118)=0.08
\end{verbatim}

The function gives us \textbf{the correct format of our effect sizes (t-values)} which we need to conduct the p-curve analysis.

The function also \textbf{automatically stores a .txt-File (input.txt) in your working directory folder on your computer}.

If you forgot where your working directory is, use the \texttt{getwd} function.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{getwd}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "C:/Git_Repositories/Doing-Meta-Analysis-in-R"
\end{verbatim}

This tells you the path to your working directory, where the \textbf{input.txt} file should be stored.

\hypertarget{creating-p-curves}{%
\subsection{Creating P-Curves}\label{creating-p-curves}}

To create p-curves, we have to use the \texttt{pcurve\_app} function. The code for this function is very long, so it is not displayed here. The code for the function has been made publically available online by \textbf{Uri Simonsohn, Leif Nelson, and Joseph Simmons} and can be found \href{http://p-curve.com/app4/pcurve_app4.06.r}{here}.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

To use the \texttt{pcurve\_app} function, we have to specify that the function should use the \textbf{input.txt} in which the data we created using the \texttt{pcurve\_dataprep} function is stored. We also have to provide the function with the \textbf{path to our folder/working directory where the file is stored}.

\begin{rmdachtung}
Please not that while the standard way that \textbf{Windows} provides
you with paths is using \textbf{backslashes}, we need to use
\textbf{normal slashes} (\textbf{/}) for our designated path.
\end{rmdachtung}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{pcurve_app}\NormalTok{(}\StringTok{"input.txt"}\NormalTok{,}\StringTok{"C://Users/Admin/Documents/R/WORKING_DIRECTORY/Meta-Analyse Buch/bookdown-demo-master"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The function automatically \textbf{creates and stores several files and figures into our folder/working directory}. Here the most important ones:

\textbf{Figure 1: ``input.png''}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-181-1.pdf}

This figure shows you the \textbf{p-curve for your results (in blue)}. On the bottom, you can also find the number of effect sizes with \(p<0.05\) which were included in the analysis. There are two tests displayed in the plot.

\textbf{The test for right-skewness}

If there is evidential value behind our data, the p-curve should be \textbf{right-skewed}. Through eyeballing, we see that this is pretty much the case here, and the \textbf{tests for the half and full p-curve} are also both \textbf{significant} (\(p_{Full}<0.001, p_{Half}<0.001\)). This means that the p-curve is heavily right-skewed, indicating that \textbf{evidential value is present in our data}

\textbf{The test for flatness}

If there is evidential value behind our data, the p-curve should also \textbf{not be flat}. Through eyeballing, we see that this is pretty much the case here. The \textbf{tests for flatness} are both not significant (\(p_{Full}=0.9887, p_{Binomial}=0.7459\)).

\textbf{Figure 2: ``input\_fit.png''}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-182-1.pdf}

This plot estimates the \textbf{power} behind our data, meaning if we have sufficient studies with sufficient participants to find a true effect if it exists. A conventional threshold for optimal power is \textbf{80\%}, but P-curve can even assess evidential value if studies are \textbf{underpowered}. In our case, the the power estimate is \textbf{90\%}.

\textbf{Figure 3: ``input\_cumulative.png''}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-183-1.pdf}

This plot provides \textbf{sensitivity analyses} in which the \textbf{highest and lowest p-values are dropped}.

\hypertarget{estimating-the-true-effect-of-our-data}{%
\subsection{Estimating the ``true'' effect of our data}\label{estimating-the-true-effect-of-our-data}}

To estimate the \textbf{true effect of our data} with p-curve (much like the \protect\hyperlink{dant}{Duval \& Tweedie trim-and-fill procedure}), we can use the \texttt{plotloss} function described and made openly available by Simonsohn et al. \citep{simonsohn2014pb}.

The code for this function is quite long, so it is not displayed here. It can accessed using this \href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/true_effect_estimation.R}{link}.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

For the \texttt{plotloss} function, we only have to provide the \textbf{data} to be used to find the ``true'' effect size underlying the data, and a \textbf{range of effect sizes in which the function should search for the true effect}, delineated by \texttt{dmin} and \texttt{dmax}.

I will use my \texttt{metacont} data again here, and will search for the true effect between \(d=0\) to \(d=1.0\).

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{plotloss}\NormalTok{(}\DataTypeTok{data=}\NormalTok{metacont,}\DataTypeTok{dmin=}\DecValTok{0}\NormalTok{,}\DataTypeTok{dmax=}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-185-1.pdf}

\begin{verbatim}
## NULL
\end{verbatim}

The function provides an effect estimate of \(d=0.64\).

\begin{rmdachtung}
It should be noted that this chapter should only be seen as \textbf{an
introduction into p-Curve}, which should not be seen as comprehensive.

Simonsohn et al. {[}@simonsohn2015better{]} also stress that P-Curve
should only be used for \textbf{outcome data which was actually of
interest for the Authors of the specific article}, because those are the
one's likely to get p-hacked. They also ask meta-researchers to provide
a \textbf{detailed table in which the reported results of each outcome
data used in the p-Curve is documented} (a guide can be found
\href{http://p-curve.com/guide.pdf}{here}).

It has also been shown that P-Curve's effect estimate are \textbf{not
robust when the heterogeneity of a meta-analyis is high}
(\emph{I}\textsuperscript{2} \textgreater{} 50\%). Van Aert et al.
{[}@van2016conducting{]} propose \textbf{not to determine the ``true''
effect using P-Curve when heterogeneity is high} (defined as
I\textsuperscript{2} \textgreater{} 50\%).

A poosible solution for this problem might be to \textbf{reduce the
overall heterogeneity using outlier removal}, or to p-Curve results in
\textbf{more homogeneous subgroups}.
\end{rmdachtung}

\hypertarget{creating-a-revman-style-risk-of-bias-summary}{%
\chapter{Creating a RevMan style Risk of Bias summary}\label{creating-a-revman-style-risk-of-bias-summary}}

\begin{figure}
\centering
\includegraphics{robplot.png}
\caption{A finished Risk of Bias summary}
\end{figure}

\hypertarget{preparing-your-risk-of-bias-data}{%
\section{Preparing your Risk of Bias data}\label{preparing-your-risk-of-bias-data}}

In many Meta-Analyses, you will also want to have a look at the quality of included studies using the \href{https://handbook-5-1.cochrane.org/chapter_8/8_6_presentation_of_assessments_of_risk_of_bias.htm}{\textbf{RevMan Risk of Bias assessment tool}}. However, many researchers don't use \textbf{RevMan} to conduct Meta-Analyses, one has to put some extra effort into inserting all study quality data by hand into RevMan, which is extremely time-consuming.

Furthermore, the quality of the Risk of Bias (RoB) summary figures in RevMan are of suboptimal picture quality. Many journals will require figures with better quality, or figures saved in another format (such as \textbf{.svg} or \textbf{.pdf}).

\begin{figure}
\centering
\includegraphics{robsummaryrevman.png}
\caption{This is the output created by RevMan}
\end{figure}

To avoid all of this, you can easily plot the \textbf{RoB Summary in RStudio yourself}.
To do this, we again have to prepare an EXCEL sheet in which we store our RoB data. In \protect\hyperlink{excel_preparation}{Chapter 3.1.1}, we already described how the preparation and import of EXCEL sheets into RStudio works in general. For this data sheet, you need to follow a few guidelines:

\begin{itemize}
\tightlist
\item
  Name the first column \textbf{Author}, and put all the study names in this column (e.g., Frogeli et al.)
\item
  Give the other \textbf{columns a name signifying the RoB criterion} you assessed. Do this for all criteria you want to have included in your plot. \textbf{Important}: when naming your column, do not use spaces between word, but use underscores or points instead (e.g.~allocation\_concealment)
\item
  In these columns, you have to describe if the study received a \textbf{High}, \textbf{Low}, or \textbf{Unclear} risk of bias rating. Use exactly these codes for your data, including upper and lowercase (R is case sensitive)
\item
  Do \textbf{not} store any other information in your data (e.g.~commentaries on your RoB decision)
\end{itemize}

\hypertarget{plotting-the-summary}{%
\section{Plotting the summary}\label{plotting-the-summary}}

To plot the summary, we have to import our dataset first. We describe how to do this in \protect\hyperlink{import_excel}{Chapter 3.2}. I simply called my dataset \texttt{rob}.

Let's have a look at the structure of the data first:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{str}\NormalTok{(rob)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Classes 'tbl_df', 'tbl' and 'data.frame':    43 obs. of  13 variables:
##  $ RoB_sequence                                : chr  "Low" "Low" "Low" "Low" ...
##  $ RoB_allocation_concealment_                 : chr  "Low" "Low" "Low" "Low" ...
##  $ RoB_Blinding of participants and personnel  : chr  "High" "High" "High" "High" ...
##  $ RoB_Blinding of outcome assessors           : chr  "Low" "Low" "Low" "Low" ...
##  $ RoB_Incomplete outcome data for all outcomes: chr  "Low" "Low" "Low" "Low" ...
##  $ RoB_Selective outcome reporting             : chr  "Low" "Unclear" "Low" "Low" ...
##  $ RoB_Co-interventions                        : chr  "Unclear" "Low" "Low" "Low" ...
##  $ RoB_Serious_flaw                            : chr  "Low" "Low" "Low" "Low" ...
##  $ RoB_ITT                                     : chr  "Low" "Low" "Low" "Low" ...
##  $ RoB_SimilarGroups                           : chr  "Unclear" "Low" "Low" "Low" ...
##  $ RoB_compliance                              : chr  "Low" "Unclear" "High" "High" ...
##  $ RoB_identical_post_timing                   : chr  "Low" "Low" "Low" "Low" ...
##  $ Author                                      : chr  "BiesheuvelLeliefeld 2017" "Bockting 2005 & 2010 & 2015" "Bockting 2018" "Bockting 2018" ...
\end{verbatim}

We can see that we have the data imported in RStudio now, with ratings for every criterion in each column. If you named your columns differently, or used less or more criteria, this is not that important now. We will get to this later.

We will plot the summary using the packages \texttt{ggplot2} and \texttt{tidyr}. They should be installed as part of the \texttt{tidyverse}, but be sure to have it on your computer, and then load them from your \textbf{library}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(ggplot2)}
\KeywordTok{library}\NormalTok{(tidyr)}
\end{Highlighting}
\end{Shaded}

We have prepared a function called \texttt{rob.summary} for you, which **automatically plots the risk of bias summary for your prepared data set. The code for this function can be seen below.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{rob.summary<-}\ControlFlowTok{function}\NormalTok{(data)\{}
\NormalTok{    rob.vars<-}\KeywordTok{data.frame}\NormalTok{(data)}
\NormalTok{    rob.vars}\OperatorTok{$}\NormalTok{Author<-}\OtherTok{NULL}
\NormalTok{    ncol.rob.vars<-}\KeywordTok{ncol}\NormalTok{(rob.vars)}
\NormalTok{    last<-}\KeywordTok{colnames}\NormalTok{(rob.vars[ncol.rob.vars])}
\NormalTok{    first<-}\KeywordTok{colnames}\NormalTok{(rob.vars[}\DecValTok{1}\NormalTok{])}
\NormalTok{    rob.long <-}\StringTok{ }\KeywordTok{gather}\NormalTok{(data,}
\NormalTok{                   condition, measurement,}
\NormalTok{                   first}\OperatorTok{:}\NormalTok{last,}
                   \DataTypeTok{factor_key=}\OtherTok{TRUE}\NormalTok{)}
\NormalTok{    rob.long}\OperatorTok{$}\NormalTok{measurement<-}\KeywordTok{as.factor}\NormalTok{(rob.long}\OperatorTok{$}\NormalTok{measurement)}
\NormalTok{    rob.long}\OperatorTok{$}\NormalTok{measurement<-}\KeywordTok{factor}\NormalTok{(rob.long}\OperatorTok{$}\NormalTok{measurement,}
                             \KeywordTok{levels}\NormalTok{(rob.long}\OperatorTok{$}\NormalTok{measurement)[}\KeywordTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{3}\NormalTok{,}\DecValTok{2}\NormalTok{)])}
\NormalTok{    rob.plot<-}\KeywordTok{ggplot}\NormalTok{(}\DataTypeTok{data=}\NormalTok{rob.long)}\OperatorTok{+}
\StringTok{      }\KeywordTok{geom_bar}\NormalTok{(}\DataTypeTok{mapping=}\KeywordTok{aes}\NormalTok{(}\DataTypeTok{x=}\NormalTok{condition,}\DataTypeTok{fill=}\NormalTok{measurement),}
           \DataTypeTok{width=}\FloatTok{0.7}\NormalTok{,}
           \DataTypeTok{position =} \StringTok{"fill"}\NormalTok{,}
           \DataTypeTok{color=}\StringTok{"black"}\NormalTok{)}\OperatorTok{+}
\StringTok{      }\KeywordTok{coord_flip}\NormalTok{(}\DataTypeTok{ylim =} \KeywordTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{))}\OperatorTok{+}
\StringTok{      }\KeywordTok{guides}\NormalTok{(}\DataTypeTok{fill =} \KeywordTok{guide_legend}\NormalTok{(}\DataTypeTok{reverse =} \OtherTok{TRUE}\NormalTok{))}\OperatorTok{+}
\StringTok{      }\KeywordTok{scale_fill_manual}\NormalTok{(}\StringTok{"Risk of Bias"}\NormalTok{,}
                    \DataTypeTok{labels =} \KeywordTok{c}\NormalTok{(}\StringTok{"    High risk of bias          "}\NormalTok{,}
                               \StringTok{"    Unclear risk of bias       "}\NormalTok{,}
                               \StringTok{"    Low risk of bias  "}\NormalTok{),}
                    \DataTypeTok{values =} \KeywordTok{c}\NormalTok{(}\StringTok{"Unclear"}\NormalTok{ =}\StringTok{ "#E2DF07"}\NormalTok{,}
                               \StringTok{"High"}\NormalTok{ =}\StringTok{ "#BF0000"}\NormalTok{,}
                               \StringTok{"Low"}\NormalTok{ =}\StringTok{ "#02C100"}\NormalTok{))}\OperatorTok{+}
\StringTok{      }\KeywordTok{scale_y_continuous}\NormalTok{(}\DataTypeTok{labels =}\NormalTok{ scales}\OperatorTok{::}\NormalTok{percent)}\OperatorTok{+}
\StringTok{      }\KeywordTok{theme}\NormalTok{(}\DataTypeTok{axis.title.x=}\KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.title.y=}\KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.ticks.y=}\KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.text.y =} \KeywordTok{element_text}\NormalTok{(}\DataTypeTok{size=}\DecValTok{18}\NormalTok{, }\DataTypeTok{color =} \StringTok{"black"}\NormalTok{),}
        \DataTypeTok{axis.line.x =} \KeywordTok{element_line}\NormalTok{(}\DataTypeTok{colour =} \StringTok{"black"}\NormalTok{,}
                                   \DataTypeTok{size =} \FloatTok{0.5}\NormalTok{, }\DataTypeTok{linetype =} \StringTok{"solid"}\NormalTok{),}
        \DataTypeTok{legend.position =} \StringTok{"bottom"}\NormalTok{,}
        \DataTypeTok{panel.grid.major =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{panel.grid.minor =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{panel.background =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{legend.background =} \KeywordTok{element_rect}\NormalTok{(}\DataTypeTok{linetype=}\StringTok{"solid"}\NormalTok{,}
                                         \DataTypeTok{colour =}\StringTok{"black"}\NormalTok{),}
        \DataTypeTok{legend.title =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{legend.key.size =} \KeywordTok{unit}\NormalTok{(}\FloatTok{0.75}\NormalTok{,}\StringTok{"cm"}\NormalTok{),}
        \DataTypeTok{legend.text=}\KeywordTok{element_text}\NormalTok{(}\DataTypeTok{size=}\DecValTok{14}\NormalTok{))}
\KeywordTok{return}\NormalTok{(rob.plot)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

To use the \texttt{rob.summary}, we have to specify our RoB data set in the function. In my case, this is \texttt{rob}. I will call my output \texttt{plot}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{plot<-}\KeywordTok{rob.summary}\NormalTok{(}\DataTypeTok{data=}\NormalTok{rob)}
\NormalTok{plot}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-191-1.pdf}

We see that the \texttt{rob.summary} function has already \textbf{created a RoB summary plot}, but there's something missing. The labels for each RoB criterion on the left are the \textbf{raw names} we gave each column, so we should change them to their actual name. To do this, we have to attach the correct labels for each criterion to our \texttt{plot} object.

To do this, we have to attach \texttt{+} a function which describes which label should be replaced with which name. A generic version looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{new.plot<-plot}\OperatorTok{+}
\StringTok{  }\KeywordTok{scale_x_discrete}\NormalTok{(}\DataTypeTok{labels=}\KeywordTok{c}\NormalTok{(}\StringTok{"former_criterion_name_1"}\NormalTok{ =}\StringTok{ "New Criterion Name 1"}\NormalTok{, }
                            \StringTok{"former_criterion_name_2"}\NormalTok{ =}\StringTok{ "New Criterion Name 2"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Just define \textbf{all} the criterion labels this was way, and do not forget to separate them with a comma within the bracket (except for the last one).

If i do this for my data, it looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{new.plot<-plot}\OperatorTok{+}\KeywordTok{scale_x_discrete}\NormalTok{(}\DataTypeTok{labels=}\KeywordTok{c}\NormalTok{(}\StringTok{"RoB_sequence"}\NormalTok{ =}\StringTok{ "Sequence generation"}\NormalTok{,}
                            \StringTok{"RoB_allocation_concealment_"}\NormalTok{ =}\StringTok{ "Allocation Concealment"}\NormalTok{,}
                            \StringTok{"RoB_Blinding of participants and personnel"}\NormalTok{ =}\StringTok{ "Blinding of participants and personnel"}\NormalTok{,}
                            \StringTok{"RoB_Blinding of outcome assessors"}\NormalTok{ =}\StringTok{ "Blinding of outcome assessors"}\NormalTok{,}
                            \StringTok{"RoB_Incomplete outcome data for all outcomes"}\NormalTok{ =}\StringTok{ "Incomplete outcome data"}\NormalTok{,}
                            \StringTok{"RoB_Selective outcome reporting"}\NormalTok{ =}\StringTok{ "Selective outcome reporting"}\NormalTok{,}
                            \StringTok{"RoB_Co-interventions"}\NormalTok{ =}\StringTok{ "Co-Interventions"}\NormalTok{,}
                            \StringTok{"RoB_Serious_flaw"}\NormalTok{ =}\StringTok{ "Serious flaw"}\NormalTok{,}
                            \StringTok{"RoB_ITT"}\NormalTok{ =}\StringTok{ "ITT Analyses"}\NormalTok{,}
                            \StringTok{"RoB_SimilarGroups"}\NormalTok{ =}\StringTok{ "Similar Groups"}\NormalTok{,}
                            \StringTok{"RoB_compliance"}\NormalTok{ =}\StringTok{ "Compliance"}\NormalTok{,}
                            \StringTok{"RoB_identical_post_timing"}\NormalTok{ =}\StringTok{ "Identical Post Timing"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

We have not updated the labels in the \texttt{plot} object, which is now called \texttt{new.plot}. Let's have a peek how it looks now:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{new.plot}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-195-1.pdf}

Looks good so far.

\hypertarget{saving}{%
\section{Saving the Summary Plot}\label{saving}}

I want to save the plot as a \textbf{PDF} file in my working directory. To do this, define the name of the file as \texttt{rob\_summary.pdf}, and save it at the same time in the correct size and orientation using this code:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{pdf}\NormalTok{(}\DataTypeTok{file=}\StringTok{'rob_summary.pdf'}\NormalTok{, }\DataTypeTok{width =} \FloatTok{11.69}\NormalTok{, }\DataTypeTok{height =} \FloatTok{8.27}\NormalTok{) }
\NormalTok{new.plot;}\KeywordTok{dev.off}\NormalTok{() }
\end{Highlighting}
\end{Shaded}

I can also save the plot as \textbf{PNG} file using this command.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{png}\NormalTok{(}\DataTypeTok{file=}\StringTok{'rob_summary.png'}\NormalTok{, }\DataTypeTok{width =} \DecValTok{842}\NormalTok{, }\DataTypeTok{height =} \DecValTok{595}\NormalTok{) }
\NormalTok{new.plot;}\KeywordTok{dev.off}\NormalTok{() }
\end{Highlighting}
\end{Shaded}

Or as a \textbf{Scalable Vector Graphic} (.svg) with this command.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{svg}\NormalTok{(}\DataTypeTok{file=}\StringTok{'rob_summary.svg'}\NormalTok{, }\DataTypeTok{width =} \FloatTok{11.69}\NormalTok{, }\DataTypeTok{height =} \FloatTok{8.27}\NormalTok{) }
\NormalTok{new.plot;}\KeywordTok{dev.off}\NormalTok{() }
\end{Highlighting}
\end{Shaded}

\hypertarget{network-meta-analysis}{%
\chapter{Network Meta-Analysis}\label{network-meta-analysis}}

\includegraphics{networks.jpg}

Often when doing meta-analysis on the effectiveness of certain interventions, we are less interested if \textbf{one particular intervention is effective} (e.g., because it is quite well established that the intervention can be efficacious), but weather \textbf{one intervention is more or less effective than another type of intervention for some condition}. Yet, once we are interested in \textbf{head-to-head} comparisons between two treatments, we often face the problem that \textbf{only very few, if any, randomized controlled trials have compared the effects of two interventions directly}. This makes it very hard, if not impossible for us to conduct conventional meta-analyses to answer questions on the comparative effects of two or more interventions for one indication or outcome (e.g., different types of psychotherapy for major depression).

Nevertheless, while direct comparisons between two or more interventions may often not exist, it is often the case that the interventions were evaluated in separate randomized controlled trials in which the intervention effects were compared to similar \textbf{control groups} (e.g., waitlist control groups, or placebos). This means that we do have \textbf{indirect comparisons} of the effects of different interventions, because they were compared to the same control condition. \textbf{Multiple-treatments meta-analysis (MTM)} is an extension of conventional meta-analysis which allows us to \textbf{incorporate indirect comparisons}, and thus the simultaneous analysis of several interventions.

These meta-analysis methods are also referred to as \textbf{network meta-analyses}, as such methods allow for \textbf{multiple interventions comparisons to be integrated into our analysis}, which can be formalized as a \textbf{``network''} of comparisons.

\begin{rmdinfo}
\textbf{The idea behind network meta-analysis}

Let's assume you have the results of two randomized controlled trials.
One trial evaluated the effect of cognitive behavioral therapy (CBT) for
depression to a control group. The second trial evaluted the effects of
short-term psychodynamic therapy (STPP) on depression compared to
control. We know the effect size \(\hat\theta\) of both interventions
which was found in the trial compared to control at post-test. These
studies produce \textbf{indirect evidence} for the comparative effect of
CBT versus STPP {[}@schwarzer2015meta{]}:

\[\hat\theta_{CBT vs. STPP}^{indirect}=\hat\theta_{CBT vs. Control}^{direct} -\hat\theta_{STPP vs.Control}^{direct} \]

On the other hand, it may also be the case that we found one study
\textbf{in which the effects of CBT were directly compared to the ones
of STPP}. We will denote this effect as
\(\hat\theta_{CBT vs. STPP}^{direct}\). In network-meta-analysis, we
want to integrate the \textbf{direct} as well as the \textbf{indirect}
evidence to get the most precise effect estimate of the comparative
effects.

According to Schwarzer et al. {[}@schwarzer2015meta{]}, there are two
conditions which have to be met to conduct network meta-analyses:

\begin{itemize}
\tightlist
\item
  The studies are \textbf{independent}
\item
  The effect sizes are \textbf{consistent}. This means that effect sizes
  of interventions comparisons we attain through direct evidence should
  be similar to the one we get from indirect evidence (e.g.,
  \(\theta_{CBT vs. STPP}^{direct}=\theta_{CBT vs. STPP}^{indirect}\)).
  Inconsistency, on the other hand, is
  \(\theta_{CBT vs. STPP}^{direct}-\theta_{CBT vs. STPP}^{indirect0} \not= 0\).
  Assessing and dealing with inconsistency is highly important in
  network meta-analysis.
\end{itemize}

Below, you can see a simple first network of the comparisons between the
control condition and the two interventions. We could also think of a
network where \textbf{some comparisons are simply not available}, as is
the case in the second network.
\end{rmdinfo}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-200-1} 

}

\caption{A simple network}\label{fig:unnamed-chunk-200}
\end{figure}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-201-1} 

}

\caption{A simple network with one missing comparison}\label{fig:unnamed-chunk-201}
\end{figure}

\begin{rmdinfo}
\textbf{Work in progress}

A full version of this Chapter will be available soon.
\end{rmdinfo}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(netmeta)}
\KeywordTok{library}\NormalTok{(plyr)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{netmetdata}
\end{Highlighting}
\end{Shaded}

TE

seTE

treat1

treat2

Author

1

-1.90

0.1414

MCT

Waitlist

DeFronzo1995

2

-0.82

0.0992

MCT

Waitlist

Lewin2007

4

-1.34

0.1435

CBT

Waitlist

Davidson2007

5

-1.10

0.1141

CBT

Waitlist

Wolffenbuttel1999

6

-1.30

0.1268

MBCT

Waitlist

Kipnes2001

7

-0.77

0.1078

CBT

Waitlist

Kerenyi2004

8

0.16

0.0849

MBCT

MCT

Hanefeld2004

9

0.10

0.1831

MBCT

CBT

Derosa2004

10

-1.30

0.1014

CBT

Waitlist

Baksi2004

11

-1.09

0.2263

CBT

Waitlist

Rosenstock2008

12

-1.50

0.1624

CBT

Waitlist

Zhu2003

13

-0.14

0.2239

CBT

MCT

Yang2003

14

-1.20

0.1436

CBT

Placebo

Vongthavaravat2002

15

-0.40

0.1549

Psychodynamic

Placebo

Oyama2008

16

-0.80

0.1432

Psychodynamic

Waitlist

Costa1997

17

-0.57

0.1291

System

Waitlist

Hermansen2007

18

-0.70

0.1273

Gestalt

Waitlist

Garber2008

19

-0.37

0.1184

MCT

Placebo

Alex1998

20

-0.74

0.1839

Psychoanalysis

Waitlist

Johnston1994

21

-1.41

0.2235

Psychoanalysis

Waitlist

Johnston1998a

22

0.00

0.2339

CBT

MCT

Kim2007

23

-0.68

0.2828

Psychoanalysis

Waitlist

Johnston1998b

24

-0.40

0.4356

MCT

Waitlist

Gonzalez-Ortiz2004

25

-0.23

0.3467

ACT

Waitlist

Stucci1996

26

-1.01

0.1366

ACT

Waitlist

Moulin2006

27

-1.20

0.3758

MCT

Waitlist

Ebert2018

28

-1.00

0.4669

Psychodynamic

Waitlist

Ebert2018

29

-0.20

0.3579

MCT

Psychodynamic

Ebert2018

Results

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{netmet <-}\StringTok{ }\KeywordTok{netmeta}\NormalTok{(TE, seTE, treat1, treat2, }\DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author), }\DataTypeTok{data=}\NormalTok{netmetdata, }\DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{,}\DataTypeTok{reference.group =} \StringTok{"Waitlist"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Note, treatments within a comparison have been re-sorted in
## increasing order.
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{netmet}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Original data (with adjusted standard errors for multi-arm studies):
## 
##                            treat1        treat2      TE   seTE seTE.adj
## DeFronzo1995                  MCT      Waitlist -1.9000 0.1414   0.1414
## Lewin2007                     MCT      Waitlist -0.8200 0.0992   0.0992
## Davidson2007                  CBT      Waitlist -1.3400 0.1435   0.1435
## Wolffenbuttel1999             CBT      Waitlist -1.1000 0.1141   0.1141
## Kipnes2001                   MBCT      Waitlist -1.3000 0.1268   0.1268
## Kerenyi2004                   CBT      Waitlist -0.7700 0.1078   0.1078
## Hanefeld2004                 MBCT           MCT  0.1600 0.0849   0.0849
## Derosa2004                    CBT          MBCT -0.1000 0.1831   0.1831
## Baksi2004                     CBT      Waitlist -1.3000 0.1014   0.1014
## Rosenstock2008                CBT      Waitlist -1.0900 0.2263   0.2263
## Zhu2003                       CBT      Waitlist -1.5000 0.1624   0.1624
## Yang2003                      CBT           MCT -0.1400 0.2239   0.2239
## Vongthavaravat2002            CBT       Placebo -1.2000 0.1436   0.1436
## Oyama2008                 Placebo Psychodynamic  0.4000 0.1549   0.1549
## Costa1997           Psychodynamic      Waitlist -0.8000 0.1432   0.1432
## Hermansen2007              System      Waitlist -0.5700 0.1291   0.1291
## Garber2008                Gestalt      Waitlist -0.7000 0.1273   0.1273
## Alex1998                      MCT       Placebo -0.3700 0.1184   0.1184
## Johnston1994       Psychoanalysis      Waitlist -0.7400 0.1839   0.1839
## Johnston1998a      Psychoanalysis      Waitlist -1.4100 0.2235   0.2235
## Kim2007                       CBT           MCT  0.0000 0.2339   0.2339
## Johnston1998b      Psychoanalysis      Waitlist -0.6800 0.2828   0.2828
## Gonzalez-Ortiz2004            MCT      Waitlist -0.4000 0.4356   0.4356
## Stucci1996                    ACT      Waitlist -0.2300 0.3467   0.3467
## Moulin2006                    ACT      Waitlist -1.0100 0.1366   0.1366
## Ebert2018                     MCT      Waitlist -1.2000 0.3758   0.4125
## Ebert2018           Psychodynamic      Waitlist -1.0000 0.4669   0.8242
## Ebert2018                     MCT Psychodynamic -0.2000 0.3579   0.3884
##                    narms multiarm
## DeFronzo1995           2         
## Lewin2007              2         
## Davidson2007           2         
## Wolffenbuttel1999      2         
## Kipnes2001             2         
## Kerenyi2004            2         
## Hanefeld2004           2         
## Derosa2004             2         
## Baksi2004              2         
## Rosenstock2008         2         
## Zhu2003                2         
## Yang2003               2         
## Vongthavaravat2002     2         
## Oyama2008              2         
## Costa1997              2         
## Hermansen2007          2         
## Garber2008             2         
## Alex1998               2         
## Johnston1994           2         
## Johnston1998a          2         
## Kim2007                2         
## Johnston1998b          2         
## Gonzalez-Ortiz2004     2         
## Stucci1996             2         
## Moulin2006             2         
## Ebert2018              3        *
## Ebert2018              3        *
## Ebert2018              3        *
## 
## Number of treatment arms (by study):
##                    narms
## Alex1998               2
## Baksi2004              2
## Costa1997              2
## Davidson2007           2
## DeFronzo1995           2
## Derosa2004             2
## Ebert2018              3
## Garber2008             2
## Gonzalez-Ortiz2004     2
## Hanefeld2004           2
## Hermansen2007          2
## Johnston1994           2
## Johnston1998a          2
## Johnston1998b          2
## Kerenyi2004            2
## Kim2007                2
## Kipnes2001             2
## Lewin2007              2
## Moulin2006             2
## Oyama2008              2
## Rosenstock2008         2
## Stucci1996             2
## Vongthavaravat2002     2
## Wolffenbuttel1999      2
## Yang2003               2
## Zhu2003                2
## 
## Results (fixed effect model):
## 
##                            treat1        treat2     SMD             95%-CI
## DeFronzo1995                  MCT      Waitlist -1.1141 [-1.2309; -0.9973]
## Lewin2007                     MCT      Waitlist -1.1141 [-1.2309; -0.9973]
## Davidson2007                  CBT      Waitlist -1.2018 [-1.2953; -1.1084]
## Wolffenbuttel1999             CBT      Waitlist -1.2018 [-1.2953; -1.1084]
## Kipnes2001                   MBCT      Waitlist -1.0664 [-1.2151; -0.9178]
## Kerenyi2004                   CBT      Waitlist -1.2018 [-1.2953; -1.1084]
## Hanefeld2004                 MBCT           MCT  0.0477 [-0.0891;  0.1845]
## Derosa2004                    CBT          MBCT -0.1354 [-0.2957;  0.0249]
## Baksi2004                     CBT      Waitlist -1.2018 [-1.2953; -1.1084]
## Rosenstock2008                CBT      Waitlist -1.2018 [-1.2953; -1.1084]
## Zhu2003                       CBT      Waitlist -1.2018 [-1.2953; -1.1084]
## Yang2003                      CBT           MCT -0.0877 [-0.2203;  0.0449]
## Vongthavaravat2002            CBT       Placebo -0.7623 [-0.9427; -0.5820]
## Oyama2008                 Placebo Psychodynamic  0.3879 [ 0.1662;  0.6095]
## Costa1997           Psychodynamic      Waitlist -0.8274 [-1.0401; -0.6147]
## Hermansen2007              System      Waitlist -0.5700 [-0.8230; -0.3170]
## Garber2008                Gestalt      Waitlist -0.7000 [-0.9495; -0.4505]
## Alex1998                      MCT       Placebo -0.6746 [-0.8482; -0.5011]
## Johnston1994       Psychoanalysis      Waitlist -0.9439 [-1.1927; -0.6952]
## Johnston1998a      Psychoanalysis      Waitlist -0.9439 [-1.1927; -0.6952]
## Kim2007                       CBT           MCT -0.0877 [-0.2203;  0.0449]
## Johnston1998b      Psychoanalysis      Waitlist -0.9439 [-1.1927; -0.6952]
## Gonzalez-Ortiz2004            MCT      Waitlist -1.1141 [-1.2309; -0.9973]
## Stucci1996                    ACT      Waitlist -0.9052 [-1.1543; -0.6561]
## Moulin2006                    ACT      Waitlist -0.9052 [-1.1543; -0.6561]
## Ebert2018                     MCT      Waitlist -1.1141 [-1.2309; -0.9973]
## Ebert2018           Psychodynamic      Waitlist -0.8274 [-1.0401; -0.6147]
## Ebert2018                     MCT Psychodynamic -0.2867 [-0.5113; -0.0622]
##                        Q leverage
## DeFronzo1995       30.89     0.18
## Lewin2007           8.79     0.36
## Davidson2007        0.93     0.11
## Wolffenbuttel1999   0.80     0.17
## Kipnes2001          3.39     0.36
## Kerenyi2004        16.05     0.20
## Hanefeld2004        1.75     0.68
## Derosa2004          0.04     0.20
## Baksi2004           0.94     0.22
## Rosenstock2008      0.24     0.04
## Zhu2003             3.37     0.09
## Yang2003            0.05     0.09
## Vongthavaravat2002  9.29     0.41
## Oyama2008           0.01     0.53
## Costa1997           0.04     0.57
## Hermansen2007       0.00     1.00
## Garber2008          0.00     1.00
## Alex1998            6.62     0.56
## Johnston1994        1.23     0.48
## Johnston1998a       4.35     0.32
## Kim2007             0.14     0.08
## Johnston1998b       0.87     0.20
## Gonzalez-Ortiz2004  2.69     0.02
## Stucci1996          3.79     0.13
## Moulin2006          0.59     0.87
## Ebert2018           0.04     0.02
## Ebert2018           0.04     0.02
## Ebert2018           0.05     0.09
## 
## Results (random effects model):
## 
##                            treat1        treat2     SMD             95%-CI
## DeFronzo1995                  MCT      Waitlist -1.1268 [-1.4291; -0.8244]
## Lewin2007                     MCT      Waitlist -1.1268 [-1.4291; -0.8244]
## Davidson2007                  CBT      Waitlist -1.2335 [-1.4839; -0.9830]
## Wolffenbuttel1999             CBT      Waitlist -1.2335 [-1.4839; -0.9830]
## Kipnes2001                   MBCT      Waitlist -1.1291 [-1.5596; -0.6986]
## Kerenyi2004                   CBT      Waitlist -1.2335 [-1.4839; -0.9830]
## Hanefeld2004                 MBCT           MCT -0.0023 [-0.4444;  0.4398]
## Derosa2004                    CBT          MBCT -0.1044 [-0.5435;  0.3347]
## Baksi2004                     CBT      Waitlist -1.2335 [-1.4839; -0.9830]
## Rosenstock2008                CBT      Waitlist -1.2335 [-1.4839; -0.9830]
## Zhu2003                       CBT      Waitlist -1.2335 [-1.4839; -0.9830]
## Yang2003                      CBT           MCT -0.1067 [-0.4304;  0.2170]
## Vongthavaravat2002            CBT       Placebo -0.8169 [-1.2817; -0.3521]
## Oyama2008                 Placebo Psychodynamic  0.4252 [-0.0951;  0.9456]
## Costa1997           Psychodynamic      Waitlist -0.8418 [-1.3236; -0.3600]
## Hermansen2007              System      Waitlist -0.5700 [-1.2640;  0.1240]
## Garber2008                Gestalt      Waitlist -0.7000 [-1.3927; -0.0073]
## Alex1998                      MCT       Placebo -0.7102 [-1.1713; -0.2491]
## Johnston1994       Psychoanalysis      Waitlist -0.9497 [-1.4040; -0.4955]
## Johnston1998a      Psychoanalysis      Waitlist -0.9497 [-1.4040; -0.4955]
## Kim2007                       CBT           MCT -0.1067 [-0.4304;  0.2170]
## Johnston1998b      Psychoanalysis      Waitlist -0.9497 [-1.4040; -0.4955]
## Gonzalez-Ortiz2004            MCT      Waitlist -1.1268 [-1.4291; -0.8244]
## Stucci1996                    ACT      Waitlist -0.7311 [-1.2918; -0.1705]
## Moulin2006                    ACT      Waitlist -0.7311 [-1.2918; -0.1705]
## Ebert2018                     MCT      Waitlist -1.1268 [-1.4291; -0.8244]
## Ebert2018           Psychodynamic      Waitlist -0.8418 [-1.3236; -0.3600]
## Ebert2018                     MCT Psychodynamic -0.2850 [-0.7908;  0.2208]
## 
## Number of studies: k = 26
## Number of treatments: n = 10
## Number of pairwise comparisons: m = 28
## Number of designs: d = 15
## 
## Fixed effects model
## 
## Treatment estimate (sm = 'SMD', comparison: other treatments vs 'Waitlist'):
##                    SMD             95%-CI
## ACT            -0.9052 [-1.1543; -0.6561]
## CBT            -1.2018 [-1.2953; -1.1084]
## Gestalt        -0.7000 [-0.9495; -0.4505]
## MBCT           -1.0664 [-1.2151; -0.9178]
## MCT            -1.1141 [-1.2309; -0.9973]
## Placebo        -0.4395 [-0.6188; -0.2602]
## Psychoanalysis -0.9439 [-1.1927; -0.6952]
## Psychodynamic  -0.8274 [-1.0401; -0.6147]
## System         -0.5700 [-0.8230; -0.3170]
## Waitlist             .                  .
## 
## Random effects model
## 
## Treatment estimate (sm = 'SMD', comparison: other treatments vs 'Waitlist'):
##                    SMD             95%-CI
## ACT            -0.7311 [-1.2918; -0.1705]
## CBT            -1.2335 [-1.4839; -0.9830]
## Gestalt        -0.7000 [-1.3927; -0.0073]
## MBCT           -1.1291 [-1.5596; -0.6986]
## MCT            -1.1268 [-1.4291; -0.8244]
## Placebo        -0.4166 [-0.8887;  0.0556]
## Psychoanalysis -0.9497 [-1.4040; -0.4955]
## Psychodynamic  -0.8418 [-1.3236; -0.3600]
## System         -0.5700 [-1.2640;  0.1240]
## Waitlist             .                  .
## 
## Quantifying heterogeneity / inconsistency:
## tau^2 = 0.1087; I^2 = 81.4%
## 
## Tests of heterogeneity (within designs) and inconsistency (between designs):
##                     Q d.f.  p-value
## Total           96.99   18 < 0.0001
## Within designs  74.46   11 < 0.0001
## Between designs 22.53    7   0.0021
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{netgraph}\NormalTok{(netmet,}\DataTypeTok{seq =} \KeywordTok{c}\NormalTok{(}\StringTok{"Waitlist"}\NormalTok{,}\StringTok{"Gestalt"}\NormalTok{,}\StringTok{"Psychodynamic"}\NormalTok{,}\StringTok{"MCT"}\NormalTok{,}\StringTok{"MBCT"}\NormalTok{,}\StringTok{"Placebo"}\NormalTok{,}\StringTok{"CBT"}\NormalTok{,}\StringTok{"System"}\NormalTok{,}\StringTok{"Psychoanalysis"}\NormalTok{,}\StringTok{"ACT"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-208-1.pdf}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{netgraph}\NormalTok{(netmet, }\DataTypeTok{start=}\StringTok{"circle"}\NormalTok{, }\DataTypeTok{iterate=}\OtherTok{TRUE}\NormalTok{, }\DataTypeTok{col=}\StringTok{"darkgray"}\NormalTok{, }\DataTypeTok{cex=}\FloatTok{1.5}\NormalTok{, }\DataTypeTok{multiarm=}\OtherTok{TRUE}\NormalTok{, }\DataTypeTok{points=}\OtherTok{TRUE}\NormalTok{, }\DataTypeTok{col.points=}\StringTok{"blue"}\NormalTok{, }\DataTypeTok{cex.points=}\DecValTok{3}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-208-2.pdf}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{forest}\NormalTok{(netmet, }\DataTypeTok{xlim=}\KeywordTok{c}\NormalTok{(}\OperatorTok{-}\FloatTok{1.5}\NormalTok{, }\FloatTok{0.5}\NormalTok{), }\DataTypeTok{ref=}\StringTok{"Waitlist"}\NormalTok{, }\DataTypeTok{leftlabs=}\StringTok{"Contrast to Waitlist"}\NormalTok{, }\DataTypeTok{xlab=}\StringTok{"Effect on Depression (SMD)"}\NormalTok{,}\DataTypeTok{sortvar =}\NormalTok{ TE, }\DataTypeTok{smlab =} \StringTok{""}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-209-1.pdf}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{forest}\NormalTok{(netmet, }\DataTypeTok{xlim=}\KeywordTok{c}\NormalTok{(}\OperatorTok{-}\FloatTok{1.5}\NormalTok{, }\FloatTok{0.5}\NormalTok{), }\DataTypeTok{ref=}\StringTok{"Placebo"}\NormalTok{, }\DataTypeTok{leftlabs=}\StringTok{"Contrast to Placebo"}\NormalTok{, }\DataTypeTok{xlab=}\StringTok{"Effect on Depression (SMD)"}\NormalTok{,}\DataTypeTok{sortvar =}\NormalTok{ TE, }\DataTypeTok{smlab =} \StringTok{""}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-209-2.pdf}

\hypertarget{multilevel-meta-analysis}{%
\chapter{``Multilevel'' Meta-Analysis}\label{multilevel-meta-analysis}}

\includegraphics{multilevel.jpg}

\[ \]

This chapter deals with the topic of pooling effect sizes in \textbf{``multilevel'' meta-analyses}. You probably wonder why we put the word ``multilevel'' into \textbf{quotation marks}. There is an increasing number of studies describing themselves as ``multilevel'' meta-analyses, which insinuates that a ``multilevel'' meta-analysis would be something special or extraordinary compared to ``standard'' meta-analyses. This, however, is not true, as every meta-analytic model presupposes a multilevel structure within the data to pool results \citep{pastor2018multilevel}. This means that basically, once you've reached this chapter in the guide, you will have already fitted a multilevel meta-analytic model to data several times (maybe even without knowing). When people talk about multilevel meta-analysis, what they often think of are \textbf{three-level meta-analytic models}, which are indeed somewhat different to the \protect\hyperlink{fixed}{fixed-effect} and \protect\hyperlink{random}{random-effects models} we presented before.

\begin{rmdinfo}
\textbf{The multilevel nature of Meta-Analysis}

To see why meta-analysis is by nature multileveled, let us go back to
the formula for the random-effects model we discussed in
\protect\hyperlink{random}{Chapter 4.2}:

\[\hat\theta_k = \mu + \epsilon_k + \zeta_k\]

We discussed that the terms \(\epsilon_k\) and \(\zeta_k\) are
introduced in a random-effects model because we assume that there are
two sources of variability. The first one is caused by the
\textbf{sampling error} (\(\epsilon_k\)) of individual studies, which
causes their effect size estimates to deviate from the true effect size
\(\theta_k\). The second one, \(\zeta_k\) is the between-study
heterogeneity caused by the fact that the true effect size of a study
\(k\) itself is again only part of an \textbf{overarching distribution
of true effect sizes}, from which the individual true effect size
\(\theta_k\) was drawn. Our aim in the random-effects model is therefore
to estimate the mean of this distribution of true effect sizes, \(\mu\).

The two error terms correspond with the two levels within our
meta-analysis data: the \textbf{``participant'' level} (level 1) and the
\textbf{``study'' level} (level 2). The figure below symbolizes this
structure. In the lower level (level 1), we have the participants
(and/or patients). These participants are, of course, part of a number
of larger units, the studies (level 2), from which we retrieved the data
for our meta-analysis. While the data on level 1 usually already reaches
us, the meta-analysts, in a ``pooled'' form (e.g., the authors of the
paper provide us with the mean and standard deviation of their studied
sample instead of the raw data), pooling on level 2, the study level,
has to be performed in meta-analysis. In the tradition of multilevel
modeling, such data is called \textbf{nested data}: in most
meta-analyses, one can say that \textbf{participants are ``nested''
within studies}.

\begin{figure}
\centering
\includegraphics{multilevel-model.png}
\caption{A simplified illustration of the multilevel structure of
conventional meta-analytic models}
\end{figure}

Thus, if we split the formula from above into the components
corresponding to the two levels, we get the following formulae:

\textbf{Level 1 (participants) model:}

\begin{equation}
  \label{eq:1}
  \hat\theta_k = \theta_k + \epsilon_k
\end{equation}

\textbf{Level 2 (studies) model:}

\begin{equation}
  \label{eq:2}
  \theta_k = \mu + \zeta_k
\end{equation}

You might have already detected that we can substitute \(\theta_k\) in
the first equation with its definition in the second equation. What we
then get is \textbf{exactly the generic formula for the meta-analytic
model from before} (even a fixed-effects model can be defined as having
a multilevel structure, yet with \(\zeta_k\) being zero). Thus, it
becomes evident that the way we define a meta-analytic model already has
multilevel properties ``built-in'' given that we assume that
participants are nested within studies in our data.
\end{rmdinfo}

\begin{rmdinfo}
\textbf{Three-level meta-analytic models}

It has become clear that meta-analysis naturally possesses a multilevel
structure. This allows us to expand this structure even further to
better reflect our data. \textbf{Three-level models}
{[}@cheung2014modeling; @assink2016fitting{]} are a good way to do this.

\textbf{Statistical independence} is one of the core assumptions of
meta-analytic pooling {[}@hedges2009statistical{]}. If there is a
dependency between effect sizes (i.e., the effect sizes are correlated),
this may artificially reduce heterogeneity and thus lead to
false-positive results.

Dependence may stem from different sources {[}@cheung2014modeling{]}:

\begin{enumerate}
\defenumi{enumi}.{enumi{enumi}.}
\tightlist
\item
  \textbf{Dependence introduced by the authors of the individual
  studies}. For example, scientists conducting the study might have
  collected data from multiple sites, or they might have compared
  \protect\hyperlink{i}{multiple interventions to one single control
  group}, or they might have used different questionnaires to measure an
  outcome of interest (e.g., they used the BDI as well as the PHQ-9 to
  assess depression in patients). For all of these scenarios, we can
  assume that some kind of dependency is introduced within the reported
  data.
\item
  \textbf{Dependence introduced by the meta-analyst herself}. For
  example, a meta-analysis could focus on a specific psychological
  mechanism and include studies which were conducted in different
  cultural regions of the world. It seems reasonable that the reported
  results are more similar when studies were conducted in the same
  cultural context.
\end{enumerate}

We can take such dependencies into account by integrating a
\textbf{third layer} into the structure of our meta-analysis model. For
example, one could model that different questionnaires are nested within
studies. Or one could create a model in which studies are nested within
cultural regions. This creates a three-level meta-analytic model, as
illustrated by the figure below.

\includegraphics{multilevel-model2.png}

The generic mathematical formulae for a three-level meta-analytic model
look like this:

\textbf{Level 1 model}

\[\hat\theta_{ij} = \theta_{ij} + \epsilon_{ij}\]

\textbf{Level 2 model}

\[\theta_{ij} = \kappa_{j} + \zeta_{(2)ij}\]

\textbf{Level 3 model}

\[\kappa_{j} = \beta_{0} + \zeta_{(3)j}\]

Where \(\theta_{ij}\) is the \textbf{true effect size},
\(\hat\theta_{ij}\) its estimator in the \(i\)th effect size in cluster
\(j\), \(\kappa_{j}\) the \textbf{average effect size} in \(j\) and
\(\beta_0\) the average population effect. Again, we can piece these
formulae together and get this:

\[\hat\theta_{ij} = \beta_{0} + \zeta_{(2)ij} + \zeta_{(3)j} + \epsilon_{ij}\]
\end{rmdinfo}

The \texttt{metafor} package is particularly well suited for fitting various three-level models in meta-analyses. When setting up meta-analytic models, we have previously used the \texttt{meta} function primarly, because we think that this package's code is a little less technical. \texttt{metafor}, however, is also fairly easy to use once you prepared the data correctly. We'll get to this in the next chapter.

\hypertarget{fitting-a-three-level-model}{%
\section{Fitting a three-level model}\label{fitting-a-three-level-model}}

\hypertarget{data-preparation}{%
\subsection{Data preparation}\label{data-preparation}}

Before we start with fitting the model, we need the \texttt{metafor} package to be loaded from our library.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(metafor)}
\end{Highlighting}
\end{Shaded}

For this chapter, i will use the \texttt{mlm.data} dataset. It has already been prepared for model fitting using \texttt{metafor}. Let's have a peek at the dataset:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mlm.data}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 80 x 6
##     ID_1  ID_2     y       v `peer-review` dissertation
##    <dbl> <int> <dbl>   <dbl>         <dbl>        <dbl>
##  1     1     1 -0.01 0.0740              1            0
##  2     1     2  0.11 0.0398              1            0
##  3     1     3  0.2  0.0481              1            0
##  4     1     4  0.3  0.0239              1            0
##  5     1     5  0.12 0.0331              1            0
##  6     1     6  0.02 0.0886              1            0
##  7     1     7  0.13 0.0115              1            0
##  8     1     8  0.18 0.00761             1            0
##  9     1     9  0.18 0.00649             1            0
## 10     2    10  0.44 0.332               1            0
## # ... with 70 more rows
\end{verbatim}

We see that the dataset has 80 rows and 5 columns (i.e., 5 variables). The first four are the most important ones:

\begin{itemize}
\tightlist
\item
  \textbf{ID\_1} and \textbf{ID\_2}. In these two columns, we stored the identifiers corresponding to the individual effect sizes and clusters on different levels of our multilevel model. We have left these columns quite generic to make clear that such multilevel data can accommodate various types of data structures. For example, one can think of a scenario where ID\_2 corresponds with the different effect sizes we find within a study (e.g., different effect sizes for different measures of depression). In this example, the effect sizes in ID\_2 would then be nested within the larger clusters defined in ID\_1, the studies. Another possible example would be that ID\_2 in fact symbolizes the unique identifier for single studies, which are then nested in larger clusters (e.g., cultural regions) shown in ID\_1.
\item
  \textbf{y} and \textbf{v}. In these columns, the effect size data is stored. The format is similar to the one we know from the \texttt{metagen} function in \texttt{meta}. The column \texttt{y} contains the effect size (\(d\), \(g\), \(r\), \(z\), or other continuous effect size metrics). The \texttt{v} column contains the sampling variance which can be obtained by \textbf{squaring the standard error}.
\end{itemize}

There are two additional columns which contain data needed for \textbf{subgroup analyses}. To conduct subgroup analyses with \texttt{metafor}, however, we need to store the data a little differently than in \protect\hyperlink{subgroup}{Chapter 7}. This time, we do not use a factor vector containing the subgroup levels as strings, but \textbf{recode all subgroup levels} as individuals columns in the data. In our case, we are interested if the publication status has a moderating effect on the pooled results. We want to compare studies published in peer-review journals to dissertations. To do this, we need to create \textbf{two variables/columns for each subgroup level} (peer-review, dissertation). In each column we provide a code if the effect size belongs to this subgroup, defined by \texttt{1\ =\ yes} and \texttt{0\ =\ no}. Of course, the subgroup codings now have to be \textbf{mutually exclusive}, as we cannot calculate subgroup effects if a study is both part of the ``peer-review'' group and the ``dissertation'' group.

\hypertarget{model-fitting}{%
\subsection{Model fitting}\label{model-fitting}}

We are now prepared to fit our model. I will save the model to the object \texttt{full.model}. For model fitting, we use the \texttt{rma.mv} function in \texttt{metafor}. The function has plenty of parameters one can specify (type \texttt{?metafor::rma.mv} in the console to see them all). For now, we will only focus on the really essential ones.

Code

Description

y

The variable in our dataset containing the effect size data

v

The variable in our dataset containing the sampling variance data corresponding to each y

random

The model specifications for our multilevel model. We describe this in more detail below

tdist

This is an old friend in disguise, the Knapp-Hartung adjustment for our confidence intervals (if set to TRUE)

data

The dataframe containing our data

method

The tau-squared estimator. Our options here are limited, and it is advised to use the restricted maximum likelihood (`REML') method, which is the default

\textbf{Setting up the formula}

Under the \texttt{random} parameter, we feed \texttt{rma.mv} with the formula for our model. As we actually have two formulas in three-level models, we have to bundle them together in a \texttt{list()}. The notation for \texttt{rma.mv} is very similar to other packages specialized for mixed-effects models such as \texttt{lme4} \citep{lme4}.

Within the \texttt{list()}, formulae are separated with commas. Within the formula, we first define the random effects variance as \texttt{\textasciitilde{}\ 1}, followed by a \textbf{vertical bar} \texttt{\textbar{}}. Behind the vertical bar, we assign this random effect to a \textbf{grouping variable} such as studies, measures, regions and so forth. We only have to define the formulae for \textbf{level 2} and \textbf{level 3}, as the sampling error in level 1 is assumed to be known. We type in the formulae in the order of our multilevel model. As in our example, ID\_2 is nested in ID\_1, we first define level 2 as \texttt{\textasciitilde{}\ 1\ \textbar{}\ ID\_2} and then level 3 as \texttt{\textasciitilde{}\ 1\ \textbar{}\ ID\_1}. Now, let's put it all together:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{full.model<-}\KeywordTok{rma.mv}\NormalTok{(y, }
\NormalTok{                   v, }
                   \DataTypeTok{random =} \KeywordTok{list}\NormalTok{(}\OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{2}\NormalTok{, }
                                 \OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{1}\NormalTok{), }
                   \DataTypeTok{tdist =} \OtherTok{TRUE}\NormalTok{, }
                   \DataTypeTok{data =}\NormalTok{ mlm.data,}
                   \DataTypeTok{method =} \StringTok{"REML"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

To see the results of our model, we can use the \texttt{summary()} function.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{summary}\NormalTok{(full.model)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Multivariate Meta-Analysis Model (k = 80; method: REML)
## 
##   logLik  Deviance       AIC       BIC      AICc  
## -44.6962   89.3924   95.3924  102.5007   95.7124  
## 
## Variance Components: 
## 
##             estim    sqrt  nlvls  fixed  factor
## sigma^2.1  0.0561  0.2368     80     no    ID_2
## sigma^2.2  0.1575  0.3968     14     no    ID_1
## 
## Test for Heterogeneity: 
## Q(df = 79) = 624.8063, p-val < .0001
## 
## Model Results:
## 
## estimate      se    tval    pval   ci.lb   ci.ub     
##   0.4345  0.1173  3.7058  0.0004  0.2011  0.6679  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

Let's go through the output one by one. First, let's have a look at the \texttt{Variance\ Components}. Here, we see the variance calculated for each level of our model, \(\sigma^2_1\) (level 2) and \(\sigma^2_2\) (level 3). Under \texttt{nlvls}, we see the number of levels (i.e., subgroups) on each level. For \texttt{ID\_2}, this is 80 because each effect size has its own ID.

Under \texttt{Model\ Results}, we see the \texttt{estimate} of our pooled effect, which is 0.4345. As our \texttt{y} column contained effect sizes expressed as Hedges' g, the effect is therefore \(g = 0.4545\), with a confidence interval of \(0.20-0.67\).

Under \texttt{Test\ for\ Heterogeneity}, we see that the variance of effect sizes within our data overall was highly significant (\(p < 0.0001\)). This however, is not very informative as we are interested how variance can be attributed to the different levels in our model.

\hypertarget{distribution-of-total-variance}{%
\subsection{Distribution of total variance}\label{distribution-of-total-variance}}

As mentioned before, what we're actually interested in is the \textbf{distribution of variance over the three levels of our model}. Cheung (2014) provides a formula to estimate the sampling variance, which is needed to calculate the variance proportions for each level. We have have programmed a function called \texttt{variance.distribution.3lm}, which implements this formula. The code for the function can be seen below. Parts of this code were adapted from Assink and Wibbelink \citep{assink2016fitting}.

R doesn't know this function yet, so we have to let R learn it copying and pasting the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{variance.distribution}\FloatTok{.3}\NormalTok{lm <-}\StringTok{ }\ControlFlowTok{function}\NormalTok{(data, m)\{}
  
  \CommentTok{# Calculate estimated sampling variance and proportions across levels  }
\NormalTok{  data <-}\StringTok{ }\NormalTok{data}
\NormalTok{    m <-}\StringTok{ }\NormalTok{m}
\NormalTok{    n <-}\StringTok{ }\KeywordTok{length}\NormalTok{(data}\OperatorTok{$}\NormalTok{v)}
\NormalTok{    vector.inv.var <-}\StringTok{ }\DecValTok{1}\OperatorTok{/}\NormalTok{(data}\OperatorTok{$}\NormalTok{v)}
\NormalTok{    sum.inv.var <-}\StringTok{ }\KeywordTok{sum}\NormalTok{(vector.inv.var)}
\NormalTok{    sum.sq.inv.var <-}\StringTok{ }\NormalTok{(sum.inv.var)}\OperatorTok{^}\DecValTok{2}
\NormalTok{    vector.inv.var.sq <-}\StringTok{ }\DecValTok{1}\OperatorTok{/}\NormalTok{(data}\OperatorTok{$}\NormalTok{v}\OperatorTok{^}\DecValTok{2}\NormalTok{)}
\NormalTok{    sum.inv.var.sq <-}\StringTok{ }\KeywordTok{sum}\NormalTok{(vector.inv.var.sq)}
\NormalTok{    num <-}\StringTok{ }\NormalTok{(n}\DecValTok{-1}\NormalTok{)}\OperatorTok{*}\NormalTok{sum.inv.var}
\NormalTok{    den <-}\StringTok{ }\NormalTok{sum.sq.inv.var }\OperatorTok{-}\StringTok{ }\NormalTok{sum.inv.var.sq}
\NormalTok{    est.samp.var <-}\StringTok{ }\NormalTok{num}\OperatorTok{/}\NormalTok{den}
\NormalTok{    level1<-((est.samp.var)}\OperatorTok{/}\NormalTok{(m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{1}\NormalTok{]}\OperatorTok{+}\NormalTok{m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{2}\NormalTok{]}\OperatorTok{+}\NormalTok{est.samp.var)}\OperatorTok{*}\DecValTok{100}\NormalTok{)}
\NormalTok{    level2<-((m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{1}\NormalTok{])}\OperatorTok{/}\NormalTok{(m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{1}\NormalTok{]}\OperatorTok{+}\NormalTok{m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{2}\NormalTok{]}\OperatorTok{+}\NormalTok{est.samp.var)}\OperatorTok{*}\DecValTok{100}\NormalTok{)}
\NormalTok{    level3<-((m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{2}\NormalTok{])}\OperatorTok{/}\NormalTok{(m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{1}\NormalTok{]}\OperatorTok{+}\NormalTok{m}\OperatorTok{$}\NormalTok{sigma2[}\DecValTok{2}\NormalTok{]}\OperatorTok{+}\NormalTok{est.samp.var)}\OperatorTok{*}\DecValTok{100}\NormalTok{)}
\NormalTok{    Level<-}\KeywordTok{c}\NormalTok{(}\StringTok{"level 1"}\NormalTok{, }\StringTok{"level 2"}\NormalTok{, }\StringTok{"level 3"}\NormalTok{)}
\NormalTok{    Variance<-}\KeywordTok{c}\NormalTok{(level1, level2, level3)}
\NormalTok{    df<-}\KeywordTok{data.frame}\NormalTok{(Level, Variance)}
\NormalTok{    df1<-df}
    \KeywordTok{colnames}\NormalTok{(df1) <-}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"Level"}\NormalTok{, }\StringTok{"% of total variance"}\NormalTok{)}
    
    \CommentTok{# Generate plot}
\NormalTok{  df}\OperatorTok{$}\NormalTok{distribution<-}\StringTok{"Distribution"}
\NormalTok{  df}\OperatorTok{$}\NormalTok{Level<-}\KeywordTok{factor}\NormalTok{(df}\OperatorTok{$}\NormalTok{Level, }\KeywordTok{levels}\NormalTok{(df}\OperatorTok{$}\NormalTok{Level)[}\KeywordTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{1}\NormalTok{)])}
\NormalTok{  g <-}\StringTok{ }\KeywordTok{ggplot}\NormalTok{(df, }\KeywordTok{aes}\NormalTok{(}\DataTypeTok{fill=}\NormalTok{Level, }\DataTypeTok{y=}\NormalTok{Variance, }\DataTypeTok{x=}\NormalTok{distribution)) }\OperatorTok{+}\StringTok{ }
\StringTok{    }\KeywordTok{geom_bar}\NormalTok{(}\DataTypeTok{stat=}\StringTok{"identity"}\NormalTok{, }\DataTypeTok{position=}\StringTok{"fill"}\NormalTok{, }\DataTypeTok{width =} \FloatTok{0.3}\NormalTok{) }\OperatorTok{+}\StringTok{ }\KeywordTok{coord_flip}\NormalTok{(}\DataTypeTok{ylim =} \KeywordTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{0}\NormalTok{)) }\OperatorTok{+}\StringTok{ }\KeywordTok{scale_y_continuous}\NormalTok{(}\DataTypeTok{labels =}\NormalTok{ scales}\OperatorTok{::}\NormalTok{percent)}\OperatorTok{+}
\StringTok{      }\KeywordTok{theme}\NormalTok{(}\DataTypeTok{axis.title.x=}\KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.title.y=}\KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.ticks.y=}\KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.text.y =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{axis.line.x =} \KeywordTok{element_line}\NormalTok{(}\DataTypeTok{colour =} \StringTok{"black"}\NormalTok{,}
                                   \DataTypeTok{size =} \FloatTok{0.5}\NormalTok{, }\DataTypeTok{linetype =} \StringTok{"solid"}\NormalTok{),}
        \DataTypeTok{legend.position =} \StringTok{"bottom"}\NormalTok{,}
        \DataTypeTok{panel.grid.major =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{panel.grid.minor =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{panel.background =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{legend.background =} \KeywordTok{element_rect}\NormalTok{(}\DataTypeTok{linetype=}\StringTok{"solid"}\NormalTok{,}
                                         \DataTypeTok{colour =}\StringTok{"black"}\NormalTok{),}
        \DataTypeTok{legend.title =} \KeywordTok{element_blank}\NormalTok{(),}
        \DataTypeTok{legend.key.size =} \KeywordTok{unit}\NormalTok{(}\FloatTok{0.75}\NormalTok{,}\StringTok{"cm"}\NormalTok{)) }\OperatorTok{+}\StringTok{ }\KeywordTok{guides}\NormalTok{(}\DataTypeTok{fill =} \KeywordTok{guide_legend}\NormalTok{(}\DataTypeTok{reverse =} \OtherTok{TRUE}\NormalTok{))}
  \KeywordTok{return}\NormalTok{(}\KeywordTok{list}\NormalTok{(g, df1))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Please be aware that the function needs \texttt{ggplot2} installed and loaded to work. There are only two parameters: \texttt{data}, the data we used for our three-level model (in which \texttt{v} signifies the effect size), and \texttt{m}, the model we just fitted using \texttt{rma.mv}. Let's have a look:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(ggplot2)}
\KeywordTok{variance.distribution.3lm}\NormalTok{(}\DataTypeTok{data =}\NormalTok{ mlm.data,}
                          \DataTypeTok{m =}\NormalTok{ full.model)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
\end{verbatim}

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-219-1.pdf}

\begin{verbatim}
## 
## [[2]]
##     Level % of total variance
## 1 level 1            8.937849
## 2 level 2           23.916209
## 3 level 3           67.145942
\end{verbatim}

From the outputs, we see that about 8.94\% of the overall variance can be attributed to level 1, 23.92\% to level 2, and as much as 67.15\% to level 3.

\hypertarget{comparing-the-fit}{%
\subsection{Comparing the fit}\label{comparing-the-fit}}

Of course, when fitting a three-level model, we are interested if this model actually \textbf{captures the variability in our data better than a two-level model}. \texttt{metafor} allows us to easily do this by comparing models \textbf{in which one of the levels is removed}.

To do this, we can use the \texttt{rma.mv} function again, but this time, we hold the variance component \(\sigma^2\) of one level constant. This can be done by specifying the \texttt{sigma2} parameter. The parameter can be specified by providing a vector telling the function which level to hold constant, with the generic version being \texttt{c(level\ 2,\ level3)}. So, if we want to hold level 2 constant while leaving the rest as is, we use \texttt{sigma2\ =\ c(0,NA)}, and vice verse for the third level.

\textbf{Model without level 2}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model.l2.removed<-}\KeywordTok{rma.mv}\NormalTok{(y, }
\NormalTok{                   v, }
                   \DataTypeTok{random =} \KeywordTok{list}\NormalTok{(}\OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{2}\NormalTok{, }
                                 \OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{1}\NormalTok{), }
                   \DataTypeTok{tdist =} \OtherTok{TRUE}\NormalTok{, }
                   \DataTypeTok{data =}\NormalTok{ mlm.data,}
                   \DataTypeTok{method =} \StringTok{"REML"}\NormalTok{,}
                   \DataTypeTok{sigma2 =} \KeywordTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{, }\OtherTok{NA}\NormalTok{))}
\KeywordTok{summary}\NormalTok{(model.l2.removed)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Multivariate Meta-Analysis Model (k = 80; method: REML)
## 
##   logLik  Deviance       AIC       BIC      AICc  
## -59.8748  119.7495  123.7495  128.4884  123.9074  
## 
## Variance Components: 
## 
##             estim    sqrt  nlvls  fixed  factor
## sigma^2.1  0.0000  0.0000     80    yes    ID_2
## sigma^2.2  0.1620  0.4025     14     no    ID_1
## 
## Test for Heterogeneity: 
## Q(df = 79) = 624.8063, p-val < .0001
## 
## Model Results:
## 
## estimate      se    tval    pval   ci.lb   ci.ub     
##   0.4217  0.1130  3.7337  0.0004  0.1969  0.6465  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

We see that the \texttt{Model\ Results} have changed, but does the second level in fact capture a significant amount of variability in the data? To check, we can use the \texttt{anova} function to compare the \texttt{full.model} to the \texttt{model.l2.removed}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{anova}\NormalTok{(full.model, model.l2.removed)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##         df      AIC      BIC     AICc   logLik     LRT   pval       QE
## Full     3  95.3924 102.5007  95.7124 -44.6962                624.8063
## Reduced  2 123.7495 128.4884 123.9074 -59.8748 30.3571 <.0001 624.8063
\end{verbatim}

We see that the \texttt{Full} model compared to the \texttt{Reduced} model does indeed have a better fit, with the \textbf{Akaike Information Criterion} (\texttt{AIC}) and the \textbf{Bayesian Information Criterion} (\texttt{BIC}) being lower for this model. The difference is significant (\(p<0.0001\)), suggesting that it was a good idea to include this level into our analysis. Now, let's do the same when holding level 3 constant.

\textbf{Model without level 3}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model.l3.removed<-}\KeywordTok{rma.mv}\NormalTok{(y, }
\NormalTok{                   v, }
                   \DataTypeTok{random =} \KeywordTok{list}\NormalTok{(}\OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{2}\NormalTok{, }
                                 \OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{1}\NormalTok{), }
                   \DataTypeTok{tdist =} \OtherTok{TRUE}\NormalTok{, }
                   \DataTypeTok{data =}\NormalTok{ mlm.data,}
                   \DataTypeTok{method =} \StringTok{"REML"}\NormalTok{,}
                   \DataTypeTok{sigma2 =} \KeywordTok{c}\NormalTok{(}\OtherTok{NA}\NormalTok{, }\DecValTok{0}\NormalTok{))}
\KeywordTok{summary}\NormalTok{(model.l3.removed)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Multivariate Meta-Analysis Model (k = 80; method: REML)
## 
##   logLik  Deviance       AIC       BIC      AICc  
## -75.4151  150.8302  154.8302  159.5691  154.9881  
## 
## Variance Components: 
## 
##             estim    sqrt  nlvls  fixed  factor
## sigma^2.1  0.3119  0.5585     80     no    ID_2
## sigma^2.2  0.0000  0.0000     14    yes    ID_1
## 
## Test for Heterogeneity: 
## Q(df = 79) = 624.8063, p-val < .0001
## 
## Model Results:
## 
## estimate      se    tval    pval   ci.lb   ci.ub     
##   0.6049  0.0691  8.7539  <.0001  0.4674  0.7424  ***
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{anova}\NormalTok{(full.model, model.l2.removed)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##         df      AIC      BIC     AICc   logLik     LRT   pval       QE
## Full     3  95.3924 102.5007  95.7124 -44.6962                624.8063
## Reduced  2 123.7495 128.4884 123.9074 -59.8748 30.3571 <.0001 624.8063
\end{verbatim}

We see the same pattern here, suggesting that overall, our three-level model is very useful.

\hypertarget{subgroup-analyses-in-three-level-models}{%
\section{Subgroup Analyses in Three-Level Models}\label{subgroup-analyses-in-three-level-models}}

Once our \textbf{three-level meta-analytic model} is set, it is easy to also check for moderators of the overall effect, i.e.~conduct subgroup analyses. To do this, we can use the \texttt{mods} parameter in \texttt{rma.mv}. The subgroup/moderator can be specified by putting a tilde (\texttt{\textasciitilde{}}) in front of it. In our case, we want to check if dissertations report higher or lower effect sizes, so we use \texttt{mods\ =\ \textasciitilde{}\ dissertation}. Here's the full code:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model.mods<-}\KeywordTok{rma.mv}\NormalTok{(y, }
\NormalTok{                   v, }
                   \DataTypeTok{random =} \KeywordTok{list}\NormalTok{(}\OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{2}\NormalTok{, }
                                 \OperatorTok{~}\StringTok{ }\DecValTok{1} \OperatorTok{|}\StringTok{ }\NormalTok{ID_}\DecValTok{1}\NormalTok{), }
                   \DataTypeTok{tdist =} \OtherTok{TRUE}\NormalTok{, }
                   \DataTypeTok{data =}\NormalTok{ mlm.data,}
                   \DataTypeTok{method =} \StringTok{"REML"}\NormalTok{,}
                   \DataTypeTok{mods =} \OperatorTok{~}\StringTok{ }\NormalTok{dissertation)}
\KeywordTok{summary}\NormalTok{(model.mods)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Multivariate Meta-Analysis Model (k = 80; method: REML)
## 
##   logLik  Deviance       AIC       BIC      AICc  
## -43.8184   87.6367   95.6367  105.0636   96.1847  
## 
## Variance Components: 
## 
##             estim    sqrt  nlvls  fixed  factor
## sigma^2.1  0.0581  0.2410     80     no    ID_2
## sigma^2.2  0.1594  0.3992     14     no    ID_1
## 
## Test for Residual Heterogeneity: 
## QE(df = 78) = 428.5167, p-val < .0001
## 
## Test of Moderators (coefficient(s) 2): 
## F(df1 = 1, df2 = 78) = 0.0226, p-val = 0.8808
## 
## Model Results:
## 
##               estimate      se    tval    pval    ci.lb   ci.ub     
## intrcpt         0.4297  0.1229  3.4972  0.0008   0.1851  0.6744  ***
## dissertation    0.0339  0.2255  0.1504  0.8808  -0.4150  0.4828     
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}

The first important output is the \texttt{Test\ of\ Moderators}. We see that \(F_{1,78}=0.0226\) with \(p=0.8808\), meaning that there is no significant difference between subgroups. The \texttt{Model\ Results} are printed within a meta-regression framework, so we \textbf{cannot simply extract the estimates directly to attain the summary effect size within subgroups}. The first row, the \texttt{intrcpt} (intercept) is the value of \(g\) when dissertation = 0, i.e., the value for peer-review articles. The predictor \texttt{dissertation}'s estimate is expressed as a \textbf{slope}, meaning that we have to add its value to the intercept to get the actual summary effect. This is:

\[g_{dissertation} = \beta_0 + \beta_{dissertation} = 0.4297+0.0339=0.4636\]

\[ \]

\hypertarget{effect-size-calculators}{%
\chapter{Effect size calculators}\label{effect-size-calculators}}

\includegraphics{effect.jpg}

Although the \texttt{meta} package can calculate all \textbf{individual effect sizes for every study} if we use the \texttt{metabin} or \texttt{metacont} function, a frequent scenario is that \textbf{some papers do not report the effect size data in the right format}. Especially older articles may often only report results of \(t\)\textbf{-tests}, \textbf{ANOVAs}, or \(\chi^2\)\textbf{-tests}. If enough data is reported, we can also use \textbf{such outcome formats to calculate effect sizes}. This way, we can calculate the \textbf{effect size (e.g., Hedges' g)} and the \textbf{Standard Error (SE)}, which we can then use in a meta-analysis with \textbf{pre-calculated effect sizes} using the \texttt{metagen} function (see \protect\hyperlink{pre.calc}{Chapter 4.1.1}).

\begin{rmdinfo}
\textbf{Hedges' g}

When dealing with \textbf{continuous outcome data}, it is conventional
to calculate the \textbf{Standardized Mean Difference} (SMD) as an
outcome for each study, and as your \textbf{summary measure}
{[}@borenstein2011{]}.

A common format to to calculate the SMD in single trials is
\textbf{Cohen's d} {[}@cohen1988statistical{]}. Yet, this summary
measure has been \textbf{shown to have a slight bias in small studies,
for which it overestimates the effect} {[}@hedges1981distribution{]}.

\textbf{Hedges \emph{g} } is a similar summary measure, but it
\textbf{controls for this bias}. It uses a slightly different formula to
calculate the pooled variance \(s_{pooled}\), \(s*_{pooled}\). The
transformation from \emph{d} to \emph{g} is often performed using the
formula by Hedges and Olkin {[}@hedges1985statistical{]}.

\[g \simeq d\times(1-\frac{3}{4(n_1+n_2)-9}) \]
\end{rmdinfo}

\begin{rmdachtung}
Hedges' g is \textbf{commonly used in meta-analysis}, and it's the
standard output format in \textbf{RevMan}. Therefore, we highly
recommend that you also use this measure in you meta-analysis.

In \texttt{meta}`s \texttt{metabin} and \texttt{metacont} function,
Hedges' g is automatically calculated for each study if we set
\texttt{sm="SMD"}. If you use the \texttt{metgen} function, however, you
should calculate Hedges' g for each study yourself first.
\end{rmdachtung}

To calculate the effect sizes, we will use Daniel Lüdecke's extremely helpful \texttt{esc} package \citep{esc}. So, please \textbf{install this package first} using the \texttt{install.packages("esc")} command, and then load it in you library.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{library}\NormalTok{(esc)}
\end{Highlighting}
\end{Shaded}

\textbf{Here's an overview of all calculators covered in this guide}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \protect\hyperlink{a}{Calculating Hedges' \emph{g} from the Mean and SD}
\item
  \protect\hyperlink{b}{Calculating Hedges' \emph{g} from a regression coefficient}
\item
  \protect\hyperlink{c}{Calculating an Odd's Ratio from \emph{Chi-square}}
\item
  \protect\hyperlink{d}{Calculating Hedges' \emph{g} from a one-way ANOVA}
\item
  \protect\hyperlink{e}{Calculating Hedges' \emph{g} from the Mean and SE}
\item
  \protect\hyperlink{f}{Calculating Hedges' \emph{g} from a correlation}
\item
  \protect\hyperlink{g}{Calculating Hedges' \emph{g} from an independent t-test}
\item
  \protect\hyperlink{h}{Calculating Hedges' \emph{g} from Cohen's \emph{d}}
\item
  \protect\hyperlink{i}{Calculating effect sizes for studies with multiple comparisons}
\end{enumerate}

\hypertarget{a}{%
\section{Calculating Hedges' g from the Mean and SD}\label{a}}

To calculate Hedges' \emph{g} from the \emph{Mean}, \emph{Standard Deviation}, and \(n_{group}\) of both trial arms, we can use the \texttt{esc\_mean\_sd} function with the following parameters.

\begin{itemize}
\tightlist
\item
  \texttt{grp1m}: The \textbf{mean} of the \textbf{first group} (e.g., the intervention).
\item
  \texttt{grp1sd}: The \textbf{standard deviation} of the \textbf{first group}.
\item
  \texttt{grp1n}: The \textbf{sample size} of the \textbf{first group}.
\item
  \texttt{grp2m}: The \textbf{mean} of the \textbf{second group}.
\item
  \texttt{grp2sd}: The \textbf{standard deviation} of the \textbf{second group}.
\item
  \texttt{grp2n}: The \textbf{sample size} of the \textbf{second group}.
\item
  \texttt{totalsd}: The \textbf{full sample standard deviation}, if the standard deviation for each trial arm is not reported
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_mean_sd}\NormalTok{(}\DataTypeTok{grp1m =} \FloatTok{10.3}\NormalTok{, }\DataTypeTok{grp1sd =} \FloatTok{2.5}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{60}\NormalTok{,}
\DataTypeTok{grp2m =} \FloatTok{12.3}\NormalTok{, }\DataTypeTok{grp2sd =} \FloatTok{3.1}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{56}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: mean and sd to effect size Hedges' g
##     Effect Size:  -0.7082
##  Standard Error:   0.1916
##        Variance:   0.0367
##        Lower CI:  -1.0837
##        Upper CI:  -0.3326
##          Weight:  27.2374
\end{verbatim}

\hypertarget{b}{%
\section{\texorpdfstring{Calculating Hedges' \emph{g} from a regression coefficient}{Calculating Hedges' g from a regression coefficient}}\label{b}}

\hypertarget{unstandardized-regression-coefficients}{%
\subsection{Unstandardized regression coefficients}\label{unstandardized-regression-coefficients}}

It is also possible to calculate \textbf{Hedges' \emph{g} } from an unstandardized or standardized regression coeffiecent \citep{lipsey2001practical}.

For \textbf{unstardardized coefficients}, we can use the \texttt{esc\_B} function with the following parameters:

\begin{itemize}
\tightlist
\item
  \texttt{b}: unstandardized coefficient \(b\) (the ``treatment'' predictor).
\item
  \texttt{sdy}: the standard deviation of the dependent variable \(y\) (i.e., the outcome).
\item
  \texttt{grp1n}: the number of participants in the first group.
\item
  \texttt{grp2n}: the number of participants in the second group.
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_B}\NormalTok{(}\DataTypeTok{b=}\FloatTok{3.3}\NormalTok{,}\DataTypeTok{sdy=}\DecValTok{5}\NormalTok{,}\DataTypeTok{grp1n =} \DecValTok{100}\NormalTok{,}\DataTypeTok{grp2n =} \DecValTok{150}\NormalTok{,}\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: unstandardized regression coefficient to effect size Hedges' g
##     Effect Size:   0.6941
##  Standard Error:   0.1328
##        Variance:   0.0176
##        Lower CI:   0.4338
##        Upper CI:   0.9544
##          Weight:  56.7018
\end{verbatim}

\hypertarget{standardized-regression-coefficents}{%
\subsection{Standardized regression coefficents}\label{standardized-regression-coefficents}}

Here, we can use the \texttt{esc\_beta} function with the follwing parameters:

\begin{itemize}
\tightlist
\item
  \texttt{beta}: standardized coefficient \(\beta\) (the ``treatment'' predictor).
\item
  \texttt{sdy}: the standard deviation of the dependent variable \(y\) (i.e., the outcome).
\item
  \texttt{grp1n}: the number of participants in the first group.
\item
  \texttt{grp2n}: the number of participants in the second group.
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_beta}\NormalTok{(}\DataTypeTok{beta=}\FloatTok{0.7}\NormalTok{, }\DataTypeTok{sdy=}\DecValTok{3}\NormalTok{, }\DataTypeTok{grp1n=}\DecValTok{100}\NormalTok{, }\DataTypeTok{grp2n=}\DecValTok{150}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: standardized regression coefficient to effect size Hedges' g
##     Effect Size:   1.9868
##  Standard Error:   0.1569
##        Variance:   0.0246
##        Lower CI:   1.6793
##        Upper CI:   2.2942
##          Weight:  40.6353
\end{verbatim}

\hypertarget{c}{%
\section{\texorpdfstring{Calculating an Odd's Ratio from \emph{Chi-square}}{Calculating an Odd's Ratio from Chi-square}}\label{c}}

To calculate the \textbf{Odd's Ratio} (or any other kind of effect size measure) from \(\chi^2\) using the \texttt{esc\_chisq} function with the following paramters:

\begin{itemize}
\tightlist
\item
  \texttt{chisq}: The value of Chi-squared (or only \texttt{p})
\item
  \texttt{p}: the chi squared p or phi value (or only \texttt{chisq})
\item
  \texttt{totaln}: total sample size
\item
  \texttt{es.type}: the summary measure (in our case, \texttt{"cox.or"})
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_chisq}\NormalTok{(}\DataTypeTok{chisq=}\FloatTok{9.9}\NormalTok{,}\DataTypeTok{totaln=}\DecValTok{100}\NormalTok{,}\DataTypeTok{es.type=}\StringTok{"cox.or"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: chi-squared-value to effect size Cox odds ratios
##     Effect Size:   2.9858
##  Standard Error:   0.3478
##        Variance:   0.1210
##        Lower CI:   1.5101
##        Upper CI:   5.9036
##          Weight:   8.2667
\end{verbatim}

\hypertarget{d}{%
\section{\texorpdfstring{Calculating Hedges' \emph{g} from a one-way ANOVA}{Calculating Hedges' g from a one-way ANOVA}}\label{d}}

We can also derive the SMD from the \(F\)-value of a \textbf{one-way ANOVA with two groups}. Such ANOVAs can be detected if you look for the \textbf{degrees of freedom} (\(df\)) underneath of \(F\). In a one-way ANOVA with two groups, the degrees of freedom should always start with \(1\) (e.g. \(F_{1,147}=5.31\)). The formula for this transformation looks like this \citep{cohen1992power, rosnow1996computing, rosnow2000contrasts}:

\[d = \sqrt{  F(\frac{n_t+n_c}{n_t n_c})(\frac{n_t+n_c}{n_t+n_c-2})}\]

To calculate \textbf{Hedges' g} from \(F\)-values, we can use the \texttt{esc\_f} function with the following parameters:

\begin{itemize}
\tightlist
\item
  \texttt{f}: \emph{F}-value of the ANOVA
\item
  \texttt{grp1n}: Number of participants in group 1
\item
  \texttt{grp2n}: Number of participants in group 2
\item
  \texttt{totaln}: The total number of participants (if the \emph{n} for each group is not reported)
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_f}\NormalTok{(}\DataTypeTok{f=}\FloatTok{5.04}\NormalTok{,}\DataTypeTok{grp1n =} \DecValTok{519}\NormalTok{,}\DataTypeTok{grp2n =} \DecValTok{528}\NormalTok{,}\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: F-value (one-way-Anova) to effect size Hedges' g
##     Effect Size:   0.1387
##  Standard Error:   0.0619
##        Variance:   0.0038
##        Lower CI:   0.0174
##        Upper CI:   0.2600
##          Weight: 261.1022
\end{verbatim}

\hypertarget{e}{%
\section{\texorpdfstring{Calculating Hedges' \emph{g} from the Mean and SE}{Calculating Hedges' g from the Mean and SE}}\label{e}}

When calculating \textbf{Hedges' g} from the \textbf{Mean} and \textbf{Standard Error}, we simply make use of the fact that the Standard error is not much more than the \textbf{Standard Deviation} when the sample size is taken into account \citep{thalheimer2002calculate}:

\[SD = SE\sqrt{n_c}\]

We can calculate \textbf{Hedges' g} using the \texttt{esc\_mean} function with the following parameters:

\begin{itemize}
\tightlist
\item
  \texttt{grp1m}: The mean of the first group.
\item
  \texttt{grp1se}: The standard error of the first group.
\item
  \texttt{grp1n}: The sample size of the first group.
\item
  \texttt{grp2m}: The mean of the second group.
\item
  \texttt{grp2se}: The standard error of the second group.
\item
  \texttt{grp2n}: The sample size of the second group.
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_mean_se}\NormalTok{(}\DataTypeTok{grp1m =} \FloatTok{8.5}\NormalTok{, }\DataTypeTok{grp1se =} \FloatTok{1.5}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{50}\NormalTok{,}
  \DataTypeTok{grp2m =} \DecValTok{11}\NormalTok{, }\DataTypeTok{grp2se =} \FloatTok{1.8}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{60}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: mean and se to effect size Hedges' g
##     Effect Size:  -0.1998
##  Standard Error:   0.1920
##        Variance:   0.0369
##        Lower CI:  -0.5760
##        Upper CI:   0.1765
##          Weight:  27.1366
\end{verbatim}

\hypertarget{f}{%
\section{\texorpdfstring{Calculating Hedges' \emph{g} from a correlation}{Calculating Hedges' g from a correlation}}\label{f}}

For \textbf{equally sized groups} (\(n_1=n_2\)), we can use the following formula to derive the SMD from the pointbiserial \textbf{correlation} \citep{rosenthal1984meta}.

\[r_{pb} = \frac{d}{\sqrt{d^2+4}}\]
And this formula for \textbf{unequally sized groups} \citep{aaron1998equating}:

\[r_{pb} = \frac{d}{\sqrt{d^2+  \frac{(N^2-2 \times N)}{n_1 n_2} }}\]
To convert \(r_{pb}\) to \textbf{Hedges' g}, we can use the \texttt{esc\_rpb} function with the following parameters:

\begin{itemize}
\tightlist
\item
  \texttt{r}: The \emph{r}-value. Either \emph{r} or its \emph{p}-value must be given.
\item
  \texttt{p}: The \emph{p}-value of the correlation. Either \emph{r} or its \emph{p}-value must be given.
\item
  \texttt{grp1n}: The sample size of group 1.
\item
  \texttt{grp2n}: The sample size of group 2.
\item
  \texttt{totaln}: Total sample size, if \texttt{grp1n} and \texttt{grp2n} are not given.
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_rpb}\NormalTok{(}\DataTypeTok{r =} \FloatTok{0.25}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{99}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{120}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: point-biserial r to effect size Hedges' g
##     Effect Size:   0.5170
##  Standard Error:   0.1380
##        Variance:   0.0190
##        Lower CI:   0.2465
##        Upper CI:   0.7875
##          Weight:  52.4967
\end{verbatim}

\hypertarget{g}{%
\section{\texorpdfstring{Calculating Hedges' \emph{g} from an independent t-test}{Calculating Hedges' g from an independent t-test}}\label{g}}

The SMD can also be derived from an \textbf{independent t-test value} with the following formula \citep{thalheimer2002calculate}:

\[d = \frac {t(n_1+n_2)}{\sqrt{(n_1+n_2-2)(n_1n_2)}}\]

We can calculate \textbf{Hedges' g} from a \textbf{t-test} using the \texttt{esc\_t} function with the following paramters:

\begin{itemize}
\tightlist
\item
  \texttt{t}: The t-value of the t-test. Either \emph{t} or its \emph{p}-value must be given.
\item
  \texttt{p}: The \emph{p}-value of the t-test. Either \emph{t} or its \emph{p}-value must be given.
\item
  \texttt{grp1n}: The sample size of group 1.
\item
  \texttt{grp2n}: The sample size of group 2.
\item
  \texttt{totaln}: Total sample size, if \texttt{grp1n} and \texttt{grp2n} are not given.
\item
  \texttt{es.type}: the \textbf{effect measure} we want to calculate. In our case this is \texttt{"g"}. But we could also calculate Cohen's \emph{d} using \texttt{"d"}.
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{esc_t}\NormalTok{(}\DataTypeTok{t =} \FloatTok{3.3}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{100}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{150}\NormalTok{,}\DataTypeTok{es.type=}\StringTok{"g"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Effect Size Calculation for Meta Analysis
## 
##      Conversion: t-value to effect size Hedges' g
##     Effect Size:   0.4247
##  Standard Error:   0.1305
##        Variance:   0.0170
##        Lower CI:   0.1690
##        Upper CI:   0.6805
##          Weight:  58.7211
\end{verbatim}

\hypertarget{h}{%
\section{\texorpdfstring{Calculating Hedges' \emph{g} from Cohen's \emph{d}}{Calculating Hedges' g from Cohen's d}}\label{h}}

We can also directly correct \textbf{Cohen's \emph{d} } and thus generate \textbf{Hedges' g} using the formula by Hedges and Olkin \citep{hedges1985statistical}:

\[g \simeq d\times(1-\frac{3}{4(n_1+n_2)-9}) \]
This can be done in R using the \texttt{hedges\_g} function with the following parameters:

\begin{itemize}
\tightlist
\item
  \texttt{d}: The value of \textbf{Cohen's d}
\item
  \texttt{totaln}: the total \emph{N} in the study
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{hedges_g}\NormalTok{(}\DataTypeTok{d =} \FloatTok{0.75}\NormalTok{, }\DataTypeTok{totaln =} \DecValTok{50}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.7382199
\end{verbatim}

\hypertarget{i}{%
\section{Calculating effect sizes for studies with multiple comparisons}\label{i}}

\includegraphics{stack.jpg}

Many randomized-controlled trials do not only include a single \textbf{intervention} and \textbf{control group}, but compare the effect of \textbf{two or more interventions} to a control group. It might be tempting in such a scenario to \textbf{simply include all the comparisons between the intervention groups and control within a study into one meta-analysis}. Yet, researchers should abstain from this practice, as this would mean that the control group is used twice for the meta-analysis, thus \textbf{``double-counting''} the participants in the control group. This results in a \textbf{unit-of-analysis} error, as the effect size are correlated, and thus not independent, but are treated as if they would stem from independent samples.

\textbf{There are two ways to deal with this:}

\begin{itemize}
\tightlist
\item
  Splitting the N of the control group: One method to control for the unit-of-analysis error to some extent would be to \textbf{split} the number of participants in the control group between the two intervention groups. So, if your control group has \(N=50\) participants, you could divide the control group into two control groups with he same mean and standard deviation, and \(N=25\) participants each. After this preparation step, you could calculate the effect sizes for each intervention arm. As this procedure only partially removes the unit of analysis error, it is not generally recommended. A big plus of this procedure, however, is that it makes \protect\hyperlink{heterogeneity}{\textbf{investigations of hetereogeneity}} between study arms possible.
\item
  Another option would be to \textbf{synthesize the results of the intervention arms} to obtain one single comparison to the control group. Despite its practical limitations (sometimes, this would mean synthesizing the results from extremely different types of interventions), this procedure does get rid of the unit-of-analysis error problem, and is thus recommended from a statistical standpoint. The following calculations will deal with this option.
\end{itemize}

To synthesize the \textbf{pooled effect size data} (pooled Mean, Standard Deviation and N), we have to use the following formula:

\[N_{pooled}=N_1+N_2\]

\[M_{pooled}=\frac{N_1M_1+N_2M_2}{N_1+N_2}\]

\[SD_{pooled} = \sqrt{\frac{(N_1-1)SD^{2}_{1}+ (N_2-1)SD^{2}_{2}+\frac{N_1N_2}{N_1+N_2}(M^{2}_1+M^{2}_2-2M_1M_2)} {N_1+N_2-1}}\]

As these formulae are quite lengthy, we prepared the function \texttt{pool.groups} for you, which does the pooling for you automatically. Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{pool.groups<-}\ControlFlowTok{function}\NormalTok{(n1,n2,m1,m2,sd1,sd2)\{}

\NormalTok{n1 <-}\StringTok{ }\NormalTok{n1}
\NormalTok{n2 <-}\StringTok{ }\NormalTok{n2}
\NormalTok{m1 <-}\StringTok{ }\NormalTok{m1}
\NormalTok{m2 <-}\StringTok{ }\NormalTok{m2}
\NormalTok{sd1 <-}\StringTok{ }\NormalTok{sd1}
\NormalTok{sd2 <-}\StringTok{ }\NormalTok{sd2}

\NormalTok{Npooled <-}\StringTok{ }\NormalTok{n1}\OperatorTok{+}\NormalTok{n2}
\NormalTok{Mpooled <-}\StringTok{ }\NormalTok{(n1}\OperatorTok{*}\NormalTok{m1}\OperatorTok{+}\NormalTok{n2}\OperatorTok{*}\NormalTok{m2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)}
\NormalTok{SDpooled <-}\StringTok{ }\KeywordTok{sqrt}\NormalTok{(((n1}\DecValTok{-1}\NormalTok{)}\OperatorTok{*}\NormalTok{sd1}\OperatorTok{^}\DecValTok{2}\OperatorTok{+}\NormalTok{(n2}\DecValTok{-1}\NormalTok{)}\OperatorTok{*}\NormalTok{sd2}\OperatorTok{^}\DecValTok{2}\OperatorTok{+}\NormalTok{(((n1}\OperatorTok{*}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2))}\OperatorTok{*}\NormalTok{(m1}\OperatorTok{^}\DecValTok{2}\OperatorTok{+}\NormalTok{m2}\OperatorTok{^}\DecValTok{2-2}\OperatorTok{*}\NormalTok{m1}\OperatorTok{*}\NormalTok{m2))     )}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2}\DecValTok{-1}\NormalTok{))}

\KeywordTok{return}\NormalTok{(}\KeywordTok{data.frame}\NormalTok{(Mpooled,SDpooled,Npooled))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\textbf{To use this function, we have to specifiy the following parameters:}

\begin{itemize}
\tightlist
\item
  \texttt{n1}: The N in the first group
\item
  \texttt{n2}: The N in the second group
\item
  \texttt{m1}: The Mean of the first group
\item
  \texttt{m2}: The Mean of the second group
\item
  \texttt{sd1}: The Standard Deviation of the first group
\item
  \texttt{sd2}: The Standard Deviation of the second grop
\end{itemize}

\textbf{Here's an example}

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{pool.groups}\NormalTok{(}\DataTypeTok{n1=}\DecValTok{50}\NormalTok{,}
            \DataTypeTok{n2=}\DecValTok{50}\NormalTok{,}
            \DataTypeTok{m1=}\FloatTok{3.5}\NormalTok{,}
            \DataTypeTok{m2=}\DecValTok{4}\NormalTok{,}
            \DataTypeTok{sd1=}\DecValTok{3}\NormalTok{,}
            \DataTypeTok{sd2=}\FloatTok{3.8}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   Mpooled SDpooled Npooled
## 1    3.75 3.415369     100
\end{verbatim}

\begin{rmdinfo}
\textbf{What should i do when an study has more than two intervention
groups}

If a study has more than one two intervention groups you want to
synthesize (e.g.~four arms, with three distinct intervention arms), you
can \textbf{pool the effect size data for the first two interventions},
and then \textbf{synthesize the pooled data you calculated with the data
from the third group}.

This is fairly straightforward if you save the output from
\texttt{pool.groups} as an object, and then use the \texttt{\$}
operator:
\end{rmdinfo}

First, pool the \textbf{first} and \textbf{second intervention group}. I will save the output as \texttt{res}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{res<-}\KeywordTok{pool.groups}\NormalTok{(}\DataTypeTok{n1 =} \DecValTok{50}\NormalTok{,}
            \DataTypeTok{n2 =} \DecValTok{50}\NormalTok{,}
            \DataTypeTok{m1 =} \FloatTok{3.5}\NormalTok{,}
            \DataTypeTok{m2 =} \DecValTok{4}\NormalTok{,}
            \DataTypeTok{sd1 =} \DecValTok{3}\NormalTok{,}
            \DataTypeTok{sd2 =} \FloatTok{3.8}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Then, use the pooled data saved in \texttt{res} and \textbf{pool it with the data from the third group}, using the \texttt{\$} operator to access the different values saved in \texttt{res}.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{pool.groups}\NormalTok{(}\DataTypeTok{n1 =}\NormalTok{ res}\OperatorTok{$}\NormalTok{Npooled,}
            \DataTypeTok{n2 =} \DecValTok{60}\NormalTok{,}
            \DataTypeTok{m1 =}\NormalTok{ res}\OperatorTok{$}\NormalTok{Mpooled,}
            \DataTypeTok{m2 =} \FloatTok{4.1}\NormalTok{,}
            \DataTypeTok{sd1=}\NormalTok{res}\OperatorTok{$}\NormalTok{SDpooled,}
            \DataTypeTok{sd2 =} \FloatTok{3.8}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   Mpooled SDpooled Npooled
## 1 3.88125 3.556696     160
\end{verbatim}

\hypertarget{power-analysis}{%
\chapter{Power Analysis}\label{power-analysis}}

\includegraphics{poweranalysis.jpg}

A big asset (and probably one of the reasons why meta-analysis can be helpful in practical research) of meta-analyses is that \textbf{they allow for large data to be combined} to attain a more precise pooled effect. \textbf{Lack of statistical power}, however, may still play an important role, even in meta-analysis. This is particularly true for the clinical field, where it is often the case that only \textbf{few studies are available for synthesis}. The median number of included studies in the \emph{Cochrane Database for Systematic Reviews}, for example, is six \citep{borenstein2011}. This is even more grave once we consider that (1) many meta-analysts will also want to perform \textbf{subgroup analyses and meta-regression}, for which even more power is required, and (2) many meta-analyses have \textbf{high heterogeneity}, which reduces our precision, and thus our power.

Power is directly related to the \textbf{Type II error level} (\(\beta\)) we defined: \(Power = 1- \beta\). It is common practice to set our \textbf{Type I error level} (\(\alpha\)) to \(\alpha=0.05\), and thus to assume that the \textbf{Type I error is four times as grave as the Type II error} (i.e., falsely finding an effect while there is no effect in reality is four times as bad as not finding an effect while there is one in reality). The \textbf{Type II error} is therefore set at \(\beta=0.20\), and the power should thus be \(1-\beta=1-0.20=80\%\).

\begin{rmdinfo}
\textbf{What assumptions should i make for my meta-analysis?}

While researchers conducting primary studies can \textbf{plan the size
of their sample based on the effect size they want to find}, the
situation is a little different in meta-analysis, where we can only work
with the published material. However, we have some \textbf{control over
the number of studies we want to include in our meta-analysis} (e.g.,
through more leniently or strictly defined inclusion criteria).
Therefore, we can change our power to some extent by including more or
less studies into the meta-analysis. There are \textbf{four things we
have to make assumptions about when assessing the power of our
meta-analysis a priori}.

\begin{itemize}
\tightlist
\item
  The \textbf{number of included or includable studies} \(k\)
\item
  The \textbf{overall size of the studies we want to include} (are the
  studies in the field rather small or large?)
\item
  The \textbf{effect size we want to determine}. This is particularly
  important, as we have to make assumptions about how \textbf{big an
  effect size has to be to still be clinically meaningful}. One study
  calculated that for interventions against depression, even effects as
  small as \(SMD=0.24\) may still be meaningful for patients
  {[}@cuijpers2014threshold{]}. If we want to study \textbf{negative
  effects of an intervention} (e.g., death or symptom deterioration),
  even \textbf{very small effect sizes are extremely important and
  should be detected}.
\item
  The \textbf{heterogeneity} of our studies' effect sizes, as this also
  affects the precision of our meta-analysis, and thus its potential to
  find significant effects.
\end{itemize}

Besides these parameters, it is also important to think about other
analyses, such as the \textbf{subgroup analyses} we want to conduct. How
many studies are there for each subgroup, and what effects do we want to
find in the subgroups? This is particularly important if we
\textbf{hypothesize that an intervention is not effective in a subgroup
of patients}, because we do not want to falsely find a treatment to be
ineffective simply because the power was insufficient.
\end{rmdinfo}

\begin{rmdachtung}
\textbf{Post-hoc power tests: the abuse of power}

Please note that power analyses should always be conducted \textbf{a
priori}, meaning \emph{before} you perform the meta-analysis.

Power analyses conducted \emph{after} an analysis (``post hoc'') are
fundamentally flawed {[}@hoenig2001abuse{]}, as they suffer from the
so-called \textbf{``power approach paradox''}, in which an analysis
yielding no significant effect is thought to show more evidence that the
null hypothesis is true when the p-value is smaller, since then, the
power to detect a true effect would be higher.
\end{rmdachtung}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{fixed.power}{%
\section{Fixed-Effect Model}\label{fixed.power}}

To determine the \textbf{power} of a meta-analysis \textbf{under the fixed-effect model}, we have to assume the \textbf{true value of a distribution when the alternative hypothesis is correct} (i.e., when there is an effect). For power analysis in a conventional study, this distribution is \(Z\). Follwing Borenstein et al. \citep{borenstein2011}, we will call the true value \(\lambda\) here to make clear that we are dealing with a meta-analysis, and not a primary study. \(\lambda\) is defined as:

\[\lambda=\frac{\delta}{\sqrt{V_{\delta}}}\]

Where \(\delta\) is the \textbf{true effect size} and \(V_{\delta}\) its variance.

\(V_{\delta}\) can be calculated for meta-analysis using the fixed-effect model with this formula:

\[V_{\delta}=\frac{\frac{n_1+n_2}{n_1xn_2}+\frac{d^2}{2(n_1+n_2)}}{k}\]

Where \(k\) are all the included studies, and \(n_1\) and \(n_2\) are the \textbf{average sample sizes in each trial arm we assume across our studies}.

Assuming a normal distribution and using \(\lambda\), we can calculate the Power:

\[Power = 1- \beta\]
\[Power = 1- \Phi(c_{\alpha}-\lambda)+\Phi(-c_{\alpha}-\lambda) \]

Where \(c_{\alpha}\) is the critical value of a \(Z\)-distribution. \(\Phi\) is the \textbf{standard normal density function}, which we we need to calcuate the power using this equation:
\[\Phi(Z)=\frac{1}{\sqrt {2\pi}}e^{-\frac{Z^2}{2}}\]

Luckily, you don't have too think about these statistical details too much, as we have prepared a \textbf{function} for you with which you can easily conduct a \textbf{power analysis} using the fixed-effect model yourself. The function is called \texttt{power.analysis.function} and its code can be found below.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{power.analysis.fixed<-}\ControlFlowTok{function}\NormalTok{(d,k,n1,n2,p)\{}

\NormalTok{  n1<-n1}
\NormalTok{  n2<-n2}
\NormalTok{  d<-d}
\NormalTok{  k<-k}
\NormalTok{  p<-p}
\NormalTok{  title<-}\KeywordTok{c}\NormalTok{(}\StringTok{"Power for a fixed-effect meta-analysis:"}\NormalTok{)}

\NormalTok{  v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))}
\NormalTok{  v.m<-v.d}\OperatorTok{/}\NormalTok{k}
\NormalTok{  lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))}
\NormalTok{  plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)}
\NormalTok{  zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)}
\NormalTok{  power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))}
  \KeywordTok{return}\NormalTok{(power)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\textbf{For this function, we have to specify the following parameters:}

Parameter

Description

d

The effect size we want to be able to detect

k

The number of studies we will likely be able to include into our analysis

n1

The average number of participats we assume in the intervention arms of our included studies

n2

The average number of participats we assume in the control arms of our included studies

p

The Type I error rate (p-level). It is common to use `p=0.05'

Now, let's give an example. I assume that an effect of \(d=0.30\) is likely and meaningful for the field of my meta-analysis. I also assume that on average, the studies in my analysis will be rather small, with 25 participants in each trial arm, and that there will be 10 studies in my analysis. I will set the \(\alpha\)-level to 0.05, as is convention.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{power.analysis.fixed}\NormalTok{(}\DataTypeTok{d=}\FloatTok{0.30}\NormalTok{,}\DataTypeTok{k=}\DecValTok{10}\NormalTok{,}\DataTypeTok{n1=}\DecValTok{25}\NormalTok{,}\DataTypeTok{n2=}\DecValTok{25}\NormalTok{,}\DataTypeTok{p=}\FloatTok{0.05}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The output of the function is:

\begin{verbatim}
## [1] 0.9155008
\end{verbatim}

Meaning that my power is 91\%. This is more than the desired 80\%, so given that my assumptions are remotely true, my meta-analysis will have \textbf{sufficient power using the fixed-effect model to detect a clinically relevant effect if it exists}.

So, if i assume an effect of \(d = 0.30\) in this example, i am lucky. If we play around with the effect size a little, however, while holding the other paramters constant, this can look very different.

\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-247-1} \end{center}

As you can see from this plot, sufficient power (see the \textbf{dashed line}) is soon reached for \(d=0.30\), even if only few studies are included. If i assume a smaller effect size of \(d=0.10\), however, \textbf{even 50 studies will not be sufficient to find a true effect}.

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{random-effects-model}{%
\section{Random-Effects Model}\label{random-effects-model}}

For power analyses under the \textbf{random-effects model}, the formula to calculate the variance of my true mean effect looks slightly different:

\[V_{\delta}^*=\frac{V_Y+\tau^2}{k}\]

We see that again, \(tau^2\) has to be included to take the \textbf{between-study heterogeneity into account} (see \protect\hyperlink{random}{Chapter 4.2} for more details). However, i \textbf{do not know the between-study heterogeneity of my analysis} before i perform it, so what value should i assume?

According to Hedges and Pigott \citep{hedges2004power}, the follwing formulae may be used to calculate the power in the random-effect model assuming \textbf{small, moderate or high heterogeneity}:

\textbf{Small heterogeneity:}

\[V_{\delta}^*=1.33\times\frac{V_Y}{k}\]

\textbf{Moderate heterogeneity:}

\[V_{\delta}^*=1.67\times\frac{V_Y}{k}\]

\textbf{Large heterogeneity:}

\[V_{\delta}^*=2\times\frac{V_Y}{k}\]

Again, you don't have to worry about the statistical details here. We have put the entire calculations into the \texttt{power.analysis.random} function, which can be found below.

Again, R doesn't know this function yet, so we have to let R learn it by \textbf{copying and pasting} the code underneath \textbf{in its entirety} into the \textbf{console} on the bottom left pane of RStudio, and then hit \textbf{Enter ⏎}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{power.analysis.random<-}\ControlFlowTok{function}\NormalTok{(d,k,n1,n2,p,heterogeneity)\{}

\NormalTok{  n1<-n1}
\NormalTok{  n2<-n2}
\NormalTok{  d<-d}
\NormalTok{  k<-k}
\NormalTok{  p<-p}
\NormalTok{  heterogeneity<-heterogeneity}
  
  \ControlFlowTok{if}\NormalTok{(heterogeneity}\OperatorTok{==}\StringTok{"low"}\NormalTok{)\{}

\NormalTok{  v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))}
\NormalTok{  v.m<-v.d}\OperatorTok{/}\NormalTok{k}
\NormalTok{  v.m<-}\FloatTok{1.33}\OperatorTok{*}\NormalTok{v.m}
\NormalTok{  lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))}
\NormalTok{  plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)}
\NormalTok{  zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)}
\NormalTok{  power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))}
  \KeywordTok{return}\NormalTok{(power)}
\NormalTok{  \}}
  
  \ControlFlowTok{if}\NormalTok{(heterogeneity}\OperatorTok{==}\StringTok{"moderate"}\NormalTok{)\{}
    
\NormalTok{      v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))}
\NormalTok{  v.m<-v.d}\OperatorTok{/}\NormalTok{k}
\NormalTok{  v.m<-}\FloatTok{1.67}\OperatorTok{*}\NormalTok{v.m}
\NormalTok{  lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))}
\NormalTok{  plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)}
\NormalTok{  zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)}
\NormalTok{  power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))}
  \KeywordTok{return}\NormalTok{(power)}
\NormalTok{  \}}
    
    \ControlFlowTok{if}\NormalTok{(heterogeneity}\OperatorTok{==}\StringTok{"high"}\NormalTok{)\{}
    
\NormalTok{      v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))}
\NormalTok{  v.m<-v.d}\OperatorTok{/}\NormalTok{k}
\NormalTok{  v.m<-}\DecValTok{2}\OperatorTok{*}\NormalTok{v.m}
\NormalTok{  lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))}
\NormalTok{  plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)}
\NormalTok{  zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)}
\NormalTok{  power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))}
  \KeywordTok{return}\NormalTok{(power)}
\NormalTok{  \}}
    
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Now you are set and ready to use the function. I will assume the same parameters from \protect\hyperlink{fixed.power}{before}, but this time i will also have to specify the \texttt{heterogeneity} in the function, which can take the values \texttt{"low"}, \texttt{"moderate"} and \texttt{"high"}. I will choose \texttt{"moderate"} for this example.

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{power.analysis.random}\NormalTok{(}\DataTypeTok{d=}\FloatTok{0.30}\NormalTok{,}\DataTypeTok{k=}\DecValTok{10}\NormalTok{,}\DataTypeTok{n1=}\DecValTok{25}\NormalTok{,}\DataTypeTok{n2=}\DecValTok{25}\NormalTok{,}\DataTypeTok{p=}\FloatTok{0.05}\NormalTok{,}
                      \DataTypeTok{heterogeneity =} \StringTok{"moderate"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The output i get is:

\begin{verbatim}
## [1] 0.7327163
\end{verbatim}

Interestingly, we see that this value is 73\%, which is smaller than the value of 91\% which was calculated using the \textbf{fixed-effect model}. The value is also below 80\%, meaning that i would not have optimal power to find the desired effect of \(d=0.30\) to be statistically significant if it exists.

This has to do with the \textbf{larger heterogeneity} i assume in this simulation, which decreases the precision of my effect size estimate, and thus increases my need for statistical power.

\textbf{The graph below visualizes this relationship:}

\begin{figure}

{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-251-1} 

}

\caption{Power in the random-effects-model. Darker colors indicate higher heterogeneity}\label{fig:unnamed-chunk-251}
\end{figure}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{power-calculator-tool}{%
\section{Power Calculator Tool}\label{power-calculator-tool}}

If you're feeling lazy, or if you want to quickly \textbf{check for the power of your meta-analysis under varying assumptions}, you might need a tool which makes it easier for you to calculate the power without having to run the R functions we described before each time.

We therefore built a online \textbf{Power Calculator Tool}, which you can find below. The calculations are based on the formulae and functions we described in the previous chapters.

\href{https://mathiasharrer.shinyapps.io/power_calculator_meta_analysis/}{View in full page mode}

\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center}

\hypertarget{r-markdown}{%
\section{R Markdown}\label{r-markdown}}

This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see \url{http://rmarkdown.rstudio.com}.

When you click the \textbf{Knit} button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{summary}\NormalTok{(cars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      speed           dist       
##  Min.   : 4.0   Min.   :  2.00  
##  1st Qu.:12.0   1st Qu.: 26.00  
##  Median :15.0   Median : 36.00  
##  Mean   :15.4   Mean   : 42.98  
##  3rd Qu.:19.0   3rd Qu.: 56.00  
##  Max.   :25.0   Max.   :120.00
\end{verbatim}

\hypertarget{including-plots}{%
\section{Including Plots}\label{including-plots}}

You can also embed plots, for example:

\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/pressure-1.pdf}

Note that the \texttt{echo\ =\ FALSE} parameter was added to the code chunk to prevent printing of the R code that generated the plot.

\bibliography{book.bib,packages.bib}


\end{document}