Over recent decades, the scientific community has grown increasingly interested in how to integrate information across data sources to promote cumulative research. Combining existing data resources offers many advantages, including the ability to examine the consistency of findings across multiple studies and the opportunity to investigate new research questions that could not have been assessed in any one contributing study alone. For the past fifteen years, Dr. Daniel Bauer has been engaged in collaborative research with fellow UNC faculty members Drs. Andrea Hussong and Patrick Curran to develop new methodological approaches for combining and analyzing data from independently conducted studies, or integrative data analysis. Their research on integrative data analysis was spurred by the goal of combining three longitudinal studies of children of alcoholics, one beginning in early childhood, another beginning in middle childhood, and a third beginning in late adolescence. Combining these three datasets made it possible to examine changes over a much longer span of development than was present in any one study and to determine whether changes were consistent across studies where participant ages overlapped.
A practical challenge for integrative data analysis is how to account for differences in measurement across studies. For instance, each of the three longitudinal studies on children of alcoholics measured internalizing and externalizing behavior, but they did not measure these variables in the same way. One study measured internalizing and externalizing behavior using the Child Behavior Checklist (CBCL), another used the Behavior Problems Index (BPI), and the last study used both the CBCL and BPI. When studies differ in the measurement of key variables it becomes important to ensure that these variables still have the same meaning and metric across studies, otherwise pooling the data risks mixing apples and oranges. To address this problem, Drs. Bauer, Hussong and Curran applied psychometric modeling techniques to empirically evaluate whether different studies are in fact measuring the same things and, if so, to generate “harmonized” scores for these variables (e.g., scores for internalizing and externalizing behaviors) that are on the same scale across studies, enabling data pooling.
To investigate the validity of this psychometric approach for generating equivalent measures across studies, Drs. Bauer, Hussong and Curran recently embarked on a new 5-year grant funded by the National Institutes of Drug Abuse (NIDA), titled Harmonizing substance use and disorder measures to facilitate multistudy analyses. Started in 2013, this project implements a novel twofold approach to methodological validation. First, artificial data simulations are used to determine the conditions under which psychometric approaches can be expected to produce validly harmonized scores that appropriately adjust for superficial differences in measurement across studies. The advantage of generating artificial data is that the true value for any given variable is known for all individuals and the scores generated by psychometric techniques can be compared to those known values. Second, data was collected on college students participating in laboratory analog studies designed to mimic an integrative data analysis. Specifically, students completed measures that were varied to resemble the measurement differences typically observed across studies in an integrative data analysis. Importantly, the same students participated in multiple “studies”, making it possible to see whether psychometric techniques would really generate the same scores (within the limits of measurement reliability) for the same students even under variations in measurement, indicating successful harmonization. The advantage of this validation approach is the ability to evaluate the performance of the methodology in a real world setting. Together, these artificial data simulations and laboratory analog studies provide a rigorous evaluation of the validity of psychometric approaches to measurement harmonization.
Ultimately, the goal of this research is to provide methodological tools that will enable researchers to validly combine data from multiple sources, permitting across-study comparisons and the assessment of novel research hypotheses. More broadly, Dr. Bauer and his collaborators hope to enhance the cumulative character of research in psychology and allied fields.
Dr. Daniel Bauer is a Professor in the Quantitative Psychology Program within the Department of Psychology and Neuroscience at UNC Chapel Hill. He is also the Director of the Quantitative Psychology Program and a faculty member of the Center for Developmental Science at UNC. His research proposes, evaluates, and applies modeling techniques to improve research on the development of negative social and health behaviors and psychopathology. To learn more about his work, visit him online.