1. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Resolving the Goldilocks problem: Presenting results Jane E. Miller, PhD

2. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Overview Labeling of variables Identifying model specifications Contents of descriptive statistics tables Prose interpretation of multivariate coefficients – In the results section – In the discussion section

3. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Guidelines for effective labeling: Conveying levels of measurement, units and categories

4. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Basic attributes of variables For every variable in your analysis, convey – Units of measurement System of measurement (e.g., British or metric?) Scale (e.g., millimeters or meters?) Level of aggregation (e.g., weekly or annual income?) – Names of categories, if nominal or ordinal Label them accordingly in – Text – Tables of univariate, bivariate, and multivariate statistics – Charts

5. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Common errors in labeling units Calling a proportion a percentage (or vice versa) – Check axis scales that are automatically generated by Excel or other graphing programs. If your data are stored as proportions, that is how they will be graphed. When you add labels, make them match the actual units. Forgetting to specify the level of aggregation – E.g., deaths per 1,000 persons is not the same as deaths per 100,000.

6. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Tables of multivariate statistics To provide the information needed to correctly interpret βs for each variable in the multivariate model, carefully label – level of aggregation – units – categories (ranges) Labels and titles should reflect whatever version of the variable is included in the specification. – Transformed or original

7. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Units for transformed variables Label the units in multivariate tables, charts, and prose as they appear in the model. E.g., – If you have transformed 1+ continuous independent or dependent variables before specifying the model. E.g., Changed the scale Taken logs Changed the level of aggregation Created a categorical version of a continuous variable

8. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Conveying model specifications in multivariate tables and charts Clearly name types of model specification in tables and charts. – Mention type of model in the title. E.g., “Standardized coefficients from a model of...” – Or label columns or axes to identify: Standardized coefficients Logged dependent variables – Identify logged independent variables in their respective row labels.

9. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Contents of tables and charts

10. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Background information for contrasts Anticipate the metrics and contrasts you will use as you write about your βs. E.g., – Interquartile range – Multiples of standard deviations – Percentiles in a reference distribution Report the corresponding statistics in tables about your own data. Cite a source for complex reference data. E.g., Federal Poverty Levels for different sizes and age compositions of household.

11. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Tables of descriptive statistics To provide readers with the basis of the numeric contrasts in the associated prose: If your models include logged versions of variable(s), report descriptive statistics in both – The original, untransformed units – The logged units If you use empirically based contrasts to interpret βs, include those values in the table of descriptive statistics. E.g., – Interquartile range – Standard deviation and mean

12. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting multivariate coefficients

13. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Presenting results to minimize Goldilocks problems Explicitly identify the nature and size of the contrast for each independent variable as you interpret its estimated coefficient. – Continuous or categorical? – Units or categories being compared? – Size of numeric contrast applied to each β? – Units of the effect measure?

14. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Common pitfalls in reporting of multivariate coefficients Simply reporting βs increases the chances of Goldilocks mistakes by failing to remind readers to consider – variable types – units – range and scale – categories The βs should be reported in your multivariate table. Don’t simply repeat (report) those values without substantive interpretation.

15. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Prose to avert Goldilocks errors in interpretation of coefficients Apply carefully selected “right-sized” contrasts to each β. – Where needed, explain the criteria used to identify fitting numeric contrasts for each of your key variables. E.g., Cite sources of substantively relevant contrasts/ Identify empirical contrasts by name. E.g., – Interquartile range – Standard deviation difference Convey the units or categories of both your IV and DV as you interpret the direction and magnitude of the βs.

16. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Reporting coefficients on nominal variables Name the categories being compared. – Such wording helps avoid implying that More than a 1-unit change in a dummy variable is possible; Directionality of movement across categories pertains to nominal variables. – This distinction is especially important when interpreting βs for both continuous and categorical independent variables.

17. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting β for a nominal variable Poor: “The β for boy was 116.0.” – If reported in a series of coefficients, this version invites readers to compare them directly, without factoring in that only a “1-unit” contrast is possible for gender. Poor, version number2: “Gender is positively associated with birth weight.” – Cannot specify direction of association for a nominal variable. Best: “At birth, boys weigh on average 116 grams more than girls (p < 0.01).” – Concepts, categories, units, direction, size, and statistical significance.

18. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Reporting coefficients on ordinal variables For ordinal variables, including composite scales or indexes that are composed from ordinal variables such as Likert scales. – Name the categories and specify the reference category. – Can write about directionality of the association because the categories are ordered. However, distance between ordinal categories cannot be treated as equal. Numeric codes don’t have mathematical meaning.

19. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting β for an ordinal variable Poor: “Self-rated health and mortality are correlated.” – Fails to convey direction or magnitude. Better: “Among middle-aged men, self-rated health was inversely related to mortality.” – Conveys directionality. – Doesn’t name categories being compared or size of mortality difference between them.

20. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting β for an ordinal variable Best: “Among middle-aged men, self-rated health was inversely related to mortality. Relative risks of mortality were 2.8, 2.2, and 1.9 for those who rated their health ‘poor/fair’, ‘good’, and ‘very good’, when each was compared with ‘excellent’ health (all p < 0.05).” – Concepts, direction, magnitude, and statistical significance. – By naming the categories, conveys what a move up the self-rated scale from one category to the next represents conceptually.

21. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting coefficients on continuous variables: Units Mention the units of your independent and dependent variables as used in the regression. – Units Original units or logged? Unstandardized βs or standardized βs? – Level of aggregation, e.g., Weekly or monthly income? Income in $1s or $1,000s? Individual or family or household income? Might differ from their original form in the data.

22. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting β for a continuous variable Poor: “The β of income on mortality was 0.02.” – By not mentioning the units of either income or mortality: Leaves the size of the β open to misinterpretation. Makes it difficult to compare against βs on the same topic by other authors. Better: “Each additional $10,000 in annual family income was associated with a 2% decrease in the age-standardized mortality rate.”

23. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Interpreting coefficients using contrasts other than a 1-unit increase If you apply a contrast other than a 1-unit increase, report the size of that contrast as you interpret the pertinent β. E.g., – “Each five-year increase in mother’s age is associated with a 53 gram increase in birth weight.” – “Students whose SAT scores were one standard deviation above the mean had 22% higher chances of graduating from college within six year as those with SAT scores at the population mean.” – “Adults at the 25th percentile of BMI were only one-third as likely to die as those at the 75th percentile.”

24. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Goldilocks guidelines for the discussion section Put your results in context in terms of both – Topic – Data Context Specific measures of the concepts under study Reiterate theoretical criteria that identify meaningful contrasts for your topic. E.g., – Clinical cutoffs – Program eligibility thresholds

25. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Statistical significance versus substantive importance Summarize what those contrasts show about your key findings, differentiating between – statistical significance – substantive importance Could mean that once the metrics of the variables were considered, one or more independent variables did not have a substantively meaningful association with the dependent variable, – even if that association was statistically significant in the expected direction.

26. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Differentiating between statistical and substantive significance Use a modifier such as “statistically” or “substantively” before the term “significant” so readers know which type of “significant” you mean. – Doing so might also remind you to discuss BOTH of those aspects of your findings. Rather than vague reference to “substantive significance,” explain that aspect with reference to topic-specific criteria. – See chapter and podcast on substantive significance.

27. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Summary To present βs from multivariate models effectively, need to convey – Units, categories, and descriptive statistics on all variables as they are specified in the model – Type of model specification – The size of the contrast applied to each β as it is Interpreted Compared with other coefficients in the model or in other papers. Close the narrative by putting the size of major findings back in substantive context.

28. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Suggested resources Miller, J. E., 2013. The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition. (“WAMA II”) – Chapter 3, on statistical significance, substantive significance, and causality – Chapter 9, on quantitative comparisons for multivariate models – Chapter 10, on the Goldilocks problem Miller, J. E. and Y. V. Rodgers, 2008. “Economic Importance and Statistical Significance: Guidelines for Communicating Empirical Research.” Feminist Economics 14 (2): 117–49.

29. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Suggested online resources Podcasts on – Statistical significance, substantive significance, and causality – Interpreting coefficients from multivariate models – Defining the Goldilocks problem – Resolving the Goldilocks problem Measurement and variables Model specification

30. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Suggested practice exercises Study guide to The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition. – Problem sets for chapter 3, question #4. chapter 10, questions #5 through 8.

31. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Suggested extensions Study guide to The Chicago Guide to Writing about Multivariate Analysis, 2nd Edition. – Suggested course extensions for chapter 3 – “Reviewing” exercise 3.a.v and 3.b. – “Writing and revising” exercises #2 and 3. chapter 10 – “Reviewing” exercises #1 and 2. – “Applying statistics and writing” question #2. – “Revising” questions #1 through 5, and 9.

32. The Chicago Guide to Writing about Multivariate Analysis, 2 nd edition. Contact information Jane E. Miller, PhD jmiller@ifh.rutgers.edu Online materials available at http://press.uchicago.edu/books/miller/multivariate/index.html