1. PSY 231: Introduction to Industrial and Organizational Psychology
We’ll be talking today about some basic psychometrics and statistics and we’ll end with a brief discussion of ethical issues involved in conducting research. Some of the stuff we cover today will be a little challenging, so I really encourage everyone to read the text and send questions if they do not understand something. I’m going to do my best to keep everything as straight-forward, conceptually speaking, as possible. So, lecture today should give you a real good handle on what’s important and how to absorb some of this stuff.
Before we begin, I have a couple of announcements:
Alex, the T.A. has very generously put together a space on ANGEL which all of you can use to send us (both Alex and I) feedback about the class. You can do so anonymously there if you choose. A word of caution: we won’t be checking this every day, so if you have a question or concern that you feel you need addressed by one of us, you would be better off contacting us via – otherwise you might end up waiting for us to respond to you. That said, you should know that I will take feedback sent through ANGEL to heart and do my best to make adjustments to meet your needs when possible. So, please utilize this tool. If there’s something you want or you’re not happy with, it’s you’re responsibility to let us know about it.
We will be having an in-class assignment today. I will be getting to that later.
On MSN.com yesterday, there was an interesting article – right on the front page, entitled, “Three ways to know your organization is failing”. The article was written by someone with a Marketing background and talked mostly about leadership, which we’ll get into a few weeks down the road. I wanted t bring this to everyone’s attention to highlight the fact that stuff we’ll be learning about in class is really relevant and out there – you can find it all over the place.
Psychometrics, Basic Statistics, & Ethics of Research

2. Some important terms…
Measurement = assignment of numbers to objects or events such as individual attributes.
Attribute = any characteristic of an individual that can be measured and can vary from person to person
Read the definitions above – elaborate – for example: attributes can = anything from eye color to attitudes or beliefs
It’s important to note here the difference between measurement in social sciences like I/O Psychology from that of Physics or Biology. Since clear objective gauges (like rulers or thermometers) for measuring internal attributes of individuals do not exist, we have to be very explicit about the strength and weaknesses of our measurement devices and research techniques. This is why we have psychometrics – characteristics of measurement devices we use – and why it is important for us to discuss them.
Again, the key here is to be aware that not everyone out there does measurement of this sort well. Knowledge of psychometrics can help you make judgments about the quality of some of the measurement devices you read about. So, although I might not be able to keep you on the edge of your seats today with some of this stuff, again, it’s really important.

3. ≈ ≈ ≈ ≈ ≈ Test-Retest Reliability Parallel Forms Reliability
Time 1
Time 2
Test-Retest Reliability ≈
Parallel Forms Reliability
Form A
Form B
Form C ≈ ≈
Reliability
What is it? The consistency or stability of a measure.
There are 3 types:
Test-retest reliability is what is sounds like -
measured by administering a test 1 time and again later - examine the consistency across administrations (do participants answer the items similarly at different times?)
Statistic used to describe this is called the stability coefficient
Parallel forms - similar to the above, but instead of using the same test - use similar test (like form A, form B, etc.)
This is often used to compare paper and pencil Vs. computer exams
Stats used to describe this = coefficient of equivalence
Internal consistency = very common - describes how well items on a test hang together. The image above is a dramatic oversimplification, but the concept is what’s important – you want the items to relate to one-another. It’s particularly important if you’re trying to create a measure of a single attribute, like conscientiousness. You want all the the items to tap into that one construct, and naturally, if they do, they should be related to one another.
There are 2 methods of obtaining a index of this: split-half and interitem – where items from one half are compared to the other OR all items are compared to all other items respectively
Stat = Cronbach’s Alpha (rule of thumb = > .70 is good)
Internal Consistency Reliability
Item 1
Item 2
Item 3 ≈ ≈

4. 3 General Types that we’ll discuss: 1. Content Validity
There are different types of validity and no one way of providing validity evidence is “correct” Instead, researchers should provide whatever and as much evidence of these different types as they can.
What is it? A test’s appropriateness for predicting or drawing inferences about criteria
3 General Types that we’ll discuss:
1. Content Validity
2. Criterion-Related Validity (Predictive and Concurrent)
3. Construct Validity (Convergent and divergent)

5. Construct Validity Convergent Validity Honesty Integrity
Trustworthiness
Integrity
Divergent Validity
Recall earlier we mentioned the definition of attribute - most of these are abstract (not tangible), like self-esteem or intelligence. When measured, these attributes are sometimes referred to as constructs. Constructs might also include attributes of organizations, like climate.
What is construct validity? = extent to which a test measures the underlying construct it was intended to measure
Remember the chart – there are 2 types:
Convergent – the degree to which similar concepts relate to the concept or construct you are interested in. For example, if you are interested in measuring “trustworthiness,” you would expect it to be related to similar concepts like “Honesty” and “Integrity,” so you would want to she that your measure related to these other measures. In other words, you would expect the results of these measures to “converge.”
On the other hand, divergent validity = the degree to which concepts that are dissimilar or unrelated relate to the construct of interest. For example, you probably would not expect that food preference (e.g., preference for spicy food or sweet food etc.) or physical ability (e.g., strength) would NOT be related to trustworthiness. So, measures of these other attributes should NOT be related – the data would be divergent or different from the construct of interest.
Trustworthiness
Food
Preference
Physical
Ability

6. Content Validity  Test of Honesty Honesty Contamination Deficiency 
What is it?
content validity = the degree to which a test covers a representative sample of quality being assessed
Look at the figure – content validity has to do with how well your test captures or covers the construct you are interested in – let’s say, just for this example, that we are interesting in developing a test of “honesty”
There are three terms that define content validity more specifically:
Relevance is the degree of overlap – this is good; you want as much overlap above as possible
Deficiency = the information about the construct you’re missing in your measure (e.g., you forget to include items about trustworthiness or integrity in your measure)
Contamination = the information captured by the test that has nothing to do with your construct (e.g., computer skills if you give you test via computer)
So, another way to understand content validity then, is to think of it as the degree to which your test taps relevant information and is NOT deficient or contaminated.
This is NOT really quantitatively assessed but one could gather this evidence by asking other research experts if the test content is comprehensive and not contaminated, etc.
Deficiency 
Contamination 
Relevance 

7. Time 1: Compare test score and appraisal ratings of current employees
Criterion Validity
Time 2: Performance Rating of the employee on-the-Job
Time 1 Pre-employment Test
Predictive Validity
Job Applicant
Time 1: Compare test score and appraisal ratings of current employees
Criterion validity relates to how well the measure predicts the criterion - Evidence that the GRE predicts college performance = an example of criterion validity
2 types of evidence one might use to demonstrate criterion validity include predictive and concurrent validity - the only difference between these types of evidence is whether the predictor is measured at the same time as the criterion
Predictive validity = the predictor and criterion are measured at 2 different points in time (predictor first)
Example - GREs
Concurrent validity = predictor and criterion measure are collected at the same time
Example - when you want to find out how well a selection test works - might administer it to current employees to see if low scoring employees on your test are also “low” performers.
Concurrent Validity
Current Employee
Appraisal Ratings

8. Analyzing test data – Looking at the Distribution of Test Scores
MEAN (also median & mode in this example)
Standard Deviation
Statistic – An efficient device for summarizing in numbers the values, characteristics, or scores describing a series of cases
Before talking about the stats directly, it’s important to get a handle on the concept of a distribution of scores. Earlier I mentioned the example of giving a test to 5 people and listing the scores they got in ascending order. Well, imagine now that you gave the test to lots and lots of people… And, instead of listing the scores in ascending order, you made a graph with the scores on the X axis and the frequency on the Y axis (I’m going to show you what that looks like in a minute). And then you drew a line connecting all of it… This creates a distribution of scores.
Lots of qualities/characteristics are what we call distributed normally (performance, intelligence, height) - that means that while there’s some people that are really tall or really smart, the majority of people tend to be somewhere in the middle.
Normal distribution…Depicted by bell-shaped curve. Most scores are around the mean with fewer at the extremes of the distribution
Use of Normal Distribution: Calculate percentile score – where person ranks compared to population and other statistics like central tendency & variability
Above is an example of a normal distribution. Notice the mean is in the middle - where the most people scored. The other lines there reflect the standard deviation that we talked about earlier. And, the percentages above reflect how much (proportion wise), under a normal distribution, of the scores are likely to fall in this range.
Sometimes, this curve will be very skinny… not too spread out. That suggests the scores did not vary much and the variance and SD will be small. Or, you might end up with a really fat distribution - if the variance and the SD are large.
What’s important for you to remember about this is that once you’ve created a distribution like this, you can explain where people fall relative others. You might recall in high school or when taking the ACT, you were told that you scored in the 80th percentile or so. What they mean is the proportion of people scoring below you.
Central Tendency:
Mode – Most frequent score in a distribution; Best for categorical data
Median – Score in the middle of a distribution; Best when some numbers are outliers making the distribution skewed
Mean – Arithmetic average of group of scores; Most useful and common measure
Dispersion - Tell how closely scores are grouped around the mean --spreadoutedness
Range – Spread of scores from lowest to highest
Variance – More useful measure of dispersion than the Range
Standard Deviation – Square root of variance, retains original metric of score
The above picture was modified from a website from a professor of education, Dr. Johnson, at the Univ of South Carolina (
Range

9. Stats Representing Mean Differences
T-tests
If you have an experiment with groups of people assigned to different conditions, and you wanted to see whether this group differed from this group on some attribute… You would probably just look at the mean differences between the group… Take the average score from group A and compare that to the average score from group B… If they are different and different enough so as not to be different just by chance, you can usually conclude that the treatment worked.
Ways to test mean differences stem from T-test (2groups) and Analysis of Variance (more than 2 groups). Sometimes you see this explained as a paired comparison. There’s variations on that like MANOVA...
I’m not going to get into these in any more detail than that… The thing I want you to know is that when you see these types of test, what’s happening is that the experimenter is comparing mean differences… And, if you have groups on which you want to compare mean scores, these are the sets of statistics you would first consider.
NEXT - we’ll move into some statistics used for looking at relationships when you might not have explicit groups - remember, I mentioned experimentation is uncomomon, it’s more common in this field that you would deal w/ the next set of stats
Figure above taken from Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: < (version current as of 1/16/05).
ANOVA test

10. Stat Representing Relationships Between Variables: Correlation
What is it? Index of the strength of relationship between two variables (r) - usually done when you have a range of scores to compare on the same individuals and not the mean between separate individuals.
Important points:
Magnitude (0.00 to +/- 1.00)
Direction (negative or positive)
Look at these graphs - here the X axis represents scores on one variable measures (mechanical comprehension) and the Y axis represents scores on another variable (job performance). If you made such a graph and put a dot in the space where all the scores of every person who took the test fell, you’d get a big messy picture of dots.
If they tend to lump together, like in the bottom 2 graphs, you probably have a relationship there. This shows the difference between a positive and negative correlation. If there was no relationship, your graph might look more like this top one.
The tighter the dots clump together and the steepness of the line you can draw through the middle reflects the strength of the relationship while the direction of the line reflects the positive or negative nature of the relationship.
Figures were taken from Levy (2003) textbook entitled “Industrial and Organizational Psychology: Understanding the Workplace”

11. r2 Regression r = .60 r2 = .36 Test Job Performance
(e.g., Math Skills)
Job Performance
R squared, the coefficient of determination, represents the amount of some variable you can explain by knowing something about the other variable...
Figures modified from Levy (2003) textbook entitled “Industrial and Organizational Psychology: Understanding the Workplace”
r = .60
r2 = .36 r2

12. … + + + Meta-Analysis Study 1 results Study 2 results Study 4 results
What is it? = Methodology used to quantify results across multiple studies; often done in conjunction with a literature review
Combine the empirical findings to quantify the relationship between two variables
Not a panacea!!

13. APA code of ethics Accurate advertising of services
Confidentiality of info gathered and reported from research
Rights of human participants
Informed consent
Voluntary
There’s an office on campus (The Office of Research Protections) from which all Penn State researchers have to gain permission to conduct research. If you ever feel uncomfortable with how research was conducted, you can contact that office – find them by searching the Penn State website.
When conducting research in industry, I/O Psychologists sometimes run into some unique ethical situations. For example, if during data collection you find out about employee theft in the organization, and the management sponsoring your research there wants you to disclose who was involved in the theft, you have to be careful. Legally, you are suppose to protect the confidentiality of your subjects, yet you might feel pressure from the organization to do otherwise.
It’s also often difficult for I/O Psychologists to do research in industry well with all of the control we talked about last week. This is because organizations have an agenda: to make money. They are typically only interested in things that will somehow help them be more efficient or research that can solve a particular problem they are currently having. So, one often needs to “sell” his or her research idea in order to get their foot in the door to collect data.
Picture taken from coercion.htm