What counts as evidence?

Ron Kennedy as ever asks great questions. Today he asked, what counts as evidence in the world of organisational psychology? My response:

Does this have the potential to be a religious war? Qualitative researchers are not always going to accept the positivist's evidence and vice versa. So prior assumptions about reality seem to play a role. For example: Can anything be objectively measured? Does the presence of the researcher affect the outcome of the investigation? Is the outcome a linear consequence of the cause or does reciprocity play a role? All of these affect the weight of evidence presented and should be acknowledged by the researcher.

From my perspective, what constitutes evidence depends on what is being investigated. Let's say a client wants to know which organisational and psychological attributes drive employee retention in their investigation. And say I am able to build a model whose predictors distinguish between stayers and leavers with 75% accuracy. On the surface, this is remarkable since the probability of achieving 75% accuracy by chance is just 2.81 x 10^-7 (binomial theorem) i.e. close to zero. But before celebrating my evidence, I must ask:

o Was this based on genuine groups of stayers or leavers or did I base it on the tenure of existing employees? If so, my evidence is weakened.

o Does this reflect the existing employee population? Can I generalise it to future populations accounting for the attitudinal shifts of Gen Y & Z etc? If not, evidence is weakened.

o Might my findings be 90% accurate for one business unit, but only 60% accurate for another? If I haven't controlled for this my evidence is weakened.

o Might external events in time affect retention (e.g. economy, M&A, etc.) have affected it? If I can't control for these, my evidence is weakened.

o And so on.

So evidence is only evidence after all factors which might affect it have been weighed and what on the surface appears to be evidence for a large effect size may dwindle down to evidence for not much at all.

On the other hand, if I need evidence for how employees feel about a newly proposed pension scheme and I am not trying to predict anything, I could probably interview a representative sample of sufficient size and come up with reasonable evidence one way or another.

Bottom line: Evidence resulting from predictive models should be treated with extreme caution and all limitations and assumptions noted. Evidence resulting from descriptive investigations is probably quite robust providing samples are representative and of sufficient size.

Original post:

http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=67058084&gid=78865&commentID=49666523&trk=view_disc