Leigh, D., & Lewis, N., Jr. (2018, August 7). Parsing Science – Differing Interpretations of Difficulty (Version 1). figshare. https://doi.org/10.6084/m9.figshare.6959309
Neil Lewis, Jr.: It turns out actually they’re not quite opposite ends of the same coin, they turn out to be two different coins.
Doug Leigh: This is Parsing Science: the unpublished story is behind the world’s most compelling science as told by the researchers themselves. I’m Doug Leigh. Ryan’s away on vacation this week but he’ll be back next time. Today in episode 29 of the show we’re joined by Neil Lewis, Jr., a social psychologist from Cornell University. He’ll talk with us about his research into what differentiates students who experience difficulty in college is a sign of the importance of succeeding academically from those that take difficulty in school to mean that successfully completing a college degree is impossible. Here’s Neil Lewis, Jr..
Neil Lewis, Jr.: Well, hi I’m Neil Lewis, Jr.. I’m an assistant professor at Cornell University in the Department of Communication and I’m also in the graduate faculty in Communication and Psychology here at Cornell. I’m broadly interested in trying to understand what are barriers that keep people from achieving their goals in a variety of domains. So done work on savings behavior I’ve done work on health behavior increasingly doing more work and in the climate change sustainability area; do a lot of work in education. But what are the barriers that keep people from succeeding and how can studying those processes help us to potentially develop interventions that can help people achieve their goals. That’s sort of the broad unifying theme of my work.
Doug Leigh: As it’s the main title of his paper we began by asking Neil to explain the basic idea behind the notion of no pain no gain.
Neil Lewis, Jr.: Yeah, when we experienced difficulties in life we often have to come up with an interpretation for what that means right? So if you have this sort of no pain no gain mindset – or in this work we talked about that is interpreting difficulty as importance – then sort of the struggle seems worth it and you’re more likely to persist afterwards. But if it doesn’t seem like it’s worth your time is another interpretation – so difficulty means the possibility – then you might as well quit. And so these are two things that we’ve been looking at largely in education and health kinds of settings but they turn out to predict persistence and performance in these domains.
Doug Leigh: Since “unimportance” and “possibility” aren’t opposites of one another Ryan and I wondered whether “importance” and “impossibility” are. Here’s what Neil had to say about the question.
Neil Lewis, Jr.: It turns out actually they’re not quite opposite ends of the same coin, they turn out to be two different coins. But what I mean by that is the interpretation of difficulty as importance and interpretation of difficulty as impossibility are actually uncorrelated. So they’re orthogonal constructs, and this is something that all of Oliver Fisher has been doing a lot of work on lately sort of figuring out what the separate variance is for these constructs and as well as other motivational constructs and social psych literature.
Doug Leigh: Neil and his team on this project – which included Cristina Aelenei who’s now with Paris Descartes University and Daphna Oyserman at the University of Southern California – carried out two separate studies both of which were related to the degree to which students viewed experiencing difficulty in school as a reminder of the importance of persisting towards attaining their educational goals so we asked him to explain what each of these studies entailed as well as what they found.
Neil Lewis, Jr.: The first study was the large correlational study where we just surveyed a bunch of Americans we had over a thousand people answer this questionnaire and then we looked at demographic correlates of their responses, finding that education was a big factor but there’s also a moderation by racial-ethnic minority status. And then the second, study we looked at that in the community college setting, because part of the backdrop of this work was really trying to understand some some gaps in educational outcomes between community college students and four-year university students. So we know that community college students are less likely to graduate in the university students who are trying to figure out what what are some key differences between community college students and for university students that might help us understand this and thus what we might be able to do about it. So the paper is a mix of [a] large online study and then a second study that we ran in community colleges … in California in Southern California.
Doug Leigh: The framework the team used to conceptualize the two studies is known as identity based motivation theory which was first proposed by Neil’s collaborator Daphna Oyserman. It concerns how motivation engagement and self-concept are affected among people who experience difficulty in attaining the goals they set for themselves. A core prediction of the theory is that it’s not their experience of difficulty per se but rather how that experience is interpreted which matters most. In particular what was of consequence in the context of a team study is whether academic identities – and the strategies to attain them – come to students minds and influence their academic engagement, as Neil explains next.
Neil Lewis, Jr.: The identity base of motivation approach really tries to figure out when in which situations people’s identities will motivate them to take action towards their goals. So we really look at both these individual level characteristics and broader macro level factors at the same time and figure out how those things work together to help us predict how people are gonna behave in different situations. And most of it has this sort of underlying goal of trying to figure out ways to develop interventions to sort of bridge or close gaps between people’s aspirations and what they ultimately achieve. It’s a big framework that guides much of our thinking and the work that we’re doing here. So we run a variety of studies – whether it’s laboratory experiments, field experiments, community-based studies – trying to understand behavior in different contexts but the identity based motivation framework really helps to generate predictions for how we think things might play out how we should then design studies to test different parts of the theory.
Doug Leigh: Neil and his team found that participants with higher levels of education were more likely to interpret difficulty as signalling importance, an effect that was particularly pronounced among racial minorities. They also found that students who tended to agree that difficulty implies importance were more certain about attaining their academic identities, and more willing to sacrifice to do so; an effect which benefitted community college students more than university students. It seemed to Ryan and me that it might be sometimes reasonable for, say, a high school student to think “sure school is important, but it’s impossible for me to do well, so it’s not really that important to me.” We asked Neil his thoughts about this issue.
Neil Lewis, Jr.: In the domains that we’ve done most of our work, and the populations where we do most of our work, that’s often a maladaptive response right. So if you’re thinking about you know middle school kids, high school kids: You need yeah you need to be persistent right. Quitting will not lead to a better life. But surely there are situations where quitting is the is the adaptive response. At this point I should not be trying to be an olympic gymnast. It’s probably not worth my time. I could spend a lot of effort trying to do that, but the odds of success are very low and I probably shouldn’t waste my time doing it. But for a teenager in school, no, I think you should be persisting, given all that we know about the benefits of education for your life outcomes.
Doug Leigh: In addition to the interpretation of experienced difficulty, two other components of identity-based motivation are “connection” and “strategies.” Ryan and I were interested in learning what each of these refers to, and how they relate to forming an identity as a college student.
Neil Lewis, Jr.: Connection is related to this idea of connecting who you are now, and the way you think about yourself now, to who you would like to be in the future. Right? So we have various goals and aspirations for a future and it can be easier to – well not necessarily easier – but, yeah, you can stay motivated to pursue those if there is a clear connection to your current self. So if it’s clear why, you know, you’re going through the pain of this extra problem set right now how that’s going to benefit you in the future then you are more likely to sit down and actually do it. Whereas if there’s no clear sense of why you should be doing this then it’s hard to stay motivated to do that. And that’s of course related to the strategies component. So, are the strategies that are necessary to succeed things that feel congruent with how you think about yourself. Am I the kind of person that – am I a school person, am I a healthy person in the health realm, can I actually do these things, do people like me engage in these behaviors? That again can be quite motivating if things don’t feel identity congruent then you’re less likely to do them.
Doug Leigh: Under identity based motivation theory, people’s identification with their social class and ethnic minority status can influence the way that they experience difficulty as do their levels of education and income. We’ll hear what Neil had to say about how these social stratifications figure into the theory after this short break.
Sponsor: This episode is sponsored by We Share Science. When researchers are curious about what is happening in science they go to We Share Science to explore video abstracts uploaded by other researchers. You can search their vast catalog of video abstracts to learn about the latest scientific findings, or you can share your research with the world. Whether your research is in progress or already published, at We Share Science you can share your science and grow your impact. Explore the world’s research at wesharescience.org. Now back to Parsing Science.
Doug Leigh: Here’s Neil Lewis, Jr.
Neil Lewis, Jr.: Yeah so what we’ve been thinking about is the relationship between these identities and how things play out in society, from lots of years research and sociology economics political science like. Society ends up being stratified along social dimensions. Life can be much harder if you’re on the lower end of the social hierarchy, if you have low income – low levels of education, racial-ethnic minorities status – things like that can really affect your day to day experience. And so given that that happens that should also, we think, influence how you think about yourself inside society and what is possible for you. What strategies come to mind for how you might pursue different goals, how you think about your experiences of difficulty, and the like. And so that’s the way that we’ve been thinking about this in our work.
Doug Leigh: Neil and his team study used a questionnaire to collect data regarding participants experience of difficulty. They also solicited information about the markers of social stratification such as those that he just discussed. So Ryan and I were interested in learning how these factors related the people’s experience of difficulty in college.
Neil Lewis, Jr.: We took the Interpretation of Difficulty Scale and just had over a thousand people fill it out. And we looked at for the correlates of their answers, and we did find these relationships between markers of central stratification and answers. So the amount of education you had mattered for how much you agree that difficulty means importance. Racial-ethnic minority status also interacted with that. So if you have high levels of education you’re more likely to agree that difficulty means importance. Lower levels of education, you’re less likely to agree that difficulty means importance. And that makes sense given what we know about differences in lived experiences. Those of us who were fortunate enough to have higher levels of education are usually operating in spaces where we do see the returns from our efforts. Working harder does usually mean getting more from it. Whereas if you’re in a situation where say you’re working two or three jobs but still not moving up very much it’s hard to agree that difficulty is important and as sort of worthwhile. So that’s sort of the way that we’ve been thinking about these things, trying to figure out how do these social categories, identities play out in the way that you think about yourself and what is possible for you to achieve depending on where you are in society.
Doug Leigh: Neil and his team made use of Amazon Mechanical Turk, or MTurk, to recruit people to complete the projects questionnaires. Since several of our prior guests had also made use of MTurk Ryan and I wanted to know what Neil sees as being the pros and cons of crowdsourcing research participants.
Neil Lewis, Jr.: Part of the reason social scientists love MTurk now, and part of the big reason we shifted so much towards it, was because from a cost perspective it’s very cheap. So you can recruit lots of people for very little money. You know back when we ran these studies 10 cents a minute was considered a fair wage among the MTurk community, now that’s gone up, but last thing I saw was like 15 cents a minute. But it’s, it’s still … Yeah, it’s more, but still in the grand scheme of things very cheap. I mean we’re talking about, if you go to a big survey firm we’re talking you know at least five dollars, so the cost difference is huge. But another thing is we’ve known for a long time that our student participant pools are very limited in diversity. So in the 80s david sears wrote this great paper called “college sophomores in the laboratory,” and it’s paper about how much social psychologists view of human behavior is limited by basically rich largely white 18 to 22 year olds. And so MTurk was a nice alternative in that you could get a broader range of people very cheaply. But it’s also … I mean it’s not it’s more diverse than college samples but it’s not super diverse either. And there’s only so much you can do, right? The things that you can study are things that can be done in an online survey for the most part. And so that’s had some drawbacks. In my mind, in the kinds of questions that get asked – because, I mean, just studying behavior, there [are] very few behaviors you can study in an online context. So we’ve had this radical shift towards MTurk, and that’s just sort of changed the kinds of questions that get asked now. So there’s another good methods paper – I forget what year it came out now – but, it’s a paper about social psychology becoming a science of finger taps and reaction times or something like that. I don’t remember the exact title, but it’s sort of a methodological critique of this shift in our sampling approaches changing what we end up studying, and narrowing the scope of what we study.
Doug Leigh: In addition the team made use of MTurk’s built-in “qualifications” feature to select only a cohort of online participants who hadn’t completed related studies in the past so we wanted to follow up to ask Neil to say a few words about how the process works.
Neil Lewis, Jr.: There are thousands of people there trying to do often lots of as many studies as they can because this is how they are sort of maximizing how much they make, and so you have to be careful about that. The website has a number of cool backend features now that you can use to make sure that, you know, if someone’s participated in studies before you can set up filters to make sure that they aren’t repeat participants. So given that, in our lab, we’ve done a number of studies with these scales on MTurk before, we didn’t want people who had completed them for to complete them again. And so what we did was we have sort of these filters on MTurk if a study contains the scale as soon as studies done you know you loop those that batch of MTurk IDs into that filter so that you can exclude them from future studies that would have the same measures because there might be contamination issues. And so that’s … it has a number of features like that that you can use to try and address some of these concerns of non-naive participants.
Doug Leigh: Neil’s been on Twitter since 2014 and both has a prodigious following and is himself a frequent poster of news about science and academia. Given the growing number of alternatives to traditional publishing, we asked him how he stays current with what’s new in his field.
Neil Lewis, Jr.: There are many times that what we think hasn’t been studied before – or maybe hasn’t been said it in our discipline or corner of the discipline – has actually been studied elsewhere, and I’m hoping now that we have increasingly more integrated databases it’ll be easier to find things. So I suspect part of the reason you all found me is because of Twitter, and this is one of the things they actually like a lot about that medium is [that] I’m constantly exposed to things that are very relevant to me but I would never read otherwise because I basically read the journals in my discipline. There’s too many journals I can’t read them all – I get the ones that are mailed to me once a month and – everything else is this whatever people are tweeting about. And so now there’s, I mean, there’s so many things in sociology and political science and public health public policy and economics that are very relevant to the work that I’m doing and I just wouldn’t know about it if I weren’t following these people on Twitter.
Doug Leigh: We followed up by asking Neil his thoughts on preprints which are early versions of scholarly or scientific papers that do to the rise of open science initiatives are now ubiquitous across the web.
Neil Lewis, Jr.: The pace of knowledge transmission has increased than they think it’s … Yes, there’s a lot of sort of noise as well that can be distracting, but there’s also a lot of really good stuff there. And I now find it funny to get the table of contents from journals in my inbox or the physical journals themselves, and open it and like “I’ve already read all of this it was a preprint you know six months ago.” And so yeah, in some sense, I think some of the journals might not be as relevant as they used to, now that everything gets posted online. Not only the preprints but blog posts, some of them are just really helpful in explaining things very clearly, in a way that actually wouldn’t end up in a journal. So that’s I think a nice feature. So you know I’m getting ready to teach research methods this fall, and at least once a week I think I’ve got a blog post on the syllabus, because I think many of them are quite helpful for synthesizing the broad themes of the week. So you know they have the core journal article readings, but some of these blog posts explain things much better than the articles ever could. And from a teaching perspective, that’s really helpful to have.
Doug Leigh: Lastly, Neil spoke with us about what future applications he hopes that researchers might take on regarding identity based motivation theory.
Neil Lewis, Jr.: An area that many of us are working on is thinking about how do we scale some of the kinds of interventions – both identity-based motivation interventions but other kinds of psychological interventions – that’s something that I do think needs a lot of work. Because from developing something developing an intervention whether it’s with lab studies or even like smaller field studies in one location, to bring to a level that it can be broadly disseminated it actually takes many steps. And I think we often stop at “well, I develop my intervention and I publish it in the journal look how beautiful it is and how it, you know, changes everything; anyone can do it now” or “it’s somebody else’s job to figure out how do you implement it, say in an entire school district.” Well, there are many things that need to happen along the way that we need to figure out what are the key levers at each of those steps. So that’s sort of another area that I think the theory can help, because of its been very intentional in thinking about these macro and micro level interfaces at the same time.
Doug Leigh: That was Neil Lewis, Jr., discussing the article “No pain no gain? Social demographic correlates and identity consequences of interpreting experience difficulty as importance,” which he published with Cristina Aelenei and Daphna Oyserman in the january 2017 issue of contemporary educational psychology. You’ll find a link to their paper on parsingscience.org/e29 along with bonus content and other material that Neil discussed during the episode.
Doug Leigh: Parsing Science just launched a free new service for listeners: A mailing list highlighting the week’s most compelling science news from anthropology to zoology and andragogy to zygotes. Just go to parsingscience.org/newsletter to sign up. Also if you’d like to receive our brand-new Parsing Science sticker, just let us know where to send it and we’ll mail one to the first 50 newsletter subscribers for free.
Doug Leigh: Next time on Parsing Science we’ll talk with Yune Lee from the Ohio State University. He’ll discuss his research into how subtle differences in hearing acuity can affect speech comprehension among young adults.
Yune Lee: The right side of the brain sort of homologous area comes into play to compensate for sort of a cognitive and hearing decline, which we typically see people after age 50. But in my recent study we were surprised to see that actually the right side of the brain is lighting up when they have a slight – like tiny bit – of hearing decline that is so slight that they’re not even aware of that.
Doug Leigh: We hope that you’ll join us again.
Lastly, Neil spoke with us about what future applications he hopes that researchers might take on regarding identity based motivation theory.@rwatkins says:
We followed up by asking Neil his thoughts on preprints which are early versions of scholarly or scientific papers that do to the rise of open science initiatives are now ubiquitous across the web.@rwatkins says:
Neil’s been on Twitter since 2014 and both has a prodigious following and is himself a frequent poster of news about science and academia. Given the growing number of alternatives to traditional publishing, we asked him how he stays current with what’s new in his field.@rwatkins says:
In addition the team made use of MTurk’s built-in “qualifications” feature to select only a cohort of online participants who hadn’t completed related studies in the past so we wanted to follow up to ask Neil to say a few words about how the process works.@rwatkins says:
Neil and his team made use of Amazon Mechanical Turk, or MTurk, to recruit people to complete the projects questionnaires. Since several of our prior guests had also made use of MTurk Ryan and I wanted to know what Neil sees as being the pros and cons of crowdsourcing research participants.@rwatkins says:
Neil and his team study used a questionnaire to collect data regarding participants experience of difficulty. They also solicited information about the markers of social stratification such as those that he just discussed. So Ryan and I were interested in learning how these factors related the people’s experience of difficulty in college.@rwatkins says:
Under identity based motivation theory, people’s identification with their social class and ethnic minority status can influence the way that they experience difficulty as do their levels of education and income. We’ll hear what Neil had to say about how these social stratifications figure into the theory after this short break.@rwatkins says:
In addition to the interpretation of experienced difficulty, two other components of identity-based motivation are “connection” and “strategies.” Ryan and I were interested in learning what each of these refers to, and how they relate to forming an identity as a college student.@rwatkins says:
Neil and his team found that participants with higher levels of education were more likely to interpret difficulty as signalling importance, an effect that was particularly pronounced among racial minorities. They also found that students who tended to agree that difficulty implies importance were more certain about attaining their academic identities, and more willing to sacrifice to do so; an effect which benefitted community college students more than university students. It seemed to Ryan and me that it might be sometimes reasonable for, say, a high school student to think “sure school is important, but it’s impossible for me to do well, so it’s not really that important to me.” We asked Neil his thoughts about this issue.@rwatkins says:
The framework the team used to conceptualize the two studies is known as identity based motivation theory which was first proposed by Neil’s collaborator Daphna Oyserman. It concerns how motivation engagement and self-concept are affected among people who experience difficulty in attaining the goals they set for themselves. A core prediction of the theory is that it’s not their experience of difficulty per se but rather how that experience is interpreted which matters most. In particular what was of consequence in the context of a team study is whether academic identities – and the strategies to attain them – come to students minds and influence their academic engagement, as Neil explains next.@rwatkins says:
Neil and his team on this project – which included Cristina Aelenei who’s now with Paris Descartes University and Daphna Oyserman at the University of Southern California – carried out two separate studies both of which were related to the degree to which students viewed experiencing difficulty in school as a reminder of the importance of persisting towards attaining their educational goals so we asked him to explain what each of these studies entailed as well as what they found.@rwatkins says:
Since “unimportance” and “possibility” aren’t opposites of one another Ryan and I wondered whether “importance” and “impossibility” are. Here’s what Neil had to say about the question.@rwatkins says:
As it’s the main title of his paper we began by asking Neil to explain the basic idea behind the notion of no pain no gain.