The Value of Reproducible Research: Sometimes the response matters more than the results

Yesterday I followed a tweet to a post by Jason Lyall responding to apparently widespread criticism of a new survey in Afghanistan done by the Asia Foundation. The post was the first I’d heard of the survey or of the response to it, so I don’t know anything more about the criticism than what Jason wrote, or much about the nature or arguments of the criticism. But the post did link to one criticism in particular, from Sarah Chayes, a journalist turned NGO-founder and regular ISAF-hired expert on Afghanistan. The general approach taken in her critique seems illustrative of something I find very valuable about systematic and reproducible research and analysis: it facilitates productive and progressive (though perhaps not always intentionally so) responses. Continue reading

Advertisements

Slightly-more-than-basic sentiment analysis

I became interested in sentiment analysis a few months ago as a matter of pure practicality. The company I work for does a lot of customer-satisfaction surveys. Respondents rate various aspects of our products, but they also have the opportunity to answer a bunch of open-ended questions in their own voices. That kind of information can be really useful – it puts a face on the surveys, it can call attention to possibilities we never considered incorporating into standardized questions, and things like that. Continue reading

Surveys, Assumptions, and the Need for Data Collection Alternatives

This is a long post. My previous posts have mostly been about my thoughts on various research subjects. This one reports an actual analysis. If you don’t want to read the whole thing, here are the highlights:

  1. We really need to stop using surveys so much.
  2. If we have to use surveys, it’s probably best to use a three-point scale where it’s clear that the middle point is a neutral option.
  3. If we have to (or really, really want to) use more than a three-point scale, we should probably use an even-numbered scale, preferably no more than a six-point, and make it clear that the top half of the scale choices indicate approval of a particular proposition while the bottom choices indicate disapproval.
  4. We really, really need to stop using surveys so much.
  5. Continue reading