Maybe we don’t actually know what we think we know

I came across this blog post the other day, talking about the recent spate of bad news coming out of Afghanistan. First there were the sometimes violent protests in response to NATO personnel inadvertently burning a number of Qu’rans taken from the detention facility in Parwan. Then there was the revelation that Staff Sgt. Robert Bales had evidently murdered 16 Afghan villagers, nine of them children. The burnings were unfortunate to say the least and the murders were downright tragic, but I want to focus on the blog post itself, because it illustrates some of what I was trying to say in my previous post on evaluating models. The blogger writes:

I predict whenever Al Qaeda realizes what a propaganda coup has fallen into their laps, news will spread all over Afghanistan much more quickly. So, I’m curious. Why are Americans always wrong? Why isn’t the press showing the civic projects completed in Afghanistan? Why aren’t we reading about new educational buildings and classes in Afghanistan? Why don’t we see statements from President Karzai thanking us for our help? Why don’t we see press reports about running water, electricity, new crops, new equipment, new businesses, new technology, better food, assistance with better living in Afghanistan?

Because it doesn’t sell newspapers, it doesn’t draw readers and viewers to websites and, therefore, it doesn’t make money. All we read about is doom and gloom. Is our world really that jaded that all we are allowed to read is about corruption, killing, and other negative news?

Postscript: This is not a piece against our media in general. This is a damning statement about editors and writers. Any good writer can make a “peaches and cream” story interesting, articles do not all have to be about “gloom and doom”. My charge to you, Mr. Editor (of every news source and website) in the world is to: “bring tears of happiness and joy to some eyes”. Grow a pair.

Now, the blog in question is specifically about Information Operations, which many people consider to be the nice way to say propaganda, so the cynic in me wants to read this post as an attempt at “counter-messaging.” I think the cynic in me is wrong. I worked with a lot of IO guys and I think this post is totally sincere. I had many conversations over the last three years with everyone from policymakers to rank-and-file analysts to Special Forces guys who had just come back from theater or were just heading out to theater (or, more often than not, both), and this kind of concern came up often. ISAF forces spend a lot of time and money doing things that are supposed to help people, but the only things people tend to notice are the mistakes.

The sincerity of the post aside, I think it’s a mistake to boil this down to “bad news sells.” Yes, journalists can make a profitable story out of bad news much easier than they can make a profitable story out of good news. And, yes, I think most journalists really are that jaded. And yes, people in general may pay more attention to bad news than to good. There would be a good reason for that – bad news happens less frequently. It’s out of the ordinary. We tend to pay attention to exceptions, not rules, because our brains have only limited storage capacity and processing power.

But there’s something more: the whole post seems to adhere to the unspoken assumption that the examples of good news not being covered by the media really are examples of good news. I’m not so sure they are.

According to a Congressional Research Service report, the bulk of U.S.-appropriated assistance to Afghanistan has gone to security-related activities such as training of Afghan National Security Forces, but 27.4% of the total aid between 2002 and 2009 went to various social and economic development efforts, and 9.4% more went to humanitarian, democratization, and governance initiatives (Tarnoff 2009). The bulk of non-security aid was spent on construction. According to SIGAR, as much as a quarter of total USAID assistance went to road construction and rehabilitation, and additional funds were spent on infrastructure projects like power plants and gas drilling. The National Solidarity Program and Commanders Emergency Response Program have focused on construction (see here). Even public health, education, and governance aid has largely gone to building stuff like hospitals and schools, government facilities and civil-society venues (see the above CRS report, also see here and here). So we have been building a lot of things, but that doesn’t mean we’ve been helping a lot of people.

Even if we assume that building stuff is the best way to help people, all of the money rolling in has been subject to little-to-no oversight (see, for just a few examples, here, here, here, here, and here). And even if we choose to believe the misdirected and misused funds and shoddy and unasked-for projects are just the rare negative examples picked up by cynical outside observers, the idea that development can win over locals and mitigate conflicts deserves skepticism on logical grounds. For example, if poverty, unemployment, and poor living conditions contribute to insurgency, it’s reasonable to ask why Afghanistan’s insurgency isn’t more successful – practically the entire country suffers from the purported influences.

There hasn’t been much systematic empirical research on this issue (which, as I’ve written, is the general case for most social and political issues). A study of development and security in Iraq found that most development spending didn’t correlate with changes in violence, and the CERP spending specifically correlated with a minor decrease in violence, but that correlation was difficult to separate from a concurrent correlation with large increases in troop numbers. Development assistance was found to exacerbate conflict in the Philippines, at least in the short run. A randomized field experiment involving 500 villages in Afghanistan found that villages that received development tended on average to be 3-6% more positive in their views of the government and NGOs than villages that didn’t get development aid, but the differences were even less pronounced in regard to villagers’ views of security, and the difference in terms of actual security incidents suffered was negligible.

In other words, an unknown number of development activities in Afghanistan are projects that actually directly impact people’s lives. An unknown percentage of those activities is subject to enough oversight to expect the activities to be well implemented. An unknown number of those well-implemented projects have the effect of negligibly improving public opinion and negatively impacting local security. But we don’t know the number of impactful projects, the percentage of well-implemented projects, or the extent to which the impact of any of those projects matches the small amount of empirical research that has been done on the subject to date.

We have moderately good records of how much money has been sent to Afghanistan. We have less-than-moderately-good records of where that money was actually spent. We have lousy records of how that money was spent, particularly when it comes to which projects were actually built. And we have practically no consistent records of the quality or practical effects of those projects.

That’s what makes me connect this to what I said about models. Models are rough estimates of what is supposed to happen in certain situations, and the way to evaluate a model is to ask how often using that model results in unrealistic expectations. The development model says that, as development goes up, public opinion ought to go up and conflict ought to go down. Yes, other things will certainly affect public opinion and conflict – we live in a complicated world – but the models says that there ought to be at least some kind of consistent relationship, and that the relationship should be positive. As far as I can tell, people who employ this model are doing so on the basis of their assumption that it is right, because there doesn’t seem to be much evidence that development counters an insurgency

I tend to think that the sell-more-papers type of journalism more often than not results in oversimplified and out-of-context stories that do little to inform people about world events, so it kind of bothers me that assertions like the ones made in the above blog post make me feel the need to defend that kind of journalism in the process of defending journalists in general. The only situation where it would be fair to challenge news media outlets to “grow a pair” and showcase development successes would be a situation where development projects could really be tied to widespread changes in local behavior. That’s what the Qu’ran burnings and village massacre did – they didn’t just make people do angry things in one place. They made people do angry things in lots of places. If development made people do not-angry things in lots of places, then that would really be a story.


7 thoughts on “Maybe we don’t actually know what we think we know

  1. Very interesting, Schaun. I’m not sure if you’ve been following my blog about my son’s recovery from an IED blast in Afghanistan, but one of my recent posts discussed a visit from his former commanding officer. He told us of his company’s successes, including building a school, a clinic and a police station. He had intelligence that these efforts were not only winning the Afghans’ hearts but disheartening the Pakistani mercenaries fighting for the Taliban. He was also proud of the things his company did on their own, not under orders. My point is twofold: here’s a firsthand account of the positive effects of “building things” and an example of “off the books” good works, for lack of a better term.

    I understand your not wanting to defend the journalists. I wish there were more clear-cut evidence that the good things we’re doing over there are helping.

  2. Sylvia,

    I hadn’t heard about your son – I’m so sorry to learn he was wounded.

    I think I’ll address your second point first. Yes, there are lot of “off the record” activities – successes and otherwise. As I wrote here, in Afghanistan and Iraq in particular there seems to be basic activities – like a number of troops moving into or out of a country – not recorded or reported with any kind of consistency. As I’m getting my feet wet in the private sector, I’m finding loads of other examples where systematic records could fairly easily be kept, but aren’t, and cases where our ideas about what coincides with what could be rigorously measured, but isn’t. (If any readers have access to the “DIME/PMESII, HSCB, and IW” group on LinkeIn, we’ve had a pretty interesting conversation going there, spurred by my post on models, that touches on these missed informational and analytic opportunities in both the private and public sectors.)

    Any analysis conducted at a scale any larger than the on-the-ground individual is going to be missing relevant information. We can and should work to minimize that information loss, but it’s still going to happen. It’s not so much of a problem that we should dismiss findings from larger-scale studies, because what those studies lack in detail about individual occurrences they partially make make up for in detail about consistent, cross-occurrence similarities and differences.

    That brings me back to your first point. My argument was about larger-scale patterns of effects – things that are true across many individual cases. I worked in the military intelligence community for three years and that experience made me skeptical – maybe even a little cynical – about claims that any intelligence can demonstrated a large-scale pattern or trend. To be frank, most intelligence reports I read weren’t very high quality and even those that were good were so focused on one or two specific occurrences that it was impossible to differentiate idiosyncrasies from larger patterns. I don’t doubt that development projects in Afghanistan and other locations have helped individual people or deterred individual fighters. I think every solider I’ve talked to who was involved in village stability operations in Afghanistan has a story about someone who appreciated the aid he or she received. However, in the context in which I was writing, development projects only have a positive effect if all their individual positive effects outweigh all their individual negative effects, and if the positive outweighs the negative to such an extent that we could expect to see the outcome of the overall conflict altered from what it would have been without the development.

    The models currently being used to justify the use of development in counterinsurgency don’t just say that development helps people, or even that development can sway individual people in their opinions – they say that development helps counterinsurgents beat insurgencies. When I said that there’s no evidence that development counters an insurgency, I was speaking specifically about that large-scale assertion. That necessarily ignores the small scale successes which, however many of them there are, don’t seem (although I would love to be shown otherwise) to accumulate into large-scale successes. I’m afraid, but of course not certain, that the reason we don’t see more examples of our development projects helping is because the aggregate of those individual projects isn’t advancing the objectives of the governments involved in the conflict.

  3. Pingback: On the virtues of deliberate inaction « House of Stones

  4. Excellent observations there, Shaun. I have many friends working with NGOs, in private security and in both the US and Canadian military and I have seen, albeit on skype, the amazing works that USAID and others are doing in Afghanistan.
    On a personal level all of the friends and colleagues I have over there make it a priority to educate the women and children of their fundamental human rights.
    Those who instruct the Afghan Police Force have forged lasting relationships and invested more than their time with the locals.
    We certainly need to hear more good news from Afghanistan because I am convinced that more good than evil is happening there.
    Keep up the great reporting!
    Kind regards,

  5. Thanks for your kind comments Lee, but I was actually arguing that we have no idea how much good or evil are going on there as a result of development efforts – again, as I said to Sylvia, speaking only about the effects of development efforts upon counterinsurgency efforts. I think there’s been a little bit of research questioning the extent to which development efforts actual help people on a basic, day-to-day, non-conflict-related level, but I’m perfectly willing to expect that development could help individuals. I just don’t think it helps a conflict.

  6. Hi Schaun!
    “To Inform is to Influence” is my blog and I agree with your observations and conclusions. Unfortunately I try to keep my blog pieces somewhere around 500 words, occasionally I reach 1,000, the space is limiting.
    I’m an IO guy but I’m also former Special Forces and retired military intelligence, and yes, I was and am sincere.
    Good research, good references, I look forward to reading more!

  7. Pingback: Some helps for running and evaluating Bayesian regression models | House of Stones

Comments are closed.