A recent LSE lecture by Stanford Profprofessor Walter Powell and supporting paper Interstitial Organizations as Conversational Bridges (Co-authored with Valeska Korff and Achim Oberg) uses text from NGOs webpages to suggest ‘a novel approach to evaluating the impact of non profit organisations’ … ‘combining social network and linguistic analysis’. The basis of the work is to analyse the diffusion of keywords through a network of NGO websites. It’s more exciting that the title makes it sound.
In the broadest terms my research is about digitally facilitated social coordination (hopefully I’ll find a snappier term), so this paper speaks exactly to the topic I’m looking at, as I’m going to attempt to describe here.
To articulate what I mean by social coordination it might be useful to rehearse how the predominant money-based method of social coordination works. A company that makes a good (popular) product will be able to sell that product for money, which they can use to make more of the product, a company with a less desirable product might go out of business when it fails to find people who want to purchase their output. So the flow of money encodes information about what people want – what’s most socially beneficial – and leads to socially optimum outcomes. I’ve described the idea with two companies, but the same idea applies to a whole economy. Obviously, practical concerns might stop it from working quite so neatly, but many economists and politicians think the market mechanism is very effective as a way of allocating resources. I’m calling this process of allocating resources social coordination.
But, non-profits don’t make a product that they can sell for money. How, as a society, should we allocate money to them – how do we solve the social coordination problem for NGOs? My contention is that all of the digital data that has started to suffuse society encodes information about what people want, an alternative flow of information to the monetary one described above. The work described in the paper can be seen as exploring exactly this idea.
As Powell explains in his presentation, organisations like the Gates foundation, or silicon valley rich kids (whose equivalent in the UK are the private equity guys apparently) want to choose the most effective organisations to give their money to. If an NGO’s raison d’etre is to bore wells in rural Africa, then clearly a measurement of success would be how many people end up with clean water, or similar. But many NGOs set themselves the goal of shifting debate or influencing policy. In these less tangible cases, how do we measure how effective an organisation is?
This is where the research comes in. In effect, it measures the spread of ideas. Their initial phase is very similar my method with localnets.org, only applied to websites, rather than Twitter. They took 36 websites they knew would be at the centre of the network they wanted to look at (a ‘seed’), and looked at which other websites the seed sites had hyperlinks to. That gave them 1394 websites to look at – the most highly linked – but they pruned the non-specialist ones (New York Times etc), leaving them with 369.
Next, they grabbed all the text and PDFs on the 369 websites, and looked for the frequency of keywords. They had a list of 105 keywords divided into 3 categories – Science, Management, and Civil Society. So a website that frequently contains the words such as ‘participation’ and ‘justice’ would be strongly about civil society. A website that uses the words ‘outcomes’ and ‘performance’ a lot would be classified using a lot of management style language, while ‘data’ and ‘survey’ are examples of science words. Websites use all three languages are ‘in the middle’.
We now have data about which sites link to which others, and what category they fit in measured by keywords. The authors use the data to describe an ‘interstitial’ group of NGO websites, which use the key words from all three categories equally, all link to each other a lot and also link out to sites that are strongly classified as science, civil society or management. By contrasts the strongly classified NGOs rarely had hyperlinks between the categories.
So this interstitial group of NGOs might be highly influential and driving conversation, we know that bridging between two parts of a network is a powerful position to be in because you can broker information between those parts. However, it could also be the case that the interstitial group has no agency, and is being driven by peripheral civil society, science and management websites who generate the ideas.
What we really need is data over time, so we can see how the language spreads, which is exactly what the authors conclude. This would tell us in which direction the keywords diffuse, allowing us to to see who is best promulgating their ideas, and help the philanthropists choose where to send their cash.
This is exactly what the authors conclude, and presumably should be fairly easy to do.
The fun bit is thinking about what you could do with this type of data. Academics are constantly under pressure to have impact, usually measured by publishing papers in high profile journals. Perhaps impact could instead be measured by looking at how ideas spread around a network. How many other researchers adopt your terminology, how often does it appear in papers, blogs, etc?
Culture is often thought to be undervalued by the market, could you allocate resource by looking at which cultural events have the most impact on social media. This would akin to the stated preference methods that have been used in cultural economics. Who’s doing better, the V&A or the British Museum? Analyse Twitter sentiment and allocate the funding accordingly. (Obviously you might think the goal of cultural institutions is not be liked but to be good.)
It’s also very reminiscent of the work I’ve done looking at think tanks.
“When a measure becomes a target, it ceases to be a good measure.”
Just as the academic imperative to publish has lead to junk journals and insubstantial papers, measuring the spread of language would incentivise NGOs to game the system. And of course this type of analysis does nothing to measure the quality of the underlying ideas. However, it does allow a way to measure the effectiveness non-market activity, and my feeling is that culturally governments (many philanthropists too) are hypnotised by the market-oriented measurement of value. The data driven measurements described above are a concrete way to justify non-market expenditures.