Nesta’s Big and Open Data for the Common Good raced through (I think) 7 different projects, all of them detailed in the report, and including my work with @lagaia, @rowanelena and @mparsfield in Hounslow.

The projects underpinned a more general debate about two recurrent topics – ethics, and who should be responsible for building the open data infrastructure.

Ethics and data

Whenever you are using data about people the question “Have the people in question given informed consent?” arises. When the data is not directly about people, there is still a question “Is the end to which we are using this data ethical?”. This topic generated much debate on the Twitter hashtag – as can be seen in the storify.

Clearly, using people’s data without their consent is an invasion of their privacy as well as a disservice to society. In @lagaia’s example, if Citizen’s Advice Bureau opened up detailed data about what people ask them about payday loans (which, by the way, they have no intention of doing) that might be very useful to unscrupulous lenders.

As upstanding, morally conscious individuals the obvious answer is to be extremely conservative with the uses we put data to. This has a number of non-obvious drawbacks:

  1. Informed Consent is extremely difficult to parse, since most people have no idea of the conclusions that can be drawn from a given set of data using statistical approaches. So strict interpretation of informed consent will be extremely limiting. Much of the activity discussed at the event would at be at best in a grey zone, for example. The ‘bigger’ the data, the harder it is to claim ‘informed’ consent because the the information that can be derived becomes more surprising.
  2. There is a free-rider problem. If one person does not consent for their medical data to be shared for research purposes, but others do, is it fair for the person who does not consent to benefit from any research breakthroughs predicated on other people’s generosity with their own personal data?
  3. Traditionally, academia and the third sector have been very strict about ethics, while unsurprisingly the commercial sector has not. On a case-by-case analysis we might see the strictest ethical interpretation as morally preferable, but if the cumulative outcome is for the commercial sector to have vast lead in theoretic and behavioral understanding, to be decisively more adept at data processing, is that really for the greater good?
  4. Perhaps the most important point is the huge opportunity cost of not doing certain big and open data activities. Being over-cautious could have as bad an outcome for society as an incautious approach. Playing it safe is not cost free.

@Stianwestlake pointed out that rules to enforce ethics are unsuccessful, suggesting that disaster in the financial sector was the result of bad faith and could not have been averted by more rules. In some countries bankers now have to take a hippocratic oath. Perhaps something similar could be beneficial for those using data?  Bankers and data scientists both work with social abstractions that make it easy to forget the human cost of bad decisions, and they both potentially face perverse incentives.

Data as infrastructure

We (nearly) all accept the governments role in enforcing contracts and standardising weights and measures. These activities are seen as precursors to all the public and private activity that makes our society work. Imagine trying to buy petrol if every station used it’s own system of measurement. Systems such as company registration, agreeing to use litres for fuel etc. become part of the furniture. We need rules about how information is recorded and transmitted to make the system work; a kind of systemic infrastructure.

Yet it seems clear the government does not have enough interest in enforcing similar rules for data formats and data sharing in the digital realm. For me this is the most fascinating part of the debate. @willperrin pointed out the huge potential for giving to local causes that is untapped in the UK simply because there is no mechanism to discover local charitable causes. @edtparkes talked about the important data the private sector has and which it easily could share. To me this issues is exactly the same requiring suppliers to list ingredients on packaging (and put me in mind of this amazing podcast, in part about Tesco’s and immigration patterns). @carljackmiller called for an ‘ebay’ style clearing house for collective social action.

How will these systems described above be built? Clearly the commercial sector is going to play a role, @edtparkes said “We’ll have no social impact if we don’t make a profit”, implying that anything that doesn’t make a profit won’t exist in the long term. On the other hand @trisml suggested the idea that for-profit companies could build all of this infrastructure was ‘magical thinking’ – noting that historically infrastructure has always been pioneered by the state. Finally, @duncan3ross, perhaps partially in answer to these questions, pointed out that when local authorities award contracts they should require that some part of the budget be allocated to open data concerns.

It’s hard to reinforce enough the idea – beautifully articulated by Keller Easterling here - that this systemic, digital infrastructure is as important to the public good as the network of roads or the hidden plumbing that we take to be the signifiers of civilisation.




We got off the train at the wrong stop, but, no matter, we’d get a taxi. Except… we’re not in London any more, and there are no taxis in Matera, especially when it’s nearly midnight.

So we walked the last mile to get to the old town, the Sassi, which runs along the side of a ravine. Looking out, I imagined the blinking light on top of a transmitter tower as a boat on the ocean, or a light aircraft over the Sahel – the town has the feel being on the edge of a great unknown, heightened by the even greater unknown of where our apartment was. In the end we found it by chance, just as panic was beginning to take hold.

In the daylight you can see across the valley to rows of caves, some of which are tiny churches with peeling frescos on their walls. Even in the daylight the old town is uncanny, there are so few people there. Before tourism made the Sassi an economic asset, the Italian government moved people out of the medieval caves and into the new town, they are only now moving back.

The Unmonastry is at the cusp of the Sassi’s ravine, the right place for a conference called Living On The Edge; the walls of an ancient cellar rubbing off on your shoulders as you listen to Emmanuele talk about memories of abandoned Italian villages.

Unmonastry, from the other side of the old town
Unmonastry, from the other side of the old town
Unmonastry, from across the ravine
Unmonastry, from across the ravine

The introductory session nicely bookended the LOTE spectrum between Robin Chase and Vinay Gupta. Robin is a Zipcar cofounder and her call was for us to look for “excess capacity” in the system and investigate ways to unlock it. If Zipcar unlocks the value of our underused cars through sharing them, what else could we bring that model to?

Compare with Vinay’s call, of similar kind but utterly different extent: to modernise anarchist theory, cease relying on the government, ignore the market economy and form radical cooperatives that act in our own interests using the internet as a platform. Charities be dammed, they are cut from the same cloth as the corporates. Zipcar type models are encompassed via the theoretical construct of guard labour, but it doesn’t stop there; climate change, wars and everything in between are in Vinay’s purview.

To round off the picture, Fra. Bembo pointed out that everyone would have to clean the toilets, and Jeff (who was dressed variously as a chef and a bin man over the weekend) spoke out in support of a guaranteed minimum income, which I think is rapidly becoming a hobby horse for me.

The stewardship in the title refers, I think, to the idea of an individuals or organisations that attempt to maximise the social benefit of a resource for non-financial reasons – people who ‘steward’ a resource. But the topic that kept recurring is actually the Zipcareque sharing economy. Similar idea, getting more value by sharing resources, but, crucially, driven by market forces rather than cooperative benevolence. This tension came out at the plenary at the end of the first day, with the question “Is Airbnb bad because it makes room sharing, which used to be a gesture of friendship, into a financial transaction?”

The answer that Robin gave, and which I’m inclined to agree with, is that before Airbnb existed people mostly didn’t let their spare rooms, because there was too much friction in the transaction. The new ‘sharing economy’ has probably displaced very little benevolent, non-monetised sharing activity, with either cars or spare rooms. Airbnb is straight out of VC-funded, bubble-valley, hypercapitalist California. I might not like the way it’s come about, but I can see the value of what it enables as despite the economic system in which it arose.

You might have guessed that by now I have an affinity for Robin’s way of thinking, and I think one of the most interesting things she posited was the idea of market failure in the sharing economy. Using economic theory as a lens in very helpful for me, but, I speculate, a total turn off for most other people at the conference. In any case, a massive case of market failure occurs around personal data, where people simply cannot understand the value of their personal data, and the way it aggregates to become incredibly powerful.

So that’s the cut-and-dry economics, but, much more than ever that ever before, Patrick persuaded of a rational position which is sceptical of economic theory. In his view, using money to value things causes people to have a different psychology. It makes it easier, for example, to abuse natural resources, because it makes them abstract. So when I explained that the market is a wonderful way of allocating resources, that it does a magical computation to prioritise what people most want, he agreed. But at the big scale, the environmental scale, the use of money causes this damaging psychological disconnect. When people explain to me why they don’t like economics, I often feel it’s because they don’t get how powerful it is, our conversation didn’t follow that pattern.

Which leads me to the unconference session that Helen and I lead, Art Vs Science. As a spur to discussion, we divided the world into Tribe A – scientists and techies – and Tribe B – Artists and the academic humanities. I proposed that the tribes need to recognise their cultural differences and reconcile them, Helen argued the tribes didn’t really exist like this, and that in any case the differences in culture were a good thing. More is available in the notes, but we did find that the tribe model rang at least a little true with the experience of those in the Unmonastry. Kat was especially good in the debate, bringing lots of useful ideas. I was interested to learn that artists-in-residence at CERN only get three months, those running the program worry  the artists might go native and start thinking like scientists otherwise.

One funny conclusion was that even though Helen identifies as Tribe B, and I identify as Tribe A, we both perceive ourselves to be in the minority – at the conference and in wider society. Obviously impossible.

Listening to the talks I noticed a recurring structure in presentations. Someone would advocate for some kind of action, then we’d lament that not many people agreed with us. This was followed by the idea that ‘people’ should be re-educated so they will want they same thing we do. eg. People should use encryption on the web, but they don’t see the value, so we should re-educate them so they do. People shouldn’t go to supermarkets, but they do, so we should re-educate them to not.

In discussion with Theresia, I realised how pervasive this type of thinking is – to the point where I’ve certainly articulated it myself. It’s not a completely fair analysis, but it gave me something to think about.

Sam, who was videoing the gathering, told me he was having difficulty making the Sassi look real on screen. In the diffused light, with buff-colour tufo buildings between grey ravine and the grey sky, the town looks as flat as film set. In fact Pasolini’s films and Mel Gibson’s Passion of The Christ both used the Sassi as a backdrop, as will a remake of Ben Hur.

At the same time the Sassi is more three dimensional than any modern town. Buttresses fly, shoulder-width steps wind through, over, under. One man’s pavement is another mans roof. The bedrock is a Swiss cheese of human activity. When we arrived at the apartment we were led through a tiny cupboard-style door under the stairs, down a staircase and into a cellar space the size of a 4 bed London flat. There’s a story that at another conference a local showed a someone into his house, to show how it connected to the cave system. They went deep enough to end up under a church, looking into a pit of human bones.

God knows what else is under there. One of the town’s churches (wonderfully called Chiese Rupestri di San Nicola dei Greci e Madonna delle Virtù) intersects with an older church carved into the hill. No one knows how old the older church is, the first docmentation comes from the 14th century – maybe the question doesn’t even make sense, geology and architecture elide in Matera. In 1991 they discovered a Roman cistern, hiding, unknown, below the town square, under everyone’s feet.

The time dimension warped too; in the way that three days a Glastonbury feels like you’ve been away for a month. You could open a door in the depths of a cave and come out of at the top of a campanile a week in the past and on the other side of the valley. Perhaps I was just confused because the clocks changed while we there.

I didn’t have much more luck with trains when I came to leave. Somehow I seemed to keep missing the main train station. I took me too long to realise what should have been obvious: the train station is underground.




This week I’ve been putting into action a plan to get more people in to meet us, which there is budget for, and it seems like it will be helpful not only for our current projects, but also for future things – even after our PhDs. In particular I’m interested to chat with people who are interested in Network Analysis, which I think will be very interesting for several of us.

I’ve had a duel with John Fass via Google Docs. Having written a long piece about the culture in the humanities vs that of the sciences, in particular thinking about the way epistemological commitments underpin that culture. We’ve had a lot of fun, but it bought up an interesting aspect of how academic writing is read: for all the very close attention that has been paid to challenging every individual assertion that I make, the overall thrust of my argument has gotten lost. In some ways I think this ungenerous way of reading hampers debate and understanding; a well-structured and enlightening sketch can be composed of lemmas that are not themselves individually beyond contestation.

In terms of the PhD itself an interesting frication has come up between normal modes of recording methodology and software development. In particular, it’s hard for me describe in any deep detail how my software works for gathering data; but at the same time it’s the cornerstone of my methodology and needs to be presented as such. Even more, I make regular changes to the software to make it better, but it means my research is slightly dependent on shifting foundations. And if I suggest the changes are not material, one, who is to decide that, and two, what if a small change has massive unintended consequences?

I wish I had time to implement proper testing, but realistically I don’t.

Finally, there is the use of a Third Party entity extraction service (alchemy) and the question of whether using this black box is acceptable, how sensitive the results are to its unknowable inner workings.

Work on Peterborough feels increasingly likely…



A particularly good exhibition of sonic art by Rafael Lozano-Hemmer, which I wanted to keep some notes about.

The first thing, the most important thing, is that everything was presented beautifully, and worked. Where ultrasonic sensors were supposed to detect a person approaching, they did; when a button was supposed to record your voice, it did. I can say from experience this is no small achievement – so hats off to that. The second thing, which is also often lacking in exhibitions that involve something digital, is that everything was beautifully presented and looked as though the artists had been able to fulfil their intentions.

The artist has a concept of ‘speakers as pixels’ and in the piece Sphere Packing below it really works. Each sphere is covered in lots (sometimes hundreds) of tiny transducers working as speakers. They are so quiet, so close together, that from a distance each sphere emits white noise. But if you put your ear very close you can hear that it is actually playing a discernible song. For example, one sphere plays Mozart, but each speaker is playing a different bit of his work, so in total its just a random mess. I haven’t seen this played with before.

Sphere Packing
Sphere Packing

In this instance the way the effect works means that you to literally put your ear against the speaker to get an individual signal. I wondered if it would be possible to have the cross over from noise to signal a bit further away, but the logarithmic nature of perception might make this rather hard to achieve. I also wonder what the effect would be if the speakers played different but more related sounds, for example just fractionally out of sync. Virgin territory, as far as I know, it’s seems to explore a really interesting cross over between noise and music in a spatial way. This especially, but also perhaps the exhibition as whole, makes me thing of applications in calm technology.

In Voice Array visitors are able to record a short sample of their voice, which is played back on it’s own, then with all the previous contributions simultaneously. For me the sound aspect of this was less exciting than the way LEDs worked, each independently twinkling in slightly different shades of white, giving an effect that set it apart from the clinical look these things normally have.

Voice Array

Finally, Pan-Anthem, which features magnetic ‘bricks’ representing each country in the world by playing back their national anthem (although I seem to remember hearing that Oman doesn’t have a national anthem because music is banned there?). The magnet in the speaker sticks to the metal sheet on the wall, so the bricks can be re-arranged to visualise different sets of data. Weirdly reminiscent of the vitamin calendar, and obviously I like the idea of stand alone devices that play one song – being, this the eventual goal of the Rifff project.

Overall, the sound was actually quite annoying and didn’t add that much, but I feel like there is a realisation that could really make the sonic dimension work. I guess the problem with country’s national anthems is that they are mostly unknown and don’t evoke anything. If the system relied on a palette of sounds or songs I knew, or triggered a mood, perhaps it would work better. As with everything else though, beautifully, functionally executed.








Mega admin week. Why? Because the alternative is coding. However, I feel like I’ve reached the end of the coding ‘hygiene’ phase and got to a point where I can move ahead without feeling frustrated by the structure of the existing code base.

One of the mental overheads that I’ve been struggling with is improving the software while leaving it compatible with deployed Hounslow version of the project. My approach to this is going to be to do one more week of development in Hounslow, before deploying for the final pass there. Then I’ll abandon the Hounslow project and starting making incompatible changes so that I can move forward.

Further attempts to connect with academics have failed – I suspect because the people I’ve met with have full research slates. Hopefully further consultation with Tom will lead to a way forward on this…

Finally, further link ups between me and Dan Lockton look as though they might be on the cards, which would be very lovely.



Coding Frames, Methodologies, Repeatable Scientific Experiments 

While Hyperlocal observatory has successfully been deployed in Hounslow, it’s been quite an ad hoc process. While that’s inevitable in the circumstances, in the longer run, and for research to be based on it, clearly it needs a well defined methodology.

Pete suggested that I write down the methodology in such a way that I could ask someone else to repeat the experiment. This proved to be a informative process. The whole thing is more complicated that I expected, but what particularly stood out are the judgement calls the person coding the data is asked to make. Some of them I believe to be relative innocuous – such as tagging with categories. While this might be quite tricky, I don’t think differences of opinion at the margin are likely to fatally undermine the data that comes out.

What is more philosophically interesting is the point that there is also a judgement call “Is this relevant community data, or should I put it in the bin?”. Defining what qualifies as community data, as well as possibly influencing the outcome of the research, seems like a key part of the theoretical underpinning.

Elevator Pitch / First Para of Abstract 

While discussing the fact that it’s hard for me to articulate the top line of what my project is, one of the suggestions was for me to again reformulate a single paragraph description. Here is the global version of that:

Communities have a ‘digital footprint’ of locally-oriented content: blogs, forums, Tweets, Facebook pages. This set of data, which barely existed a decade ago, is increasing in richness rapidly. My PhD seeks to address how this (intuitively valuable) dataset can be used to improve quality of life for a community.

In fact I formulated three versions, with two more specific version fleshing out how the value is to be extracted from this data. Presumably as my PhD narrows one direction will start to come out as a winner.

Development Work 

I continue to try and get myself into a place where I’m happy to look at and modify the code base, which is still elusive. A place where I can just add something in without feeling the need to re-evaluate old work. In part this is because new architectures suggest themselves constantly, in that is in tension with the fact that I need to deploy updates to an already deployed code base. Secondarily, having rushed so much previously there is lots of underlying technical debt – misnamed things, poorly considered folder structure etc. This is exacerbated by the fact that Meteor itself still has evolving conventions on how to structure projects.



Met with DCLG, which proved to be a interesting opportunity to explain my project to people who are very familiar with community issues, but less so with the tech. One of the things that came from this is the increasing need for me to position myself with regard to projects like CASM, where my headline idea is that they are doing Big Data, and I’m doing Little Data.

Also tried to get some Meteor work done, which involved me upgrading to Meteor 0.9, which was a nightmare. Such are the joys of using a platform that hasn’t reached version 1. Can’t wait for that, unless of course it introduces massive changes.

It’s hard to write about the Creative Citizens conference, it’s given me so much to think about that I can’t corral all the ideas into any sensible shape. A lot of the specific topics – participation, creativity, community, the city – have been in the air for so long that I won’t recount my notes here.

Collectively, the big-picture keynote talks, plus all the on the ground research, snapped into focus a macro view of policy, politics, money and economics in a way that was completely fresh to me.

The panel at the end of the first day, composed of representatives from four think tanks, was the peg on which I mentally hung the rest of the conference.  It was during their discussion that I realised that the measurement of value was, for me,  the concept that tied everything together.

The research presented at Creative Citizens was asking people to value social cohesion, inclusivity, creativity, empowerment.

On another level, the wonks, quite bluntly, pointed out that politicians would evaluate policy by how it helped them claw way across the next electoral threshold – services delivered cheaper, better education as measure by exam results, reduced benefit expenditure etc.

On the third, even more dismal, level everyone accepts that as a society economic value the default setting for measuring everything, which we shorthand as neoliberalism. This is inimical to the Creative Citizens agenda, which is two levels away on my just-invented policy measurement vagueness hierarchy (PMVH?).

When I worked at the (co-operative) council in Lambeth we said the co-design agenda was about, approximately, ‘getting more for your money in the era of austerity’. Very often I think academia gives the same impression, but it’s a bit of charade in both cases – one because it’s not clear how co-design or hyperlocal etc. convert to economic value, and two because I’m not sure that’s what we truly care about anyway.

What Geoff Mulgan’s talk made me think is that what’s really going on is an intellectual rejection of the notion of economic value. We aren’t really interested in hyperlocal media or co-design because it will help eek out the budget, but instead because it’s alternative value system to the remorselessly market based one, a system which we suddenly realised was horribly dysfunctional in 2008.

I heard four different speakers talk about the Occupy movement, regarding things like horizontal organisations, the hyperlocal perspective, what Occupy tells us about participation. But isn’t there a part of us that is interested in Occupy because it was literally manning the barricades against neoliberalism? Surely it’s a factor.

This ties into Adam Greenfield’s talk at LSE of the same week, where he was absolutely frank about his political views. I saw huge crowds thronging to see FT economist Martin Wolf speak on the financial crisis, before finding a more modest lecture theatre for Greenfield’s talk – I now take this to have symbolic significance. His thinking focuses on Creative Citizen themes, but from the perspective of ‘the city’, and I should note that he comes from a very different place on this.

The city, rather than the country, naturally becomes the unit of analysis, because a country, as abstraction, encourages abstract statistical and economic thinking, while the idea of a city makes us think of concrete things – town halls, street parties, the homeless. This is the mode of thought which gives rise to the Creative Citizens agenda, the two are one and then same. Geoff Mulgan and Paola Antonelli both spoke a great deal about projects led by mayors rather than presidents or prime ministers, I think for this reason.

So what should we make of the wonks telling us the Creative Citizen worldview wasn’t sufficiently ‘instrumental’? Creative Citizen ideas promise to serve up a little bit of everything with a selection of intangible benefits on the side, but as I’ve noted, politicians care about social indices – GDP, educational attainment, life expectancy, and in the short term.

Another question – wonkspeak alert – does community-led design “go to scale?”, or, how would it look if you did a lot of it? In my experience this isn’t something co-design proponents are particularly concerned with, but if you want to affect a change, surely it’s an issue?

I sensed that a lot of the audience felt that the think tankers didn’t ‘get it’. But it’s more interesting to assume that they did.

I wish I had a more intellectual reference point, but I kept on thinking of Ian Hislop on Have I Got News For You, along time ago, when Bush was in power. He said that intellectual lefty Americans loved watching The West Wing because it let them pretend the President was a left-wing nobel laureate played by Martin Sheen, rather than confront the reality that he was a neo-conservative malapropism-prone Texan.

I wondered if there is a sense in which advocating small-scale, community-led, DIY policy could be seen as hiding out too, doing well-motivated, beautifully crafted projects, but failing to engage with governmental thinking – instead doing projects that aren’t expected to scale and aren’t persuasive to policy makers.

When I spoke to Leon Cruickshank about the community-led project he led in Lancaster he said that as part of his process he absolutely expected local government experts to have closed meetings where they could use technical language and voice expert opinions. It seems to me that many people wouldn’t always want to highlight that part of their project because it seems to go against the ethos.

But it absolutely addresses one of the points raised by the think tank panel, which was that community-led design ignores the experts who are needed to implement complex and technical aspects of projects. Perhaps these concessions to reality are should be made more of.

I do sometimes admire the brutally prescriptive approach that ‘deliberative democracy’ takes for exactly this reason, although Leon did mention some drawbacks to this approach.  Deliberative democracy also interests me because it seems so on-topic for these types of discussion but it never gets mentioned, perhaps because from an American university?

Anyway… it seems to me measurement could be part of the answer too. If it was possible to articulate measurements of inclusivity or community cohesion perhaps they would become more attractive targets for policy, and move up politician’s agenda. Where economic value and social values are in tension, one could make the tradeoff explicit. Currently, economic value wins because it can often be captured by a number.

Tying this back into my own research, what I’m looking at is studying community cohesion by looking at the digital signature it leaves behind, which I really hope has some potential to make more visible slippery constructs such as community cohesion, and play a part in this measurement idea.

Which again loops back on the Creative Industries workshop I attended in Beijing, where the idea of measuring the economic impact of creativity was discussed in some detail, including the notions of stated preferences as alternative to the revealed preferences of standard economic thought.

The conference ended on the day of the Indyref result, with all of the talk of revivified political culture that bought. Yesterday Ed Miliband proposed breaking up the banks and more local powers, perhaps the economists and the wonks are underestimating the Creative Citizens approach to politics, and it can be part of a new era of civic dynamism.



This week was mostly eaten by preparing for the Creative Citizen conference. One of the key points has been the amount of work of installing even a modest exhibit – in this instance cleaning data that I normally would bother about, and much testing of the Raspberry Pi / Banana Pi  setup.

Despite this, during the install there were still problems with connecting the computers to the Internet via wifi. A key lesson is that any kind of connection that relies on wifi is always a possible source of concern, and multiple fall backs are required. Given how flawlessly Ethernet seems to work, it seems like a sensible idea to use it where ever possible.

Finally there was the issue of documentation, after doing so much work to get the thing installed it’s not that easy to find the energy to get nice photos taken. Ben says: “Friends document each other’s work”.

The conference itself is written up separately.



Having been round in some circles with getting our projects off the ground, in the end I decided to try and get a meeting together between the all the students to discuss the issues, which resulted in Google Doc of questions that everyone agreed needed answering.

I added a new screen to the observatory software, showing the graph and map view for use in the Creative Citizen conference.

I also had to resurrect the JSON API for the Adafruit IOT printer to work. Pleased to report the printer worked straight out of the box.

Also went to the LMI For All hackday at Hub Westminster and finally had a meeting with Elvira Grob and John Fass about finding a research designer for taking John’s pin boards forward.