You have reached 1,202,199 people – really?

Measuring the impact of e-Diplomacy social media tools like Twitter or Facebook is a notoriously contested area. On the one hand, there is a friendly crowd of free-services, such as Klout.com that purport to tell me precisely how signficant (or, mostly, insignificant) I am on the web. On the other there is a constantly renewing pool of free data aggregation services, like Keyhole.co, which display a compelling, best-case picture of how a campaign or event is registering on the web.

I mention Keyhole because the figure quoted in the title comes from a dashboard set up to track the social buzz around a recent event (the ICT4Ag conference organised by MinAgri, Rwanda, and CTA). To be clear, there was a huge online buzz around the event, in large part the result of careful and detailed preparation. There was an onsite team of young social reporters from ACP countries, who’d been trained in using digital media immediately before the event. This is a well-worn strategy – Diplo has been doing similar training with Diplo fellows at events, since the 2009 IGF – and can be supplemented these days by networking before and during events. The ICT4Ag social reporting coordinator, my friend and colleague, Pier Andrea Pirani, had connected with 250+ people online before the event, people who couldn’t attend but were interested in following and sharing with their friends and followers material published online (also something Diplo does for events, as with the IGF13 aggregator site). And of course there were many habitual social media users reporting out from the event. 

He doth protest too much, methinks¹

The problem is that nonsense numbers get in the way of analysing properly the impact of our communication. For example, the ‘reach’ calculation is based on an algorithmic calculation of how user’s networks of followers interact. Although the sites won’t publish their formulas, it’s a calculation that takes some account of the ‘echo chamber’ effect. In other words, calculating reach simply by adding together Pirani’s and my followers, for example, would miss the fact that we have many followers in common (192 in our case), and indeed, many of those followers also overlap (figures from Twiangulate.com) This overlap is characteristic of any thematic or sector group, very evident in the diplomatic community, where, for example, embassies and MFA re-tweet each other and the output of colleagues in alliance ministries. It’s a powerful online campaigning tool, analogous to chanting in a football crowd, to drown out the opposition. But in the bounded diplomatic community  – as in the similarly, though less distinctly bounded, ICT for Agriculture (ICT4Ag) networks  – it makes a mockery of claims about reach.

Even more remarkable is the claim that the social content from ICT4Ag13 generated 4,099,190 impressions. That is to say, the combination of people who were reached and the number of times each of them tweeted. Keyhole, like most similar sites, acknowledges that impression numbers is a theoretical maximum, but then why publish it? The second problem with this kind of smart, well presented data is that the figures are as seductive and dangerous as pure, refined white sugar. Sugar coated numbers, large enough to generate a positive ROI on communication investment costs, are hard to resist, albeit hedged about with qualifications. And I’d bet a kilo of unrefined best Barbadian that it’s the mega-figures that stay in the memory.

Andreas Sandre, the increasingly influential communication specialist at the Italian embassy in the US, blogged on this site about influence and what it means. His pieces (links available below) underline what matters in reality, which is the amount of engagement generated by social media output. It is much harder to measure engagement, in the sense of conversation between people, and the extent to which any particular piece of content is trafficed on the web. There are a number of subscription services, such as Radian 6, which provide more sophisticated social listening services, including the ability to set paramenters that can generate analyses of the mood of the engagement, in terms of positive and negative. For everyday use the number of free or free-mium services continues to expand. Some that we have used include:

• twittercounter.com
• socialmention.com
• mentionmapp.com
• tweetreach.com/
• followerwonk.com/
• analytics within dashboards like Hootsuite

In terms of good practice, I think my own primary principle is that trend matters more than absolute numbers. It is essential to do some tracking and comparing, and the free/freemium services provide enough information for comparison purposes. And I spend more time doing than measuring: it remains true that if you feed the web it will feed you back. Spending time to connect with others on the web is not only valuable in itself – isn’t social media about being social – but also the best way to generate continuing conversation around the issues and content that matters. 

1. Apologies to Mr Shakespeare