One of the easiest ways to get someone started on data visualisation is to introduce them to word clouds (it also demonstrates neatly how not all data is numerical).

Using tools like Wordle and Tagxedo, you can paste in a major speech and see it visualised within a minute or so.

But is a word cloud the best way of visualising speeches? The New York Times appear to think otherwise. Their visualisation (above) comparing President Obama’s State of the Union address and speeches by Republican presidential candidates chooses to use something far less fashionable: the bar chart.

Why did they choose a bar chart? The key is the purpose of the chart: comparison. If your objective is to capture the spirit of a speech, or its key themes, then a word cloud can still work well, if you clean the data (see this interactive example that appeared on the New York Times in 2009).

But if you want to compare it to speeches of others – and particularly if you want to compare on specific issues such as employment or tax – then bar charts are a better choice. Compare, for example, ReadWriteWeb’s comparison of inaugural speeches, and how effective that is compared to the bar charts.

In short, don’t always reach for the obvious chart type – and be clear what you’re trying to communicate.

UPDATE: More criticism of word clouds by New York Times software architect here (via Harriet Bailey)

Obama inaugural speech word cloud by ReadWriteWeb

via Flowing Data

(The first release of the spreadsheet actually omitted the column containing who the meeting was with, but that seems to be fixed now… There are, however, still plenty of character encoding issues (apostrophes, accented characters, some sort of em-dash, etc) that might cripple some plug and play tools.)

Looking over the data, we can use it as the basis for a network diagram with actors (Ministers and lobbiests) with edges representing meetings between Minsiters and lobbiests. There is one slight complication in that where there is a meeting between a Minister and several lobbiests, we ideally need to separate out the separate lobbiests into their own nodes.

This probably provides an ideal opportunity to have a play with the Stanford Data Wrangler and try forcing these separate lobbiests onto separate rows, but I didn’t allow myself much time for the tinkering (and the requisite learning!), so I resorted to Python script to read in the data file and split out the different lobbiests. (I also did an iterative step, cleaning the downloaded CSV file in a text editor by replacing nasty characters that caused the script to choke.) You can find the script here (note that it makes use of the networkx network analysis library, which you’ll need to install if you want to run the script.)

The script generates a directed graph with links from Ministers to lobbiests and dumps it to a GraphML file (available here) that can be loaded directly into Gephi. Here’s a view – using Gephi – of the hearth of the network. If we filter the graph to show nodes that met with at least five different Ministers…

we can get a view into the heart of the UK lobbying netwrok:

I sized the lobbiest nodes according to eigenvector centrality, which gives an indication of well connected they are in the network.

One of the nice things about Gephi is that it allows for interactive exploration of a graph, For example, I can hover over a lobbiest node – Barclays in this case – to see which Ministers were met:

Alternatively, we can see who of the well connected met with the Minister for Welfare Reform:

Looking over the data, we also see how some Ministers are inconsistently referenced within the original dataset:

Note that the layout algorithm is such that the different representations of the same name are likely to meet similar lobbiests, which will end up placing the node in a similar location under the force directed layout I used. Which is to say – we may be able to use visual tools to help us identify fractured representations of the same individual. (Note that multiple meetings between the same parties can be visualised using the thickness of the edges, which are weighted according to the number of times the edge is described in the GraphML file…)

Unifying the different representations of the same indivudal is something that Google Refine could help us tidy up with its various clustering tools, although it would be nice if the Datastore folk addressed this at source (or at least, as part of an ongoing data quality enhancement process…;-)

I guess we could also trying reconciling company names against universal company identifiers, for example by using Google Refine’s reconciliation service and the Open Corporates database? Hmmm, which makes me wonder: do MySociety, or Public Whip, offer an MP or Ministerial position reconciliation service that works with Google Refine?

A couple of things I haven’t done: represented the department (which could be done via a node attribute, maybe, at least for the Ministers); represented actual meetings, and what I guess we might term co-lobbying behaviour, where several organisations are in the same meeting.

There have been quite a few tools springing up over the past few months that I’ve not had time to blog about, so here’s a roundup post on all of them – a bumper Something For The Weekend (let me know how you find these).

1. Junar – for scraping websites and sharing data

Junar presents a much easier way to scrape data from online tables with its ‘Collect Data‘ tool – and the team behind it tell me they have plans to build functionality allowing users to scrape linked pages, as well as the ability to scrape PDFs.

2. BuzzData – for sharing data

BuzzData is a platform for sharing data – essentially a social network where you can follow other data journalists or datasets, tag and license your data, and – importantly – add visualisations, articles and attachments. When someone else builds on your data, it tells you, which is nice.

3. DataMarket – for finding data

DataMarket is exactly what it says on the tin: a market for data from organisations including the UN, BP, Eurostat, the IMF, USGS, and various other acronyms. You can access the data for free, or pay for extra functionality such as exporting to Excel.

4. Google News Scraper – for grabbing data on news coverage

This scraper will allow you to gather data on coverage of a particular issue, event or person. It only gathers the teaser text but the country data may if you want to map coverage, while the URLs can provide a starting point for further scraping experiments.

5. Metadata extraction tool – a first step for searching document dumps?

This is aimed at file preservation activities, but it has a few possible applications for journalists. Firstly, it has a Windows interface for exploring the metadata of a bunch of files, making it possible to sort in different ways to more quickly look for information you’re seeking. Secondly, the generation of an XML file will give some structure which could allow you to, for example, plot your documents on a timeline, spotting patterns or outliers.

6. Roambi – data visualisation on your iPhone

Sadly, it’s only your iPhone, not anyone else’s, so this is more if you’re on the move but want to go through some private data visualisations which might hide a story.

7. Data Wrangler – web-based data cleaning tool

This looks pretty powerful, if not pretty full stop. Here’s a video:

http://vimeo.com/19185801

8. Impure – visual programming language

From the About page:

“Impure is a visual programming language aimed to gather, process and visualize information. With impure is possible to obtain information from very different sources; from user owned data to diverse feeds in internet, including social media data, real time or historical financial information, images, news, search queries and many more. Impure is a tool to be in touch with data around internet, to deeply understand it. Within a modular logic interface you can quickly link information to operators, controls and visualization methods, bringing all the power of the comprehension of information and knowledge to the not programmers that want to work with information in a professional way.”

9. Zanran – PDF/spreadsheet/table search engine

This looks a very useful tool for narrowing down searches to PDFs, spreadsheets, and tables within webpages (the advanced search allows further narrowing by filetype, date, server location and site). Clever stuff behind it – particularly in the way it looks at images and decides if they’re charts. The site says they plan to add Word documents and PowerPoint presentations soon.

Here’s a quick experiment in data visualisation to provide an instant insight into a story on how the blame game is being played by columnists.

The data is taken from a Liberal Conspiracy blog post – I’ve transferred that into a spreadsheet with limited categories and used the Gauges gadget to visualise the totals.

A screengrab is below – but there is also an embed code that provides a gauge that will be updated whenever a new columnist is added. See the spreadsheet for both the gauge and the raw data.

Columnist Blame Game Gauge

Mauro Accurso has followed up his rapid translation of last week’s inverted pyramid of data journalism with a Spanish version of part 2: the 6 C’s of communicating data journalism. It’s copied in full below.

La semana pasada les traduje la primera parte de La Pirámide Invertida del Periodismo de Datos de Paul Bradshaw que prometió extender en el aspecto de comunicación del extenso proceso que significa el periodismo de datos.

En esta segunda parte Paul recorre 6 formas diferentes de comunicar en periodismo de datos que pueden ver en el cuadro de arriba y al final encontrarán un gráfico que resume toda la teoría (la cual está en desarrollo todavía y Bradshaw pide aportes, comentarios y sugerencias):

“El periodismo de datos moderno ha crecido junto con un gran aumento en visualización y esto puede llevarnos algunas veces a dejar de lado diferentes formas de contar historias que involucren grandes números. La intención de lo siguiente es funcionar como un manual para asegurar que todas las opciones sean consideradas:

1. VISUALIZACIÓN

La visualización es la forma más rápida de comunicar los resultados del periodismo de datos: herramientas gratuitas como Google Docs lo permiten con un sólo click y herramientas más poderosas como Many Eyes sólo requieren que el usuario pegue la data cruda y seleccione de un grupo de opciones de visualización.

Pero facilidad no es igual a efectividad. El surgimiento de cuadros-basura demuestra que la visualización no es inmune al churnalism o al espectáculo sin profundidad. Hay una rica historia de visualizaciones en gráfica que se mantiene relevante para la generación de las infografías online: enfocarse en no más de 4 puntos de datos, evitar el 3D y asegurarse que el gráfico es autosuficiente son sólo algunas.

No es un proceso simple pero, sin embargo, la visualización tiene una gran ventaja que hace que ese esfuerzo valga la pena: puede hacer que la comunicación sea increíblemente efectiva. Puede proveer de un método de distribución de tu contenido que no puede ser igualado por otros tipos de comunicación listado acá.

Pero su mayor fortaleza es también su mayor debilidad: la naturaleza instantánea de las infografías también significa que las personas a menudo no pasan demasiado tiempo mirándolas. Las hace muy efectivas para la distribución pero no para el engagement, así que es importante pensar estratégicamente acerca de 1) asegurarse que la imagen contenga un enlace a la fuente; y 2) asegurarse que haya algo más en la fuente cuando la gente llegue.

2. NARRACIÓN

Un artículo tradicional puede luchar para contener la clase de números que el periodismo de datos suele recorrer, pero aún así provee una forma accesible para que las personas entiendan la historia, si está hecho bien.

Como con la visualización, menos suele ser más. Pero también, como en la mayoría de la narrativa, necesitas pensar en el significado y tus objetivos en comunicar esos números.

Las cifras abstractas pueden ser impresionantes, pero sin sentido e inútiles. ¿Qué significa que 10 millones hayan sido gastados en algo? ¿Eso es más o menos que lo usual? ¿Más o menos que algo similar? Traten de bajar los montos a cantidades manejables: las sumas por persona o por día, por ejemplo. Finalmente, usen la edición para enfocarse en las cuestiones principales y asegúrense de enlazar al conjunto.

3. COMUNICACIÓN SOCIAL

La comunicación es un arte social y el éxito de infografías a través de medios sociales es un testamento de eso. Pero no son sólo las infografías que son sociales, la información también lo es. The Guardian ha demostrado eso de forma exitosa con la rica comunidad del Data Blog y alrededor de su API. Iniciativas de Crowdsourcing con el objetivo de recolectar data también pueden brindar una dimensión social a la información (ejemplos que remarca Paul: proyecto “Investigate your MP’s expenses” del Guardian que liberó más de 450 mil documentos para que los revisen los usuarios y cuando hicieron un crowdsource de las supuestas especificaciones del iPad). Hay otros ejemplos también, especialmente cuando no hay otra forma de conseguir la información.

La conectividad de la web ofrece nuevas oportunidades para presentar al periodismo de datos en una forma social. La aplicación de ProPublica que provee resultados basados en tu perfil de Facebook (escuelas a las que fueron; amigos que usaron la aplicación) es un ejemplo de como el periodismo de datos puede aprovechar la data social y, al mismo tiempo, como comunicar los resultados del periodismo de datos puede ser orientado alrededor de dinámicas sociales usando elementos como concursos, compartir, competiciones, campañas y colaboración. Estamos recién en el comienzo de este aspecto del periodismo online.

4. HUMANIZAR

Los programas de noticias a menudo utilizan casos de estudio para tratar el problema de presentar historias basadas en números en televisión o radio. Si los tiempos de espera en hospitales han aumentado, hablan con alguien que ha tenido que esperar un montón de tiempo por una operación. En otras palabras, humaniza los números.

Más recientemente, el crecimiento de gráficos en movimiento generados por computadora ha bajado la presión de cierta forma, ya que los presentadores pueden utilizar animaciones poderosas para ilustrar una historia.

Pero una vez más, surge el punto de hacer historias relevantes para las personas. Como escribí en “One ambassador’s embarrassment is a tragedy, 15,000 civilian deaths is a statistic” (resumen del post en español: periodismo de datos y filtraciones masivas – cuando la muerte es una estadística): cuando te mueves más allá de escalas que podamos manejar a un nivel humano, luchas para enganchar a la gente en el tema que estás cubriendo, no importa cuán impresionante sea el gráfico.

Así que después de estar enterrado en información abstracta necesitamos recordar que salir y grabar una entrevista con una persona cuya vida haya sido afectada por la data puede hacer una gran diferencia para propulsar nuestra historia.

5. PERSONALIZAR

Uno de los grandes cambios del periodismo online es que abre toda clase de posibilidades alrededor de la interactividad. En cuanto al periodismo de datos eso significa que los usuarios pueden, potencialmente, controlar qué información es presentada a ellos en varias entradas.

Hay algunas formas relativamente bien establecidas de esto. Por ejemplo, cuando un gobierno presenta su último presupuesto, los sitios web de noticias muchas veces invitan al usuario a ingresar sus propios detalles y averiguar cómo el presupuesto los afecta. Una variante reciente de esto fueron esos sitios interactivos donde invitaban al usuario a hacer sus propias decisiones de cómo recortarían el deficit (la versión del Financial Times llevó eso más allá agregando estrategias de partidos y políticas).

Otra forma común es personalización geográfica: el usuario es invitado a entrar su código posta y otra información geográfica para descubrir como un tema en particular está resultando en su lugar de residencia. Una tercera es simplemente “tus intereses”, como demostraron los acercamientos de Popvox a engagement político y el Newsmatch de LA Times.

Mientras más y más data personal está en manos de sitios de terceros, las posibilidades de personalización se expanden. El ejemplo de ProPublica de arriba demuestra como la información de perfil de Facebook puede ser usada para personalizar automáticamente la experiencia de una historia. Y existen variasaplicaciones que ofrecen presentar información basada en localización vía GPS.

Esto también indica que puede haber varias formas en las cuales la personalización y estrategias sociales pueden combinarse. Las noticias personalizadas pueden, de muchas maneras, ser usadas como una expresión de nuestra identidad: acá es donde vivo, de esta forma me afecta, en esto estoy interesado. El COO de Facebook predijo que todos los medios van a ser personalizados en 3-5 años; está claro que eso es algo donde las redes sociales nos van a llevar.

6. UTILIZAR

La forma más compleja de comunicar los resultados del periodismo de datos es crear algún tipo de herramienta basada en la información. Las calculadoras son opciones populares, así como herramientas con GPS, pero hay un montón de amplitud para aplicaciones más complejas mientras más información está disponible del publisher y el usuario.

Una vez más, hay un entrecruzamiento acá con personalización, pero es posible proveer utilidad sin personalización. Y muy a menudo, la complejidad y barrera consiguiente con respecto a los competidores presenta también oportunidades comerciales.

En Reed Business Information, por ejemplo, su modelo está orientado hacia este tipo de utilidad: atraer usuarios en varios puntos de la cadena de comunicación (actualizaciones online, revistas impresas, noticias móviles) y direccionarlos hacia el punto donde están más cercanos a una decisión de compra. La idea es que mientras más cerca tu información está de su acción, más valiosa es para el usuario.

Crear utilidad de la información es ahora relativamente costoso, pero esos costos están bajando como resultado de la competencia y la estandarización.

UN MEDIO PARA EXPLORAR

Lo que todo lo anterior hace evidente es que hay áreas enteras de periodismo online que todavía faltan ser adecuadamente exploradas, y de hecho en la mayoría todavía falta establecer convenciones claras o ideas de buenas prácticas. Esto trata de ser un resumen de aquellas convenciones que están surgiendo pero sería genial agregar más. Mientras tanto acá tienen ambas partes del modelo juntas”:

Last week I published an inverted pyramid of data journalism which attempted to map processes from initial compilation of data through cleaning, contextualising, and combining that. The final stage – communication – needed a post of its own, so here it is.

UPDATE: Now in Spanish too.

Below is a diagram illustrating 6 different types of communication in data journalism. (I may have overlooked others, so please let me know if that’s the case.)

Modern data journalism has grown up alongside an enormous growth in visualisation, and this can sometimes lead us to overlook different ways of telling stories involving big numbers. The intention of the following is to act as a primer for ensuring all options are considered.

1. Visualisation

Visualisation is the quickest way to communicate the results of data journalism: free tools such as Google Docs allow it with a single click; more powerful tools like Many Eyes only require the user to paste their raw data and select from a range of visualisation options.

But ease does not equal effectiveness. The rise of chartjunk illustrates that visualisation is not immune to churnalism or spectacle without insight.

There is a rich history of print visualisation which remains relevant to the generation of online infographics: focusing on no more than 4 data points; avoiding 3D and ensuring the graphic is self-sufficient are just some.

Kaiser Fung’s trifecta is one useful reference point for ensuring a visualisation is effective, and this explanation of how a chart was transformed into something that could be used in a newspaper is also instructive (summarised by Kaiser Fung here).

In short: it’s not a simple process.

Visualisation has one major advantage which makes that effort worthwhile, however: it can make communication incredibly effective. And it can provide a method of distributing your content which cannot be matched by the other types of communication listed here.

But its major strength is also its main weakness: the instant nature of infographics also means that people often do not spend much time looking at it. It makes it very effective for distribution, but not for engagement, and so it is worth thinking strategically about 1) making sure the image contains a link back to its source; and 2) making sure that there is something more at the source when people arrive.

2. Narration

A traditional article can struggle to contain the sort of numbers that data journalism tends to turf up, but it still provides an accessible way for people to understand the story – if done well.

There are books providing useful guidance on how to write with numbers most clearly – and some guidance for web writing too (you should use numerals rather than words, as this helps people who are scanning the page).

As with visualisation, less is often more. But also, as in most narrative, you need to think about meaningfulness and your objectives in communicating these numbers.

Abstract amounts can be impressive, but meaningless and useless. What does it mean that £10m has been spent on something? Is that more or less than usual? More or less than something similar?

Try to bring down amounts to manageable quantities – the amount per person, or per day, for example.

Finally, use editing to focus in on the essentials: and make sure you link to the whole.

3. Social communication

Communication is a social act, and the success of infographics across social media is a testament to that. But it’s not just infographics that are social – data is too. The Guardian has demonstrated this particularly successfully with the cultivation of a healthy community around its Data Blog (which enjoys higher stickiness than the average Guardian article), and around its API.

Crowdsourcing initiatives aimed at gathering data can also provide a social dimension to the data. The Guardian are, again, pioneers here, with their MPs’ expenses project and Charles Arthur’s attempt to crowdsource predictions about the specifications of the iPad. But there are other examples, too – especially when it is difficult to obtain the data any other way.

The connectivity of the web presents new opportunities to present data journalism in a social way. ProPublica’s app that provides results based on your Facebook profile (schools attended; friends who have used the app) is one example of how data journalism can leverage social data, and, equally, how communicating the results of data journalism can be geared around social dynamics, using elements such as quizzes, sharing, competition, campaigning and collaboration. We are barely at the start of this aspect of online journalism.

4. Humanise

Broadcast news reports often use case studies to get around the problem of presenting numbers-based stories on television and radio. If waiting times have increased, speak to someone who had to wait a long time for an operation. In other words, humanise the numbers.

More recently the growth of computer-generated motion graphics has relaxed that pressure somewhat, as presenters can call on powerful animation to illustrate a story.

But once again, the point of making stories relevant to people comes through. As I wrote in One ambassador’s embarrassment is a tragedy, 15,000 civilian deaths is a statistic: when you move beyond scales we can deal with on a human level, you struggle to engage people in the issue you are covering – no matter how impressive the motion graphics (that post outlines some other considerations in humanising stories, such as ensuring that case studies are representative).

So after being buried in abstract data we need to remember that going out and recording an interview with a person whose life has been affected by that data can make a big difference to the power of our story.

5. Personalise

One of the biggest changes in journalism’s move online is that it opens up all sorts of possibilities around interactivity. When it comes to data journalism that means that the user can, potentially, control what information is presented to them based on various inputs.

There are some relatively well-established forms of this. For example, when a government presents its latest budget, news websites often invite the user to input their own details (for example, their earnings, or their family make up) to find out how the budget affects them. A recent variant of this are those interactives which invite the user to make their own decisions on how they might cut the deficit (the FT’s version took this further, adding in party strategies and policies).

Another common form is geographical personalisation: the user is invited to enter their postcode, zip code or other geographical information to find out how a particular issue is playing out in their home town.

A third is simply ‘your interests’, as demonstrated by Popvox’s approach to political engagement and the LA Times’ Newsmatch.

As more and more personal data is held by third party sites, the possibilities for personalisation expand. The ProPublica example given above, for example, demonstrates how Facebook profile information can be used to automatically personalise the experience of a story. And there are various apps that offer to present information based on location data provided via GPS.

This also indicates that there may be various ways in which personalisation and social strategies might be combined. Personalised stories can, in many ways, be used as an expression of our identity: this is where I live; this is how I am affected; this is what I’m interested in.

And when the COO of Facebook is predicting that all media will be personalised in 3-5 years, it’s clear that this is something the social networks are going to drive towards too.

6. Utilise

The most complex way of communicating the results of data journalism is to create some sort of tool based on the data. Calculators are popular choices, as are GPS-driven tools, but there is a lot of scope for more complex applications as more data becomes available both from the publisher and the user.

Again, there is overlap here with personalisation – but it is possible to provide utility without personalisation. And quite often, the complexity and consequent barrier to competitors presents commercial opportunities too.

At Reed Business Information, for example, their model is geared towards this sort of utility: attracting users at various points of the communication chain – online updates, printed magazines, mobile news – and steering them towards the point where they are closest to a purchasing decision. The idea is that the closer your information is to their action, the more valuable it is to the user.

Creating utility from data is currently relatively costly – but those costs are going down as a result of competition and standardisation. For example, as increasing numbers of news organisations adopt standard ways of storing story data (e.g. XML files), it is easier to create apps that pull data from datasets. Meanwhile, app creation becomes increasingly templated (in many ways you can see the process following a similar path to that of web design) and platform independent.

A medium up for grabs

What all of the above makes apparent – and I may have missed other methods of communicating data journalism (please let me know if you can think of any) – is that there are whole areas of online journalism that have yet to be properly explored, and certainly most have yet to establish clear conventions or ideas of best practice.

I’ve tried to scope out an overview of those conventions that are emerging, and the best practice that’s currently available, but it would be great if you could add more. What makes for good humanisation? Utility? What are great examples of personalisation or data journalism that involves a social dimension? Comments below please.

Meanwhile, here are both parts of the model shown together (click to magnify):

.

The starting point is actually quite a long way down the “how did you that?” chain, but I have to start somewhere, and the middle’s easier than the beginning, so that’s where we’ll step in (I’ll give some clues as to how the beginning works at the end…;-)

Here’s what we’ll be working towards: a diagram that shows how the people on Twitter that @wiredUK follows follow each other:

The tool we’re going to use to layout this graph from a data file is a free, extensible, open source, cross platform Java based tool called Gephi. If you want to play along, download the datafile. (Or try with a network of your own, such as your Facebook network.)

From the Gephi file menu, Open the appropriate graph file:

Import the file as a Directed Graph:

The Graph window displays the graph in a raw form:

Sometimes a graph may contain nodes that are not connected to any other nodes. (For example, protected Twitter accounts do not publish – and are not published in – friends or followers lists publicly via the Twitter API.) Some layout algorithms may push unconnected nodes far away from the rest of the graph, which can affect generation of presentation views of the network, so we need to filter out these unconnected nodes. The easiest way of doing this is to filter the graph using the Giant Component filter.

To colour the graph, I often make us of the modularity statistic. This algorithm attempts to find clusters in the graph by identifying components that are highly interconnected.

This algorithm is a random one, so it’s often worth running it several times to see how many communities typically get identified.

A brief report is displayed after running the statistic:

While we have the Statistics panel open, we can take the opportunity to run another measure: the HITS algorithm. This generates the well known Authority and Hub values which we can use to size nodes in the graph.

The next step is to actually colour the graph. In the Partition panel, refresh the partition options list and then select Modularity Class.

Choose appropriate colours (right click on each colour panel to select an appropriate colour for each class – I often select pastel colours) and apply them to the graph.

The next thing we want to do is lay out the graph. The Layout panel contains several different layout algorithms that can be used to support the visual analysis of the structures inherent in the network; (try some of them – each works in a slightly different way; some are also better than others for coping with large networks). For a network this size and this densely connected,I’d typically start out with one of the force directed layouts, that positions nodes according to how tightly linked they are to each other.

When you select the layout type, you will notice there are several parameters you can play with. The default set is often a good place to start…

Run the layout tool and you should see the network start to lay itself out. Some algorithms require you to actually Stop the layout algorithm; others terminate themselves according to a stopping criterion, or because they are a “one-shot” application (such as the Expansion algorithm, which just scales the x and y values by a given factor).

We can zoom in and out on the layout of the graph using a mouse wheel (on my MacBook trackpad, I use a two finger slide up and down), or use the zoom slider from the “More options” tab:

To see which Twitter ID each node corresponds to, we can turn on the labels:

This view is very cluttered – the nodes are too close to each other to see what’s going on. The labels and the nodes are also all the same size, giving the same visual weight to each node and each label. One thing I like to do is resize the nodes relative to some property, and then scale the label size to be proportional to the node size.

Here’s how we can scale the node size and then set the text label size to be proportional to node size. In the Ranking panel, select the node size property, and the attribute you want to make the size proportional to. I’m going to use Authority, which is a network property that we calculated when we ran the HITS algorithm. Essentially, it’s a measure of how well linked to a node is.

The min size/max size slider lets us define the minimum and maximum node sizes. By default, a linear mapping from attribute value to size is used, but the spline option lets us use a non-linear mappings.

I’m going with the default linear mapping…

We can now scale the labels according to node size:

Note that you can continue to use the text size slider to scale the size of all the displayed labels together.

This diagram is now looking quite cluttered – to make it easier to read, it would be good if we could spread it out a bit. The Expansion layout algorithm can help us do this:

A couple of other layout algorithms that are often useful: the Transformation layout algorithm lets us scale the x and y axes independently (compared to the Expansion algorithm, which scales both axes by the same amount); and the Clockwise Rotate and Counter-Clockwise Rotate algorithm lets us rotate the whole layout (this can be useful if you want to rotate the graph so that it fits neatly into a landscape view.

The expanded layout is far easier to read, but some of the labels still overlap. The Label Adjust layout tool can jiggle the nodes so that they don’t overlap.

(Note that you can also move individual nodes by clicking on them and dragging them.)

So – nearly there… The final push is to generate a good quality output. We can do this from the preview window:

The preview window is where we can generate good quality SVG renderings of the graph. The node size, colour and scaled label sizes are determined in the original Overview area (the one we were working in), although additional customisations are possible in the Preview area.

To render our graph, I just want to make a couple of tweaks to the original Default preview settings: Show Labels and set the base font size.

Click on the Refresh button to render the graph:

Oops – I overdid the font size… let’s try again:

Okay – so that’s a good start. Now I find I often enter into a dance between the Preview ad Overview panels, tweaking the layout until I get something I’m satisfied with, or at least, that’s half-way readable.

How to read the graph is another matter of course, though by using colour, sizing and placement, we can hopefully draw out in a visual way some interesting properties of the network. The recipe described above, for example, results in a view of the network that shows:

- groups of people who are tightly connected to each other, as identified by the modularity statistic and consequently group colour; this often defines different sorts of interest groups. (My follower network shows distinct groups of people from the Open University, and JISC, the HE library and educational technology sectors, UK opendata and data journalist types, for example.)
- people who are well connected in the graph, as displayed by node and label size.

Here’s my final version of the @wiredUK “inner friends” network:

You can probably do better though…;-)

To recap, here’s the recipe again:

- filter on connected component (private accounts don’t disclose friend/follower detail to the api key i use) to give a connected graph;
- run the modularity statistic to identify clusters; sometimes I try several attempts
- colour by modularity class identified in previous step, often tweaking colours to use pastel tones
- I often use a force directed layout, then Expansion to spread to network out a bit if necessary; the Clockwise Rotate or Counter-Clockwise rotate will rotate the network view; I often try to get a landscape format; the Transformation layout lets you expand or contract the graph along a single axis, or both axes by different amounts.
- run HITS statistic and size nodes by authority
- size labels proportional to node size
- use label adjust and expand to to tweak the layout
- use preview with proportional labels to generate a nice output graph
- iterate previous two steps to a get a layout that is hopefully not completely unreadable…

Got that?!;-)

Finally, to the return beginning. The recipe I use to generate the data is as follows:

1. grab a list of twitter IDs (call it L); there are several ways of doing this, for example: obtain a list of tweets on a particular topic by searching for a particular hashtag, then grab the set of unique IDs of people using the hashtag; grab the IDs of the members of one or more Twitter lists; grab the IDs of people following or followed by a particular person; grab the IDs of people sending geo-located tweets in a particular area;
2. for each person P in L, add them as a node to a graph;
3. for each person P in L, get a list of people followed by the corresponding person, e.g. Fr(P)
4. for each X in e.g. Fr(P): if X in Fr(P) and X in L, create an edge [P,X] and add it to the graph
5. save the graph in a format that can be visualised in Gephi.

To make this recipe, I use Tweepy and a Python script to call the Twitter API and get the friends lists from there, but you could use the Google Social API to get the same data. There’s an example of calling that API using Javscript in my “live” Twitter friends visualisation script (Using Protovis to Visualise Connections Between People Tweeting a Particular Term) as well as in the A Bit of NewsJam MoJo – SocialGeo Twitter Map.

I don’t know how good a statistician you need to be to distinguish between Anscombe’s quartet, but the differences are obvious to the eye:

Anscombe's quartet /via Wikipedia

Another shamistician (h/t @daveyp) heuristic (or maybe it’s a crapistician rule of thumb?!) might go something along the lines of: “if you use the right visualisations, you don’t necessarily need to do any statistics yourself”. In this case, the implication is that if you choose a viualisation technique that embodies or implements a statistical process in some way, the maths is done for you, and you get to see what the statistical tool has uncovered.

Now I know that as someone working in education, I’m probably supposed to uphold the “should learn it properly” principle… But needing to know statistics in order to benefit from the use of statistical tools seems to me to be a massive barrier to entry in the use of this technology (statistics is a technology…) You just need to know how to use the technology appropriately, or at least, not use it “dangerously”…

So to this end (“democratising access to technology”), I thought it was about time I started to play with R, the statistical programming language (and rival to SPSS?) that appears to have a certain amount of traction at the moment given the number of books about to come out around it… R is a command line language, but the recently released R-Studio seems to offer an easier way in, so I thought I’d go with that…

Flicking through A First Course in Statistical Programming with R, a book I bought a few weeks ago in the hope that the osmotic reading effect would give me some idea as to what it’s possible to do with R, I found a command line example showing how to create a simple box plot (box and whiskers plot) that I could understand enough to feel confident I could change…

Having an F1 data set/CSV file to hand (laptimes and fuel adjusted laptimes) from the China 2001 grand prix, I thought I’d see how easy it was to just dive in… And it was 2 minutes easy… (If you want to play along, here’s the data file).

Here’s the command I used:
boxplot(Lap.Time ~ Driver, data=lapTimeFuel)

Remembering a comment in a Making up the Numbers blogpost (Driver Consistency – Bahrain 2010) about the effect on laptime distributions from removing opening, in and out lap times, a quick Google turned up a way of quickly stripping out slow times. (This isn’t as clean as removing the actual opening, in and out lap times – it also removes mistake laps, for example, but I’m just exploring, right? Right?!;-)

lapTime2 <- subset(lapTimeFuel, Lap.Time < 110.1)

I could then plot the distribution in the reduced lapTime2 dataset by changing the original boxplot command to use (data=lapTime2). (Note that as with many interactive editors, using your keyboard’s up arrow displays previously entered commands in the current command line; so you can re-enter a previously entered command by hitting the up arrow a few times, then entering return. You can also edit the current command line, using the left and right arrow keys to move the cursor, and the delete key to delete text.)

Prior programming experience suggests this should also work…

boxplot(Lap.Time ~ Driver, data=subset(lapTimeFuel, Lap.Time < 110))

Something else I tried was to look at the distribution of fuel weight adjusted laptimes (where the time penalty from the weight of the fuel in the car is removed):

boxplot(Fuel.Adjusted.Laptime ~ Driver, data=lapTimeFuel)

Looking at the release notes for the latest version of R-Studio suggests that you can build interactive controls into your plots (a bit like Mathematica supports?). The example provided shows how to change the x-range on a plot:
manipulate(
plot(cars, xlim=c(0,x.max)),
x.max=slider(15,25))

Hmm… can we set the filter value dynamically I wonder?

manipulate(
boxplot(Lap.Time ~ Driver, data=subset(lapTimeFuel, Lap.Time < maxval)),
maxval=slider(100,140))

Seems like it…?:-) We can also combine interactive controls:

manipulate(boxplot(Lap.Time ~ Driver, data=subset(lapTimeFuel, Lap.Time < maxval),outline=outline),maxval=slider(100,140),outline = checkbox(FALSE, "Show outliers"))

Okay – that’s enough for now… I reckon that with a handful of commands on a crib sheet, you can probably get quite a lot of chart plot visualisations done, as well as statistical visualisations, in the R-Studio environment; it also seems easy enough to build in interactive controls that let you play with the data in a visually interactive way…

The trick comes from choosing visual statistics approaches to analyse your data that don’t break any of the assumptions about the data that the particular statistical approach relies on in order for it to be applied in any sensible or meaningful way.

[This blog post is written, in part, as a way for me to try to come up with something to say at the OU Statistics Group's one day conference on Visualisation and Presentation in Statistics. One idea I wanted to explore was: visualisations are powerful; visualisation techniques may incorporate statistical methods or let you "see" statistical patterns; most people know very little statistics; that shouldnlt stop them being able to use statistics as a technology; so what are we going to do about it? Feedback welcome... Err....?!]

Tweetminster maintains separate lists of tweeting journalists for several different media groups, so it was easy to grab the names on each list, use the Twitter API to pull down the names of people followed by each person on the list, and then graph the friend connections between folk on the lists. The result shows that the hacks are follow each other quite closely:

Nodes are coloured by media group/Tweetminster list, and sized by PageRank, as calculated over the network using the Gephi PageRank statistic.

The force directed layout shows how folk within individual media groups tend to follow each other more intensely than they do people from other groups, but that said, inter-group following is still high. The major players across the media tweeps as a whole seem to be @arusbridger, @r4today, @skynews, @paulwaugh and @BBCLauraK.

I can generate an SVG version of the chart, and post a copy of the raw Gephi GDF data file, if anyone’s interested…

PS if you’re interested in trying out Gephi for yourself, you can download it from gephi.org. One of the easiest ways in is to explore your Facebook network

PPS for details on how the above was put together, here’s a related approach:
Trying to find useful things to do with emerging technologies in open education
Doodlings Around the Data Driven Journalism Round Table Event Hashtag Community.

For a slightly different view over the UK political Twittersphere, see Sketching the Structure of the UK Political Media Twittersphere. And for the House and Senate in the US: Sketching Connections Between US House and Senate Tweeps

The data set I’m using is a partial one, taken from the Guardian Datastore: Tuition fees 2012: what are the universities charging?. (If you know where there’s a full list of UK course fees data by HEI and course, please let me know in a comment below, or even better, via an answer to this Where’s the fees data? question on GetTheData.)

My first thought was to go for a proportional symbol map. (Does anyone know of a javascript library that can generate proportional symbol overlays on a Google Map or similar, even better if it can trivially pull in data from a Google spreadsheet via the Google visualisation? I have an old hack (supermarket catchment areas), but there must be something nicer to use by now, surely? [UPDATE: ah - forgot this: Polymaps])

In the end, I took the easy way out, and opted for Geocommons. I downloaded the data from the Guardian datastore, and tidied it up a little in Google Refine, removing non-numerical entries (including ranges, such 4,500-6,000) in the Fees column and replacing them with minumum fee values. Sorting the fees column as a numerical type with errors at the top made the columns that needed tweaking easy to find:

The Guardian data included an address column, which I thought Geocommons should be able to cope with. It didn’t seem to work out for me though (I’m sure I checked the UK territory, but only seemed to get US geocodings?) so in the end I used a trick posted to the OnlineJournalism blog to geocode the addresses (Getting full addresses for data from an FOI response (using APIs); rather than use the value.parseJson().results[0].formatted_address construct, I generated a couple of columns from the JSON results column using value.parseJson().results[0].geometry.location.lng and value.parseJson().results[0].geometry.location.lat).

Uploading the data to Geocommons and clicking where prompted, it was quite easy to generate this map of the fees to date:

Anyone know if there’s a way of choosing the order of fields in the pop-up info box? And maybe even a way of selecting which ones to display? Or do I have to generate a custom dataset and then create a map over that?

What I had hoped to be able to do was use coloured proportional symbols to generate a two dimensional data plot, e.g. comparing fees with drop out rates, but Geocommons doesn’t seem to support that (yet?). It would also be nice to have an interactive map where the user could select which numerical value(s) are displayed, but again, I missed that option if it’s there…

The second thing I thought I’d try would be an interactive scatterplot on Many Eyes. Here’s one view that I thought might identify what sort of return on value you might get for you course fee…;-)

Click thru’ to have a play with the chart yourself;-)

PS I can;t not say this, really – you’ve let me down again, @datastore folks…. where’s a university ID column using some sort of standard identifier for each university? I know you have them, because they’re in the Rosetta sheet… although that is lacking a HESA INST-ID column, which might be handy in certain situations… ;-) [UPDATE - apparently, HESA codes are in the spreadsheet.... ;-0]

PPS Hmm… that Rosetta sheet got me thinking – what identifier scheme does the JISC MU API use?

PPPS If you’re looking for a degree, why not give the Course Detective search engine a go? It searches over as many of the UK university online prospectus web pages that we could find and offer up as a sacrifice to a Google Custom search engine ;-)