Justin Marney

Getting XML Into Your Processing Sketch

During one of the recent Learning Processing sessions, Keith asked how one would go about getting data from a web service, such as YQL, into a Processing sketch. After poking around the Processing library reference we were able to create an example of how to pull in data from a web service and use it to control the parameters of a sketch.

It turns out getting XML data into Processing from a web service is pretty easy. The first thing we need to do is import the built-in XML library.

import processing.xml.*;

Next we construct our query string using a YQL query that returns the most interesting flickr photos. You can take a look at the result here select * from flickr.photos.interestingness(10). One of the cooler aspects of XMLElement is that you can initialize an object with a url and it will take care of making the http request and fetching the XML.

String query = "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20flickr.photos.interestingness(" + count + ")&format=xml";
XMLElement xml = new XMLElement(this, query);

There are two children returned in the XML: diagnostics and results. We want to examine the data in results, so we get that block of XML via the getChild method. Children are indexed starting at 0, so the second child is at index 1.

XMLElement results = xml.getChild(1);

At this point we can stop and party for a while because we have technically achieved our goal. We have pulled some data into our sketch and it is sitting inside our XMLElement object. The next step is to iterate through the data and do something with it.

Before we started to code we took some time to analyze the data and figure out a few different ways to display it. I noticed that with each record flickr returned a server id. We assumed this identified the server that contains the picture. I was curious as to how distributed the results were and we decided to write a sketch based on this aspect of the data. One of the most interesting things to do in Processing is to find new ways of looking at data in order to expose hidden characteristics.

The first thing we'll do is iterate through the records using a for loop. The getChildCount method allows us to define the for loop conditions and we use getChild inside the loop to access individual results. The getStringAttribute lets us get specific attribute values from a particular record so we'll use that to get back the server id.

for(int i = 0; i < results.getChildCount(); i++) {
  XMLElement result = results.getChild(i);
  String server = result.getStringAttribute("server");
}

We tie all together by first checking to see if we've created an entry in our HashMap. If we haven't, we create one and put a 1 in the counter. If we find an entry already exists, we increment the counter stored at that key and save it.

HashMap hm = new HashMap();
for(int i = 0; i < results.getChildCount(); i++) {
  if(hm.get(server) == null) {
    hm.put(server, 1);
  } else {
    Integer sum = (Integer)hm.get(server);
    sum += 1;
    hm.put(server, sum);
  }
}

When the loop finishes running we'll have a HashMap with keys that correspond to server ids and values that contain the number of images that server holds. We took that data and made a simple processing sketch that creates ellipses which are larger and more red if they are serving more images. In the end, the sketch turned out to be a strange sort of load monitoring visualization for flickr. You can get the code for this sketch here. Here is the full code for our HashMap creation method.

import processing.xml.*;

HashMap serverCounts(int count) {
  String query = "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20flickr.photos.interestingness(" + count + ")&format=xml";
  XMLElement xml = new XMLElement(this, query);
  XMLElement results = xml.getChild(1);

  HashMap hm = new HashMap();
  for(int i = 0; i < results.getChildCount(); i++) {
    XMLElement result = results.getChild(i);
    String server = result.getStringAttribute("server");
    if(hm.get(server) == null) {
      hm.put(server, 1);
    } else {
      Integer sum = (Integer)hm.get(server);
      sum += 1;
      hm.put(server, sum);
    }
  }
  return hm;
}