Thursday, March 11

• Big Data: Analyzing legislation with machine learning. Always good to hear what Chase Davis has been up to.
• Open Source GIS. Now that mapping is more and more accessible, it pays to stay on top of what people are using.
• Juice up your stories with advanced methods. After a few months in the academy, Sarah Cohen should have some good stuff to share.
• Some lightning talks!

Friday, March 12

• Semantic tagging and DocumentCloud. Really need to get more in-depth on this.
• New Frontiers in Reporting Tools. More of the New New Stuff.
• Forensic Accounting for Reporters. Nice to have some outside expertise.

Saturday, March 13

• GeoDjango & OpenLayers. Yes, I’m on this panel. But you should come anyway to hear from Ben Welsh of the LA Times.
• Not a programmer? Not a worry. The new software from ProPublica that helps publish data on the Web. Very interested to see this in action.

There’s also the Django bootcamp and plenty of opportunities for demos, discussions and debates. Hope to see you there!

App Engine’s Python runtime, unsurprisingly, sticks pretty close to the language’s core tenets: it uses YAML files for configuration (hey, it’s whitespace!) and can run pretty much an entire app using just 2-3 files. A NYT colleague, Derek Gottfrid, built a sample app for our article search API comprising five files, including the README. Yes, it violates the separation of logic and design that most frameworks try to respect, but it works.

GAE provides the basic building blocks a lot of Web apps need, nearly all optional: a backend in Datastore, a URL Fetch service that is wrapped by Python’s familiar urllib and urllib2 libraries, mail and messaging services and memcache. Webapp is a basic framework for building apps not exactly like Django but not so unfamiliar, either.

The development server will be familiar to anyone who has tinkered with Django, and GAE handles static files via separate servers, which is how it should be. And since it comes with a version of Django built-in, you can bring along some handy utilities, like simplejson, with a single import statement. And as I said earlier, you don’t have to separate display logic into template files, but you can, and the syntax is nearly identical to Django templates.

The sample app takes two random members of the Senate and compares their voting and bill sponsorship records in the 111th Congress. The app’s code is like the app itself: fairly tightly-focused and without a lot of trappings. It’s just service calls to the API and a single template for display. In building it, I didn’t make use of any persistent storage, so I didn’t delve into Datastore, but it looks pretty useful. One of its helpful features is that as you develop your app, it generates indexes used to help return data in the most efficient manner.

If you’re already familiar with Django, making the small step to App Engine isn’t that big of a trip. Have Guido explain things to you first, and then try it out. You can also run a stripped-down version of Django on GAE, and I’m looking to see if there’s a project I can adapt to try it out. In the meantime, if you want to tinker with the sample app, by all means fork it and see what else you can do with the API. And let me know what you come up with!

A staple of technical conferences, lightning talks are based on the notion that while 45-50 minutes presentations are good, sometimes you only need about 5 minutes to express an idea or show off an example of something interesting or useful. And I’m pretty sure, based on sessions at the hotel bar at past conferences, that there are plenty of ideas out there. Attracting and organizing them is the goal of this app built by my boss, Aron Pilhofer, and launched today.

This is a new thing for the CAR community, but I think it could be a useful way to draw out ideas from people who aren’t doing panels or who might shrink from a 50-minute session. So let’s get started. Want to give a talk?

(Yes, I know I shouldn’t be running beta software in production, but even I have to say that Fumblerooski isn’t “mission-critical” software. Not yet, at least.)

Awhile back, Jeff Self suggested that he’d like to see head-to-head records for head coaches – how one person has fared against another. Me too, I thought. But how best to do that? In Fumblerooski’s models, it’s not a simple query, since a Coach has many CoachingJobs and CollegeCoach assignments. Plus the Game model refers to the individual College, not a specific head coach. So I added head coaches to the Game model and some utilities to populate those fields. That work continues, especially for people who were head coaches earlier in the decade but are not now. That made it easy to grab games in which two specific head coaches faced off.

But to create a list of head coaches that Mack Brown has faced since 2000, I turned to one of Django newest features: improved raw SQL querying. Much like ActiveRecord’s find_by_sql method for Rails, Model.objects.raw() makes it easy to write custom SQL when you need to, and still be able to access instantiated objects as the result.

You can select only those fields you need, deferring others, or pass parameters into the SQL. And you can add additional calculated fields, too. So when I needed to pull in a QuerySet of distinct coaches that Mack Brown had faced, plus the number of games, I could do something like this:

coach = Coach.objects.get(last_name='Brown', first_name='Mack')
coach_list = Coach.objects.raw(
"SELECT college_coach.id, college_coach.slug, count(college_game.*) as games
from college_coach inner join college_game on college_coach.id = college_game.coach2_id
where coach1_id = %s group by 1,2 order by 3 desc", [coach.id])

And thus you can see head coaching opponents for Mack Brown (or another coach). The idea isn’t new, but I really like the implementation – it’s clean and makes the mixing of Python code and SQL about as seamless as it could possibly be.

For a “15-minute project“, Simon Willison’s geocoders library is pretty handy if you’re doing geocoding with Python. It offers a common interface to the geocoding services provided by Google, Yahoo and other sources. When we were looking at replacing the home-grown geocoding system that Andrei Scheinkman built for Represent, Simon’s project seemed a natural choice.

It was an easy drop-in, but there was one thing about it that was just slightly off. A successful geocoding result looks like this:

(u'New York, NY, USA', (40.756053999999999, -73.986951000000005))

Notice the coordinate pair is latitude, longitude. For folks using GeoDjango alongside Simon’s library, the way you build a Point object from coordinates is to pass the longitude first, like so:

>>> from django.contrib.gis.geos import Point
>>> p = Point((5, 23)) # 2D point, passed in as a tuple

So on Friday I forked Simon’s project and reversed the ordering of the coordinates in a successful result. That way you can pass that portion of the result directly to a Point constructor:

>>> from django.contrib.gis.geos import *
>>> from geocoders.google import geocoder
>>> geocode = geocoder('GOOGLE-API-KEY')
>>> results = geocode('new york')
(u'New York, NY, USA', (-73.986951000000005, 40.756053999999999))
>>> pnt = Point(results[1])

Not a huge deal, but in keeping with the spirit of library, I think.

That constraint – which still exists on the web, albeit it less so than for print publications – makes it easier to justify working with hundreds of thousands or millions of rows of data to build an interface that allows readers to find out about polluters or drinking water systems close to them.

These kinds of apps aren’t easy; the hardest part is the interface, for which Tyson Evans and Matt Bloch deserve the credit for the water series. But the difference between those apps and the “data ghetto” kind of app that provides a search box and not much else is more than time and talent. In some cases, it’s less a “nice-to-have” than a must-have. Earlier this month, Duhigg and Griff Palmer wrote a powerful piece showing that millions of Americans are drinking contaminated water. A great story, but check out the first dozen or so comments by readers. Half of them asked for a list of drinking water systems or a map or some other way to find out if their local systems were providing unhealthy water. That app accompanied the story published later in December.

Readers aren’t dumb; they want to know what we know, and they know that the web makes it possible for us to share with them at a national and even local level. The level of commitment and effort that we put into responding to their need for relevant and meaningful information will go a long way towards building a better relationship with them. The kind of web application that provides a summary of millions of records requires a different approach from, say, a lookup of government employees. But that doesn’t necessarily mean less effort and thought, just a willingness to treat both in a manner that respects the unique characteristics of each. So as we are in the season of gift-giving, all of us – myself included – need to think more about what we’re really giving to our readers when we post data on the web.

The trouble is that IRE is a non-profit organization tied to an industry that increasingly has cut back on spending for training, travel and other “luxuries”. So while attendance at this year’s IRE conference in Baltimore was very strong, a lot of folks there were paying their own way. Same deal for the annual computer-assisted reporting conference in Indianapolis. That’s simply not sustainable, given IRE’s current orientation towards providing hands-on training and data services to news organizations. My IRE membership will be pried from my cold, dead hands (and you should join, too), but attracting new members and offering them the kinds of training and services they’ll need will be increasingly difficult.

If you ask me, IRE needs to reorient its training and service offerings to take advantage of the distributed nature of the Web and the broadening of acts of journalism. Yes, hands-on training must continue, but do trainers need to travel all the time? What about video-based training? Yes, IRE should collect the best expertise of its members, but the age of the tipsheet alone is gone. We have so many other options: screencasts, podcasts, YouTube – hell, even Google Wave – to deliver the kind of knowledge that is the lifeblood of IRE.

Crucial to that effort is the recognition that as the potential base of members and users of IRE services expands, so too the need for individual training modules. Yes, IRE should still offer a 5-day bootcamp on computer-assisted reporting. But it also should offer half-day refreshers on SQL, or 10-minute screencasts on a useful Excel function. Look at what PeepCode does – and I don’t think IRE would need to have such high production values to be valuable – and you’ve got an idea of what I’m talking about. IRE members are some of the leading experts in journalism on subjects such as the Census, mapping and various obscure datasets. Yet the only option for purchasing audio from conferences is all or nothing.

Similarly, the Resource Center needs a good update. IRE books, which are tremendously useful but can have a short shelf-life, need to be sold in print and revised online to remain attractive and lower the costs of doing new editions. Future tipsheets should be digital-only, and categorized not just by keyword but also by speaker. If I want to overdose on Paul Overberg’s Census material, I should be able to do that without searching (they should also be sold to journalism schools for classes). Nearly all training exercises should be available online – this would require greater standardization, but that’s not a terrible thing – and members or paid users should be able to schedule video/chat time with an IRE trainer or volunteer as a follow-up.

While we’re at it: Uplink is, well, I don’t think even IRE knows for sure, but it definitely isn’t working. Simplify it. Give it a narrow mission. Make it easier to find the expertise that currently is spread across email, blog posts, tipsheets and tutorials. The value of the organization lies in the ability of its staff and volunteers to intelligently organize and disseminate the unique and valuable information it has within it.

All of these things require changes to the way IRE currently works, and that’s the tough part; it’s hard to argue that IRE hasn’t been doing good work, and I am definitely not making that claim. What I’m saying is that as both the market for IRE’s services and the methods for delivering them undergo some significant changes, it’s time for IRE to meet those changes head-on.

This all started back at the NICAR conference in Indianapolis where several of us led a mini-bootcamp on using Django. At one point, as we talked about how projects are organized, I showed off the models for Fumblerooski. They went on forever. Looking back, it wasn’t the message that I wanted to get across – I think several people gasped.

Fumblerooski still is far more tightly coupled together than I’d like – the scrapers can’t really be separated out as an independent app, which would be the right thing to do. But it’s getting closer. Same for the rankings app. Coaches could be the next one, or maybe players. The scrapers, even though they don’t constitute an actual app, are better organized. The point is that now the code is probably easier for someone else to follow, but it’s also easier for me to locate specific functions. I spend less time hunting and more time actually doing things.

How does this actually work? Python’s ability to import libraries into others means that Django apps can share common code (and, if you’re working in the same database, data) inside a single project just by typing an import statement:

from fumblerooski.rankings.models import RushingSummary

And I get access to the rushing rankings wherever I need to use them. Because this is so trivial, it sometimes led me to think that where I put things didn’t matter. But it does, it really does, for your sake and the sake of anyone who attempts to look at your code.

First, let me say that my observations about the general history of WPNI and its relationship with the paper are colored by my own experiences, but I agree with folks like Jay Rosen who say that at one point, washingtonpost.com was clearly a national leader – not just in technical capability, but in the kind of mindset necessary for a news organization prepared to take advantage of the Internet’s possibilities. I supported the creation of WPNI as a separate operation, to allow it more creative freedom, but both the people of WPNI and their colleagues at the Post should have done more to foster a better environment for working together. It’s something that I failed at when I was there.

But back to the kind of environment that leads to the departures, voluntarily or otherwise, of so many talented and dedicated employees. I don’t know the people who currently run the Washington Post, but I do think I understand a bit about how the organization works and thinks, having spent about three years there (more than two at the paper and about nine months at WPNI). When I wrote about moving from the paper to the website back in 2007, I left out some details about how that process happened. And I think, in hindsight, that they shed some light on how the organization operates.

It’s true that there were a number of people at the Post who were supportive and encouraging of my ambitions to work at WPNI. Among them were my supervisor at the time, Lucy Shackelford, and the paper’s editor, Len Downie. But once I had seriously pursued the idea of working on the website, it took months for the move to happen, and not just for reasons of simple corporate bureaucracy. In a very real way, my transition was held up – I (jokingly at first, and then angrily) referred to it as a filibuster or a senatorial hold – by a few people at the paper. These people, most of whom no longer occupy the positions they held then, are not stupid. They are among the smartest folks I’ve ever worked with, and I have a high regard for their journalistic abilities. But the thinking that caused the editor of the paper to become involved in whether a mid-level staffer moved to the website was, in essence, this: this is a bad idea, because it will hurt the paper. My ego might like to think that this was really true, but I think the reality is that these people could not compare the value of my work for the website to the paper because they did not understand what it is I wanted to do. So they went with what they knew, and that seemed to be a net deficit for them. And thus it was that I mooted the option of simply resigning from the paper in order to join its website.

I don’t envy the people who run the Washington Post (or any news organization) today. They have a ton of thankless choices to make, and critics on every side. From a certain standpoint, I can appreciate the idea that the paper edition, which generates the overwhelming share of the revenue, should be protected and bolstered as much as possible. But I cannot agree with the idea that this means that you take employees who have proven expertise doing valuable and informative things that don’t always translate into print and cannibalize (or toss away) their talents for the sake of the paper.

My fear as a Washington Post subscriber and reader of washingtonpost.com is that, when the folks running the organization turn things around (and I believe that it is not an impossibility or even a long-shot), what emerges will be not only a news organization that is a shadow of its former self – most orgs will have to face that reality – but that it will have put so much emphasis on the paper that it cannot take advantage of the possibilities online. That the folks running things are literally rolling back the progress and smart work that has been done, and will not be able to get it back as fast as they might think. And the people who remain – those who will be charged with the task of rebuilding a news operation that embraces all of the ways that its readers and users can gain value – will have neither the support nor the depth to make it happen.

The intriguing title of a recent report from scholars at Duke is “Accountability Through Algorithm: Developing the Field of Computational Journalism“. Semi-related to CAR, Computational Journalism is defined as “the combination of algorithms, data, and knowledge from the social sciences to supplement the accountability function of journalism.” I take each of those – algorithms, data and knowledge from the social sciences – as separate elements, because while journalists do have plenty to learn from the social sciences, we also operate in an environment that is not quite academic (and sometimes not at all).

The report identifies four areas of potential exploration: techniques for data transformation and pattern discovery in investigative reporting; a digital “dashboard” for journalists; new social and technical structures for interactions among readers and reporters; and sense-making advances from other disciplines. All are interesting and worthy, but to me the first two are particularly so.

On the first, the best investigative journalists have been developing tools for extracting meaning from reams of information for years. The change now is that we have a greater platform for these tools in the Internet, and an effort like DocumentCloud is a clear example of that change. The challenge we face is that patterns are interesting to different people for different reasons; what an accountant finds interesting may not always be of interest to a journalist, and vice versa. The current deficit is not in the area of tools; it is the occasionally trickier area of adapting those for the task of journalism. That requires the guiding influence of people like Sarah Cohen, a newly minted Knight Chair at Duke, who is studying these issues right now. But it also requires the active participation of a wide range of news organizations and journalists. In the Internet, we have a leveling platform, but only if more journalists participate. That may be a greater challenge than the technical one.

One way to get there is the second idea – a journalist’s dashboard. This would provide reporters with a way to keep track of the deluge of information coming into newsrooms. But again, the technological side of that equation, as difficult as it is, is less of a concern to me than the implementation and adoption of the results. We know how to gather various bits of information in one place. We’re not that good at distilling the best of them, or even knowing where to start. The good news is that we have blueprints for this kind of thing: the people and companies who make great Web apps that distill masses of data into understandable results. The bad news is that we, as a business, work very differently. We don’t really share much, outside of experiences at conferences or over drinks, and particularly not at the institutional level. And we’re downright awful, in general, at adapting good ideas for our own uses.

For the idea of Computational Journalism to work, a lot is riding on a movement that is slowly growing but urgently necessary for the news industry: the increasing adoption, use and proliferation of open-source tools. The CAR community has seen an influx of use of various types of open-source software, from databases to GIS systems to web frameworks. More and more reporters and editors are embracing different styles of journalism. But the broader concept of opening up our newsrooms, both philosophically and in terms of our content and efforts, has been slow in coming. It requires not just the creation of tools, but also the development of journalists and readers who will use those tools most effectively. And that’s more than an algorithm – to say nothing of Twitter – can solve alone.

Oh, and Duke folks? Can we get a version of that report that embraces the Web as much as the concept? HTML will do fine.