First, I stand by what I said about how little we understand about the way that elections are won or lost these days. It’s not that political journalism has strayed from its roots, or stopped covering important elements of a modern campaign. It’s that the elements of a modern campaign have changed, and as journalists, we have not kept pace.

You might respond that campaigns still involve quite a lot of stuff that we do understand, such as debates and visits to state fairs and town hall meetings. True. But the nature of media and technology has brought extensive changes to the electoral system, and I don’t believe that we as journalists devote enough attention to understanding those changes. Remember the Dean campaign in 2003? Most of the coverage was on the, for then, staggering online fundraising managed by some doctor from Vermont. But that Wired piece I referenced had it right; Dean’s accomplishment was less a mastery of the Internet but a willingness to embrace its fundamental aspect: you give up some control by bringing other people in, and you gain a host of possibilities. You may, of course, choose badly or falter in some other way, but the lessons and possibilities are becoming clear. At the time, as a Web geek who loved politics, I felt that journalists couldn’t really explain the Dean campaign, because it was so alien to us. Today’s campaigns make me long for the simplicity of 2003.

But let’s stick with fundraising for a bit. Political fundraising can be hugely expensive, because campaigns need to amass large number of donors. Unless you’re the President, it’s hard to repeatedly gather the wealthiest Americans and have them fork over $2,500 or more for the pleasure of your company. So a smart campaign sticks with what works: direct mail is costly, for example, but it’s also effective. Telemarketing takes time and money, but it also works pretty well. Let’s not mess with the script too much. But what if you can mess with the script? Now it’s possible, even trivial, to experiment with Web site design or even advertisements in order to gauge their effectiveness and improve upon them. President Obama had a Director of Analytics for his 2008 campaign, and has been hiring data scientists experienced in predictive modeling.

White men smoking cigars in cramped rooms making gut calls is how we’ve usually understood campaign decision-making. This? Whole new ballgame. Yes, there is still a mass audience that is shaped by the media and big events. But there are now thousands and thousands of “small” audiences – or rather, they always were there. Now campaigns can identify them and deliver precision messages to them. And they can find them online in different ways; an hour after posting on Twitter about the Obama’s campaign use of Github, the campaign’s Digital Director was following me. And that’s the easy part.

While campaigns have a public presence that is mostly recorded and observed, the stuff that goes on behind the scenes is so much more sophisticated than it has been. In 2008 we were fascinated by the Obama campaign’s use of iPhones for data collection; now we’re entering an age where campaigns don’t just collect information by hand, but harvest it and learn from it. An “information arms race,” as GOP consultant Alex Gage puts it.

For most news organizations, the standard approach to campaign coverage is tantamount to bringing a knife to a gun fight. How many data scientists work for news organizations? We are falling behind, and we risk not being able to explain to our readers and users how their representatives get elected or defeated.

None of this is to say that we need to completely abandon our ways of covering elections. Horse-race coverage is and should be a part of campaign coverage, because in many respects elections are like horse races. Things can change rapidly, and small things can have big impact. We still should be on the ground, talking to voters, showing up at town halls and covering debates. We still need to show up and do the legwork.

But if we can’t appreciate, much less understand, what modern campaigns are doing to win elections, how can we hope to explain elections? If we don’t collect at least some of the information available to us – realizing that we can’t get our hands on everything that the campaigns do – we’ll miss the story. Elections will become even bigger surprises to us, and then how long will it be before readers start to ask whether we actually know the people and places we cover?

Surprises make the news. Some of my favorite stories from the 2004 presidential election are in a book by my friends Peter Wallsten and Tom Hamburger, then of the Los Angeles Times. Here’s one anecdote from the key state of Ohio:

One suburban African American woman in Ohio, for example, told us that though she tends to vote Democratic, she was deluged in 2004 with calls, e-mail messages and other forms of communication by Republicans who somehow knew that she was a mother with children in private schools, an active church attendee, an abortion opponent and a golfer.

Think about what this kind of thing means. It means that we cannot assume that the campaign visible to the mass audience is the same campaign that’s being pitched to individuals and groups around the nation, and that winning coalitions can be built not just by harnessing large groups (unions, religious voters, etc.) but also by piecing them together in small units. President Bush’s margin in Ohio in 2004? About 2.5 percent. The only thing that I don’t like about this anecdote is that Wallsten and Hamburger’s book appeared nearly two years later. Is there any evidence that we as journalists have closed the gap since then?

To understand how elections are now being waged, we need to have as many of the tools as do the campaigns. We need to build our own storehouses of data – voter registration, voter history, Census, campaign finance, advertisements and more. We need to be able to tap into the rich stream of material that’s being created and disseminated every day. We need to be able to see the value in small data points that can lead to bigger things.

Elections are great stories. They deserve to be told from a position of confidence and knowledge. We have work to do.

Done snickering? Let’s move on.

The Tools Team built mostly internal applications and services that helped the Web site run better. I mainly got to work on front-facing projects like the Congress Votes Database, the 2008 presidential campaign and an innovative series on lobbyist Gerald Cassidy. But I did work on a few internal tools, and since I joined The Times in late 2007 I’ve built a few more. I’ve found that such tools are not so different from what we now consider to be journalism by Web development. Chosen wisely and done well, they can have impacts that go far beyond a single story or series. We should not dismiss them as “not journalism.”

If you’re at the geekier end of the journalism spectrum, then chances are your colleagues know about the stuff you can do. They may not understand it or be able to explain it; a former managing editor of mine, when told about the various technical steps to accomplish something useful, would invariably respond with a touch of wonder: “Fuckin’ Internet!” You can explain your work to a decent percentage of your colleagues by invoking Harry Potter or the Lord of the Rings and leave it at that.

But that doesn’t mean that building tools that can be used by broad segments of the newsroom is a one-way street or has to lead to a divide between you and the other journalists. There will be people in every newsroom who mainly take and rarely give, and in those situations being a technologist is no different from being a clerk. Good tools, like good apps, are a product of collaboration and improve the ability of the newsroom in general. They also make for more and better apps.

Case in point: At The Times we have an Inside Congress app that displays information about votes and bills in Congress. The tool that underlies that app is enormous – it has tons more information, and we’re working to surface more and more of it. But the tool – an internal interface – has uses for our congressional reporters, our graphics editors and for me as a developer. I can point a reporter to the vote record comparison tool instead of having to run a database query or, worse, asking someone else to manually recreate something. We use the tool as a sort of canary in the mine to alert us to odd or interesting events, from committee assignment changes to bill sponsorship withdrawals to unusual voting patterns. In some cases, having the data internally has led to improvements in the app itself, such as our “key amendments” pages for certain bills. I didn’t think of that, but someone else who saw the internal tool did, and we built it together.

Perhaps most important to me as a developer, building the internal tool has broadened the number of people I work with and has given me a range of ideas for making apps easier to build and better. Not all of them pan out, but some of them do. Put another way: the tool actually helps me develop closer working relationships with my colleagues.

A good tool doesn’t just make it easier for a reporter to create a story. It actually seeds the story, or makes it possible for more people in a newsroom to collaborate. When you have data but no tool, you become a gatekeeper of a sorts – which is appropriate in many circumstances, but not all. I can’t possibly know what my colleagues are thinking about, considering or being alerted to, but I can make it easier for them to test out theories and do some exploration on their own. Some of them prefer to do their own work, and we certainly miss some opportunities for apps that way. But others consult with me quite a bit, since they now have a much better idea of what we have and what we might be able to do with it.

Skeptics might respond that there is a difference between tools built around journalistic content, like the Congress app, and those that “merely” solve a technical problem. This is a short-sighted argument. What we do as builders of Web applications (external or internal) is informed by everything we touch. Pulling a piece of one tool for use someplace else is a useful technique because it reinforces the value of not repeating yourself and because it sometimes enables you to look at an old problem or situation from a new vantage point.

Back at washingtonpost.com, my former colleague Adrian Holovaty liked to say that we didn’t build internal versions of our apps because the public version was the internal version. Fair enough, to a point, but I think that line can veer into the data ghetto when not rigidly policed.

Most of my colleagues, I’m confident, have very little idea what it is that I specifically do. Sometimes I spend the time educating, and sometimes I let our tools help with the evangelization process. However they see my work, I’m pretty happy as long as it contributes to our journalism together. App developer? Sure. Tool maker? Why not. Labels don’t interest me much, and most of my colleagues don’t seem to care. The results – the journalism – are what matter.

You might ask, “How is it that you’ve managed to keep a job in journalism, then?”

There’s no single answer to that, although the majority of it would have to be everything I learned from being a member of Investigative Reporters & Editors. And of that part, what I’ve really learned to love and work at is the other kind of interviewing. The one you don’t hear much about in journalism school: interviewing data.

To be fair, you really don’t hear all that much about the craft of interviewing people at journalism school, either. There is the occasional class, but the way that most people I know get better at it is simply by doing. When people ask me how I can approach complete strangers and ask them detailed and occasionally personal questions, I’m quick to reply that I spent four summers delivering breakfast in bed to newlyweds in the Poconos. When you’ve had a naked man answer the door at 8 a.m. and tell you to put the trays down next to the tripod-mounted video camera, talking to evenly partially-clothed strangers gets pretty easy.

Interviewing data takes practice, too, although I can’t really find a parallel from my days waiting tables. Both kinds of interviewing have much in common: you want to be as prepared as possible so as to better evaluate the results and be able to adapt your questions to the situation. Both require you to place a solid block of skepticism, even suspicion, on your shoulders as you embark. And both, if done well, can result in an unexpected admission – something even the subject of the interview didn’t really “know”.

This is why I continue to teach spreadsheets in classes, because they make for excellent initial interview tools. Looking at some data in a spreadsheet, you can easily size it up with basic sorting and filtering. That’s kind of the “getting-to-know-you” phase of the data interview. What are the ranges of this data? What looks unusual? Just as you get first impressions upon meeting someone, you get similar feelings about data.

With data you have to ask all the basic questions you do with a person, just so you know exactly what you’re dealing with. Questions like: “How old are you?”, “Where were you born?”, “Who do you report to?” work for both people and data (although I suppose “made” is a better word than “born”). And then, once you’ve got a solid foundation, you ask the trickier questions, the ones that you need to really think about. The ones that, when you’re planning a big interview with the subject of your investigation, you game-plan and write out as if they were lines in a soap opera.

And that’s where the big difference is: with data, you can ask a lot of potentially embarrassing questions, and the data won’t complain, walk out or threaten to sue. You can ask variations of the same question 20 times and the data won’t mind. When I say that I prefer interviewing data to people, this is why. Data will only lie to you if it’s just bad data or if you misunderstand the question. Unfortunately, almost every data set is “bad” in some way. But once you find that out, you usually can deal with it.

With the increased availability of information in structured forms, the skill of interviewing data is even more valuable now that it has been in the past. And yet it’s still considered a niche, a specialty skill. It’s odd, because what makes a good interviewer is not whether she uses a digital recorder or a pen. The technology itself is a tool. The crucial factor is the skill in being an interviewer – preparation, knowing what questions to ask and knowing when something isn’t right.

You wouldn’t stumble into an interview with a source having done no research, no preparation. Why in the world should journalists treat data sources any differently?

The simple answer is that APIs are an extension of what reporters do every day: ask questions. The difference is that instead of forcing reporters to gather data from multiple sources, format it to fit your local database needs and then update that database when new releases are available, APIs allow reporters to query live data from all over the Web. If you have experience working with, say, Microsoft Access and setting up an ODBC connection to a remote database, APIs are kind of like that – except that you have near-instant access to more sources of data, more useful tools (like geocoders) and more timely information than ever before.

My path working with data went something like this: spreadsheets came first, which I routinely describe as the “gateway drug” of computer-assisted reporting. Some people become such Excel wizards that it almost doesn’t make sense for them to move beyond that expertise; there is so much you can do in a spreadsheet that alone it would be worth the time to learn. But there were things about spreadsheets that annoyed and frustrated me. Pivot tables were a clumsy fit for me – they got me close to what I wanted in many instances but never quite there. And so I moved onto databases.

Databases are still one of my favorite things. They are powerful, relatively flexible and range in utility from the ultra-portable SQLite to the transactional goodness that is PostgreSQL. But they take time and effort to build, maintain and – perhaps most importantly in the long run – connect to additional sources of information. APIs are not a complete solution to these problems, but they provide a very good one that data journalists should be familiar with and consider incorporating into their work.

A simple example is the reporter who wants to track the votes of his or her state’s delegation in Congress. There are several APIs for this data, including the one I work on and another by OpenCongress. The reporter could build a database of these votes by hand or write scripts to parse the House and Senate vote data and insert them into it. But why, when the data is freely available via HTTP?

It can’t be that simple, can it? Well, no. But it can be simpler. The data you get from APIs usually comes in XML or JSON. Data journalists have, for better or worse, been dealing with XML for awhile now. JSON may be less familiar, but it is quite nice to deal with and there are plenty of libraries with which to do so. But even better than that is the fact that other people have already solved that problem for you. Not long after we released the NYT Congress API I noticed a Ruby client library for it on Github. I had never met the author; he had never contacted me. Just the same, he made it easier for people using Ruby to query the API and get back data. There’s also an excellent Python library for it, written by NPR’s Chris Amico.

Thus can you, the data journalist, benefit from other people who need and use APIs. Check out GovKit, a Ruby wrapper to multiple government and political APIs, created by the folks at the Participatory Politics Foundation. Go play with it, and figure out what sorts of things you can do when the number of data sources you’re able to tap into multiplies overnight. The possibilities for journalists are only limited by the kinds of questions we can imagine and try to answer. APIs can make it easier to act on that greatest of questions: What if?

The first article, a not-unfamiliar tale of D.C. overachievement, describes how 20-somethings working for politically-minded organizations arise well before dawn in order to scan and condense that morning’s news for bigger fish at their employers.

The second article dealt with a similar topic but in a different vein – how the leap provided by technology can lead to data overload for the military, and what is being done to help the armed forces deal with it.

The goal of the people described in both articles is remarkably similiar: identify the information they absolutely need from a sea of sources. “It’s reading the 1,000 stories in the papers and Hill rags, and finding that one needle in the haystack that’s going to matter.” Or, describing the environment of a combat aviator, “the screens in jets can be so packed with data that some pilots call them ‘drool buckets’ because, they say, they can get lost staring into them.”

Both are worrying information situations, leaving aside the nature and consequences of combat. But when it comes to the ability to handle information, I worry less about the military and more about those D.C. organizations, because although the way the latter group is dealing with information overload “works”, it has some severe problems. And most newsrooms are much more like those political groups than they are the U.S. military.

The main problem with the “hire a young person willing to sacrifice their sleep patterns” approach is that it yields inconsistent results. Some people are better at spotting valuable information than others, and such skills aren’t always transferable. Thus the organization may be missing things – the Rumsfeldian “unknown unknowns”, if you will. And while it seems clear from the article that the people tasked with such duties are fairly tech-savvy, I’d be willing to bet that their efforts don’t extend much beyond keyword-based search alerts, RSS feeds and plain reading. They (and we) can do better.

Librarians know that, in general, most people are pretty terrible searchers of indexes and search engines. We rarely take advantage of advanced search techniques. We stop after the first page of results. We tie ourselves to a dominant source. Computers, by contrast, can be told not to do all of those things. But for organizations that really need to know what’s happening in their subject areas, relying mainly on other people to ferret out and publish information is a risky approach if you believe that information is power.

Let’s say that you work for a trade association and you want to know when your opponents are contributing money to federal politicians. You can, of course, pay a person to repeatedly submit searches on the Federal Election Commission’s site. Unless you’re willing to pay for round-the-clock coverage, however, you might not find out until hours or days after it actually happens. That is, if you’re not thinking about an automated way to obtain that information. The whole phrase “win the morning and win the afternoon” means that you need the overnight to figure things out – and you could still easily lose the evening.

Sure, you could hire a firm to track stuff for you. But each step you are removed from the person(s) who actually acquire that information, the less trust you have in the process and resulting information. Churning through sleep-deprived kids in an attempt to keep up with the flood of information makes as much sense as trench warfare does for today’s military.

I don’t know whether the military will figure out how to deal with information overload on the battlefield, although I hope they do. But already they’re doing much better than these D.C. organizations (and most newsrooms) because they have a better handle on the problem: the flood of information cannot be managed by having people rise earlier and earlier. It must be managed using human and technology resources, with an eye towards increasing reliance on the latter while making the former more valuable (and more sustainable). Sleep well, folks.

DOCS had kind of languished of late, although I’ve continued to add stories here and there, but I noticed that I also wanted to add other kinds of stories and in general highlight different forms of journalism. So GJ will be more subjective, sure, but hopefully more interesting and useful. I also couldn’t believe that the domain was available. Any folks who would like to become contributors to GJ, please contact me via gmail (dwillis) or on Twitter (derekwillis).

Hey there, journalism student!

A bunch of your colleagues are having a get-together in February, and you should come. Actually, you need to be there.

I’m talking about Investigative Reporters and Editors’ annual Computer-Assisted Reporting Conference, held in Raleigh, North Carolina. Several days of panels, hands-on training and talks with journalists from all kinds of news organizations. There’s even a great side event on data visualization. This conference changed my career, and I’m betting it can do the same for you.

For many of you, this may be a step into the unknown. Your universities probably don’t have much instruction in using spreadsheets, databases or mapping software for journalism. You’ve been focusing on the other fundamentals – writing, editing, maybe some multimedia skills – all of which are a proper part of your education. So what is “computer-assisted reporting” and why should you attend this conference?

I was pretty nervous to attend my first NICAR conference, as they used to be called, and when I sat around the hotel bar for the first time I’m pretty sure I made no worthwhile contributions to any discussion. But I listened to, and learned from, people who are giants in our line of work, and they were generous and awe-inspiring. And I found my people.

What do we do? We work with information in its many formats: text, maps, spreadsheets, databases. We learn to crawl, then walk and sometimes break into a light jog. Mostly, we get ideas that we can put to work. We try to find stories in places that other reporters don’t look – in documents or data, for example. And we try to use many different methods of storytelling, because not every story requires 800 words and a locator map.

Let me give you an example. In 1996 I worked for The Palm Beach Post doing some metro reporting and trying to do data work when I could, and my window to that world was the NICAR community. One of my colleagues, Andrew Metz, had heard about a local official doing land swaps with his county’s largest landowner that gave the official a larger, more valuable piece of personal property. But when Andrew went to try and nail it down, he found that this county kept its property records on 3×5 index cards. Even for a small county, it would take weeks of work to figure out.

But I remember hearing from the NICAR-L listserv about stories that analyzed property records kept at the state level, and found out that Florida indeed maintained a database of property records from all 67 counties. I promptly ordered the data from the state and was informed that two 9-track tapes were on their way to our office.

At this point you might be saying, “9-track tapes?” I sure did. But once again the NICAR community saved me. In this case, one of the legends of CAR, Elliot Jaspin, had written software to convert information stored on a 9-track tape into something that a PC could read. To quote Jaspin: “A reporter who can’t read a magnetic tape is as illiterate as the 15th Century peasant confronted by Gutenberg.” You probably never heard that in your classes, but it’s true. Andrew was able to write the story, using the data to prove what previously had only been rumor.

The point here is not that each of you must master a broad set of data-related skills and be experts at everything. The point is that by connecting with this community of journalists, you’ll find out that so much more is possible than you ever knew. Stories you may have dismissed as undoable suddenly are much closer to becoming reality, and new forms of storytelling are now within your grasp. That you literally can make yourself more productive and more valuable. The community around this conference is entirely about sharing what we know and bringing more people into the fold.

Now, we’re journalists here, so we can also be cranky, or slightly inappropriate at times. IRE members are, by and large, self-starters driven to better themselves, and they don’t have a ton of indulgence for those who aren’t willing to work. But if you’re willing to listen and try (and sometimes fail), there are plenty of rewards.

Once you’re in, you’ll become a contributor, too – on the listserv and at conferences (see some of my early presentations if you’d like a chuckle). You’ll become an evangelist for making better use of data in your own newsrooms, and you’ll stand out among your peers who cannot do what you can.

IRE is friendly to students, too. Registration for the conference is $100 for students, and that includes does not include IRE membership, but that’s just $25. If you can get to Raleigh, there’s probably someone who is willing to share a hotel room (ask Andy Boyle, who drove from Nebraska to Indianapolis in March 2009 to attend). If you want to find a community of people who are doing interesting and valuable things in journalism, who are among the leading practitioners of the craft and who are eager to share what they know, then you should be in Raleigh in February. I will be, and I’ll be happy to say hello and welcome you to the club.

First, let me say that my observations about the general history of WPNI and its relationship with the paper are colored by my own experiences, but I agree with folks like Jay Rosen who say that at one point, washingtonpost.com was clearly a national leader – not just in technical capability, but in the kind of mindset necessary for a news organization prepared to take advantage of the Internet’s possibilities. I supported the creation of WPNI as a separate operation, to allow it more creative freedom, but both the people of WPNI and their colleagues at the Post should have done more to foster a better environment for working together. It’s something that I failed at when I was there.

But back to the kind of environment that leads to the departures, voluntarily or otherwise, of so many talented and dedicated employees. I don’t know the people who currently run the Washington Post, but I do think I understand a bit about how the organization works and thinks, having spent about three years there (more than two at the paper and about nine months at WPNI). When I wrote about moving from the paper to the website back in 2007, I left out some details about how that process happened. And I think, in hindsight, that they shed some light on how the organization operates.

It’s true that there were a number of people at the Post who were supportive and encouraging of my ambitions to work at WPNI. Among them were my supervisor at the time, Lucy Shackelford, and the paper’s editor, Len Downie. But once I had seriously pursued the idea of working on the website, it took months for the move to happen, and not just for reasons of simple corporate bureaucracy. In a very real way, my transition was held up – I (jokingly at first, and then angrily) referred to it as a filibuster or a senatorial hold – by a few people at the paper. These people, most of whom no longer occupy the positions they held then, are not stupid. They are among the smartest folks I’ve ever worked with, and I have a high regard for their journalistic abilities. But the thinking that caused the editor of the paper to become involved in whether a mid-level staffer moved to the website was, in essence, this: this is a bad idea, because it will hurt the paper. My ego might like to think that this was really true, but I think the reality is that these people could not compare the value of my work for the website to the paper because they did not understand what it is I wanted to do. So they went with what they knew, and that seemed to be a net deficit for them. And thus it was that I mooted the option of simply resigning from the paper in order to join its website.

I don’t envy the people who run the Washington Post (or any news organization) today. They have a ton of thankless choices to make, and critics on every side. From a certain standpoint, I can appreciate the idea that the paper edition, which generates the overwhelming share of the revenue, should be protected and bolstered as much as possible. But I cannot agree with the idea that this means that you take employees who have proven expertise doing valuable and informative things that don’t always translate into print and cannibalize (or toss away) their talents for the sake of the paper.

My fear as a Washington Post subscriber and reader of washingtonpost.com is that, when the folks running the organization turn things around (and I believe that it is not an impossibility or even a long-shot), what emerges will be not only a news organization that is a shadow of its former self – most orgs will have to face that reality – but that it will have put so much emphasis on the paper that it cannot take advantage of the possibilities online. That the folks running things are literally rolling back the progress and smart work that has been done, and will not be able to get it back as fast as they might think. And the people who remain – those who will be charged with the task of rebuilding a news operation that embraces all of the ways that its readers and users can gain value – will have neither the support nor the depth to make it happen.

The intriguing title of a recent report from scholars at Duke is “Accountability Through Algorithm: Developing the Field of Computational Journalism“. Semi-related to CAR, Computational Journalism is defined as “the combination of algorithms, data, and knowledge from the social sciences to supplement the accountability function of journalism.” I take each of those – algorithms, data and knowledge from the social sciences – as separate elements, because while journalists do have plenty to learn from the social sciences, we also operate in an environment that is not quite academic (and sometimes not at all).

The report identifies four areas of potential exploration: techniques for data transformation and pattern discovery in investigative reporting; a digital “dashboard” for journalists; new social and technical structures for interactions among readers and reporters; and sense-making advances from other disciplines. All are interesting and worthy, but to me the first two are particularly so.

On the first, the best investigative journalists have been developing tools for extracting meaning from reams of information for years. The change now is that we have a greater platform for these tools in the Internet, and an effort like DocumentCloud is a clear example of that change. The challenge we face is that patterns are interesting to different people for different reasons; what an accountant finds interesting may not always be of interest to a journalist, and vice versa. The current deficit is not in the area of tools; it is the occasionally trickier area of adapting those for the task of journalism. That requires the guiding influence of people like Sarah Cohen, a newly minted Knight Chair at Duke, who is studying these issues right now. But it also requires the active participation of a wide range of news organizations and journalists. In the Internet, we have a leveling platform, but only if more journalists participate. That may be a greater challenge than the technical one.

One way to get there is the second idea – a journalist’s dashboard. This would provide reporters with a way to keep track of the deluge of information coming into newsrooms. But again, the technological side of that equation, as difficult as it is, is less of a concern to me than the implementation and adoption of the results. We know how to gather various bits of information in one place. We’re not that good at distilling the best of them, or even knowing where to start. The good news is that we have blueprints for this kind of thing: the people and companies who make great Web apps that distill masses of data into understandable results. The bad news is that we, as a business, work very differently. We don’t really share much, outside of experiences at conferences or over drinks, and particularly not at the institutional level. And we’re downright awful, in general, at adapting good ideas for our own uses.

For the idea of Computational Journalism to work, a lot is riding on a movement that is slowly growing but urgently necessary for the news industry: the increasing adoption, use and proliferation of open-source tools. The CAR community has seen an influx of use of various types of open-source software, from databases to GIS systems to web frameworks. More and more reporters and editors are embracing different styles of journalism. But the broader concept of opening up our newsrooms, both philosophically and in terms of our content and efforts, has been slow in coming. It requires not just the creation of tools, but also the development of journalists and readers who will use those tools most effectively. And that’s more than an algorithm – to say nothing of Twitter – can solve alone.

Oh, and Duke folks? Can we get a version of that report that embraces the Web as much as the concept? HTML will do fine.

So here’s an idea: reward innovation with concrete responses. Yesterday, J-Lab at American University announced the winners of its annual Knight-Batten Award for innovations in journalism, and my employer won the grand prize for a body of work that included Represent, an app that my colleague Andrei Scheinkman and I built along with Stephan Weitburg. The honor and attention from that award is really great, and a cash reward doesn’t hurt, either. But we didn’t build Represent with Knight-Batten in mind.

We built Represent because The Times gave us the incentive and motivation, via a company-wide technology challenge designed to solicit working prototypes or applications for nytimes.com. Winners get a cash bonus (always a good incentive to enter) but also the resources to see their ideas come to life on the site (or internally, since internal apps also qualify). The former is a very nice thing indeed, but the latter is more important in the long-term, since people like to see their work showcased. Our contest is open to all employees of the company, and can be built in pretty much anything, which means that technology itself is an enabler of progress, not a barrier.

The key here is that for a small investment, the Times got some of its employees to work on projects that they were personally interested in, on their own time. The winners and the company benefit from new ideas, and the prospect of winning helps bring more people into the process. Does your news organization do this? Why not?