First, let me say that my observations about the general history of WPNI and its relationship with the paper are colored by my own experiences, but I agree with folks like Jay Rosen who say that at one point, washingtonpost.com was clearly a national leader – not just in technical capability, but in the kind of mindset necessary for a news organization prepared to take advantage of the Internet’s possibilities. I supported the creation of WPNI as a separate operation, to allow it more creative freedom, but both the people of WPNI and their colleagues at the Post should have done more to foster a better environment for working together. It’s something that I failed at when I was there.

But back to the kind of environment that leads to the departures, voluntarily or otherwise, of so many talented and dedicated employees. I don’t know the people who currently run the Washington Post, but I do think I understand a bit about how the organization works and thinks, having spent about three years there (more than two at the paper and about nine months at WPNI). When I wrote about moving from the paper to the website back in 2007, I left out some details about how that process happened. And I think, in hindsight, that they shed some light on how the organization operates.

It’s true that there were a number of people at the Post who were supportive and encouraging of my ambitions to work at WPNI. Among them were my supervisor at the time, Lucy Shackelford, and the paper’s editor, Len Downie. But once I had seriously pursued the idea of working on the website, it took months for the move to happen, and not just for reasons of simple corporate bureaucracy. In a very real way, my transition was held up – I (jokingly at first, and then angrily) referred to it as a filibuster or a senatorial hold – by a few people at the paper. These people, most of whom no longer occupy the positions they held then, are not stupid. They are among the smartest folks I’ve ever worked with, and I have a high regard for their journalistic abilities. But the thinking that caused the editor of the paper to become involved in whether a mid-level staffer moved to the website was, in essence, this: this is a bad idea, because it will hurt the paper. My ego might like to think that this was really true, but I think the reality is that these people could not compare the value of my work for the website to the paper because they did not understand what it is I wanted to do. So they went with what they knew, and that seemed to be a net deficit for them. And thus it was that I mooted the option of simply resigning from the paper in order to join its website.

I don’t envy the people who run the Washington Post (or any news organization) today. They have a ton of thankless choices to make, and critics on every side. From a certain standpoint, I can appreciate the idea that the paper edition, which generates the overwhelming share of the revenue, should be protected and bolstered as much as possible. But I cannot agree with the idea that this means that you take employees who have proven expertise doing valuable and informative things that don’t always translate into print and cannibalize (or toss away) their talents for the sake of the paper.

My fear as a Washington Post subscriber and reader of washingtonpost.com is that, when the folks running the organization turn things around (and I believe that it is not an impossibility or even a long-shot), what emerges will be not only a news organization that is a shadow of its former self – most orgs will have to face that reality – but that it will have put so much emphasis on the paper that it cannot take advantage of the possibilities online. That the folks running things are literally rolling back the progress and smart work that has been done, and will not be able to get it back as fast as they might think. And the people who remain – those who will be charged with the task of rebuilding a news operation that embraces all of the ways that its readers and users can gain value – will have neither the support nor the depth to make it happen.

The intriguing title of a recent report from scholars at Duke is “Accountability Through Algorithm: Developing the Field of Computational Journalism“. Semi-related to CAR, Computational Journalism is defined as “the combination of algorithms, data, and knowledge from the social sciences to supplement the accountability function of journalism.” I take each of those – algorithms, data and knowledge from the social sciences – as separate elements, because while journalists do have plenty to learn from the social sciences, we also operate in an environment that is not quite academic (and sometimes not at all).

The report identifies four areas of potential exploration: techniques for data transformation and pattern discovery in investigative reporting; a digital “dashboard” for journalists; new social and technical structures for interactions among readers and reporters; and sense-making advances from other disciplines. All are interesting and worthy, but to me the first two are particularly so.

On the first, the best investigative journalists have been developing tools for extracting meaning from reams of information for years. The change now is that we have a greater platform for these tools in the Internet, and an effort like DocumentCloud is a clear example of that change. The challenge we face is that patterns are interesting to different people for different reasons; what an accountant finds interesting may not always be of interest to a journalist, and vice versa. The current deficit is not in the area of tools; it is the occasionally trickier area of adapting those for the task of journalism. That requires the guiding influence of people like Sarah Cohen, a newly minted Knight Chair at Duke, who is studying these issues right now. But it also requires the active participation of a wide range of news organizations and journalists. In the Internet, we have a leveling platform, but only if more journalists participate. That may be a greater challenge than the technical one.

One way to get there is the second idea – a journalist’s dashboard. This would provide reporters with a way to keep track of the deluge of information coming into newsrooms. But again, the technological side of that equation, as difficult as it is, is less of a concern to me than the implementation and adoption of the results. We know how to gather various bits of information in one place. We’re not that good at distilling the best of them, or even knowing where to start. The good news is that we have blueprints for this kind of thing: the people and companies who make great Web apps that distill masses of data into understandable results. The bad news is that we, as a business, work very differently. We don’t really share much, outside of experiences at conferences or over drinks, and particularly not at the institutional level. And we’re downright awful, in general, at adapting good ideas for our own uses.

For the idea of Computational Journalism to work, a lot is riding on a movement that is slowly growing but urgently necessary for the news industry: the increasing adoption, use and proliferation of open-source tools. The CAR community has seen an influx of use of various types of open-source software, from databases to GIS systems to web frameworks. More and more reporters and editors are embracing different styles of journalism. But the broader concept of opening up our newsrooms, both philosophically and in terms of our content and efforts, has been slow in coming. It requires not just the creation of tools, but also the development of journalists and readers who will use those tools most effectively. And that’s more than an algorithm – to say nothing of Twitter – can solve alone.

Oh, and Duke folks? Can we get a version of that report that embraces the Web as much as the concept? HTML will do fine.

So here’s an idea: reward innovation with concrete responses. Yesterday, J-Lab at American University announced the winners of its annual Knight-Batten Award for innovations in journalism, and my employer won the grand prize for a body of work that included Represent, an app that my colleague Andrei Scheinkman and I built along with Stephan Weitburg. The honor and attention from that award is really great, and a cash reward doesn’t hurt, either. But we didn’t build Represent with Knight-Batten in mind.

We built Represent because The Times gave us the incentive and motivation, via a company-wide technology challenge designed to solicit working prototypes or applications for nytimes.com. Winners get a cash bonus (always a good incentive to enter) but also the resources to see their ideas come to life on the site (or internally, since internal apps also qualify). The former is a very nice thing indeed, but the latter is more important in the long-term, since people like to see their work showcased. Our contest is open to all employees of the company, and can be built in pretty much anything, which means that technology itself is an enabler of progress, not a barrier.

The key here is that for a small investment, the Times got some of its employees to work on projects that they were personally interested in, on their own time. The winners and the company benefit from new ideas, and the prospect of winning helps bring more people into the process. Does your news organization do this? Why not?

One of the common responses to such entreaties is exemplified in this comment, which includes this plea: “I understand the need to bolster one’s skill set. But what happened to the days when we actually, you know, worried about reporting rather than slavishly trying to master every piece of technology?”

If only that was the real problem.

The real problem is the way that we as journalists manage information, because that determines so much else: the kinds of stories we’re able to envision and construct, the amount of context we’re able to bring to bear in a short amount of time and our ability to connect the dots. In general, and this is my scientific conclusion, we suck at managing information.

That’s nothing new, you might say, and you’d be right. But what has changed is that a lot of the people and institutions we cover are now getting smarter about this stuff, and are using better tools to help them manage information. From tracking crime to measuring customer loyalty, the sophisticated use of information is a crucial factor in many modern activities. Us? We’re still knocking rocks around hoping to generate a spark.

I’m not knocking learning skills like how to maintain a blog. I’m just saying that if all we do is teach new tools and skills, we’re making the underlying problem harder to solve, not easier, because we’re just encouraging the production of even more separate and disconnected piles of information. More photos over here, more spreadsheets over there. We’re still drowning in information and we can’t figure out how to use it to our best advantage, like finding undiscovered patterns and coming up with definitive explanations instead of the ol’ three-person anecdote story.

So yeah, teach those CAR and multimedia skills. Have everybody Twitter. But please, let’s find a way to address the fact that for many journalists, Microsoft Word is the primary tool for organizing any and all kinds of information. Let’s make sure that our silos of content (text archives, photo archives, databases, etc.) can at least be made to talk to each other, if not naturally, then through APIs or metadata or something. And let’s start talking about how a news organization’s information belongs to the organization, not just to individual reporters and editors, and how our products could be so much better if we adhered to that principle before a story/photo/slideshow is published, not just after.

The Case for Access

Before I offer the case against using it, let’s look at why we would use Access to teach database concepts. First, it’s widely available as part of Microsoft’s popular Office suite. A lot of news organizations have it already installed, or can do so without much trouble. Second, it has a familiar look and feel for people who have used Excel – sorting and filtering work nearly the same way, for example – and imports and exports Excel files with ease. Third, the query grid that Access has a default makes it easy to get started on actually getting answers from your data.

That’s about it, as far as I’m concerned. I guess you could throw in the ability to generate reports and construct forms, but these are less of an advantage as web-based apps have become more popular and added features. In fact, the last reason I cited, the query grid, isn’t really an advantage at all, as I’ll explain below.

The Case Against Access

Access costs money. In SQLite, MySQL and PostgreSQL, there are superior database programs that are free and open-source. If you’re asking your students, many of whom may be buying Mac laptops, to get Access, you’re putting an additional burden on them. And if that’s all they know once they graduate and manage to land a job, if that place doesn’t have Access, they may need to get it (or have nothing at all).

The Access query grid hides the fact that underneath, Access runs SQL queries. So a user is able to construct and execute a SQL query without writing any SQL whatsoever. This is, imho, a bad thing, as it makes it possible to get results without actually knowing what you are doing. When we teach the query grid, we’re teaching behavior over understanding, or at the very least we’re allowing behavior to compete with understanding. And that doesn’t even begin to address the issue that the query grid doesn’t do everything that SQL can. In terms of teaching, this is critical; we’re not properly equipping students for the opportunities and challenges they could face.

Another issue is data portability: Access databases don’t support dumping to a .sql file, which is a great way to transfer SQL data without losing data types. Access does export to many formats, including Excel, CSV and XML, but the lack of SQL dump ability is a pain for transferring data. If you want to send somebody an Access database, you can either send them the entire file (providing they have Access installed), or you can export each of the tables and have them re-import them. And if you do email that .mdb (or now, .accdb) file, be warned that they do get quite big. To demonstrate this, I loaded the same three tables into Access 2007 and SQLite and the Access file was nearly 3 times the size of the SQLite database.

Finally, there’s the Web. Know many popular Web sites that run off an Access database? Me neither. If all you know is Access and not the underlying SQL concepts, your transition to a popular server software like MySQL is going to be more difficult. Sure, you say, but it’s better than nothing. But as far as the Web goes, Access is almost nothing itself. So why would you teach a program that has very little future on the Web – the platform of today and tomorrow?

The Case for SQLite

SQLite is my choice for the candidate to replace Access in journalism education. In addition to the advantages listed above, it’s also easy to “install.” If you can download files, unzip them and move them to a location on your hard drive, you can “install” SQLite. If you can install a Firefox add-on, you can manage it in the browser. And you can take your database files home with you or email them around. The add-on supports importing CSV files, SQL dumps and XML (although all databases can have issues with importing XML). It looks and works the same on a PC or a Mac. Most importantly, it demands an understanding of SQL that you can avoid when learning Access.

When I first learned SQL at an IRE bootcamp, we were using FoxPro and we learned how to type the SQL commands. That knowledge only becomes more valuable as you learn the limits and possibilities of SQL. Journalism educators and trainers should commit to teaching SQL on the broadest platform possible and with an emphasis on the syntax and meaning of the language itself, not on which buttons to click. Otherwise we risk sending students out into this new journalism world even less-prepared to handle data intelligently, and I don’t think we can afford that.

Liz Donovan, one of the best news researchers ever, died of lung cancer in Gainesville, Ga. Truly a pioneer among news librarians, she was instrumental in everything from research for Woodward and Bernstein during Watergate to embracing (and leading other researchers to adopt) the use of blogs, wikis and other new technologies in newsrooms. She worked at the Washington Post and Miami Herald before retiring to the mountains of North Carolina where she continued to learn and spread her knowledge (and some beautiful photographs of the area).

Liz and Margot Williams were the first to welcome me to the News Division and to encourage me to participate in conferences, the mailing list and other activities. Liz encouraged me to write about the technological challenges and opportunities faced by newsrooms, and was generous to a fault with her colleagues seeking help. She was a constant innovator, a skilled researcher and a wonderful person. Among the people in journalism, it is rare to find someone about whom no one has a bad thing to say. Liz was one of those people. I’ll miss her friendship, advice and leadership.

There’s little doubt that a good reporter who looked hard enough at the LIRR would start to learn that disability payments for retirees were more frequent than one might expect. Tales about people retiring and getting huge disability payments, as the story details. You’d be able to get the pictures of folks receiving large disability payments playing golf, as the reporters on this story did. But nailing the story – removing any doubt that this isn’t a handful of isolated cases – takes the kind of analysis that Andy Lehren did. The kind that leads to a reporter being able to write this:

“Virtually every career employee – as many as 97 percent in one recent year – applies for and gets disability payments soon after retirement.”

Case closed. Airtight. Dare I say George Tenet might trot out “slam dunk” again for this one. My only quibble is that Andy didn’t get a byline. Congrats to Andy, Walt and the rest of my colleagues who worked on this. And for showing the difference that good data analysis can make to journalism.

Mindy asks for some bullet points on why news organizations would do better to not use Caspio for their Web database needs. Feel free to add on:

1. SEO. If you like building databases that are not indexed by Google and other search engines, then Caspio’s right for you. Go ahead, Google “Powered by Caspio“.
2. Owning vs. Renting. You will never stop paying for Caspio unless you quit it entirely. And then you’ll still need to rewrite your apps. All you’ve gained is more work.
3. You will need programming. Caspio says “no more programming,” but to do anything beyond basic search and display, you will need some. Oh, but you can’t get access to that functionality during a free trial.
4. Like using Flash? Caspio doesn’t.
5. Nickel and Dime. Zip code searching costs $150 to setup and $50 a month.
6. As my boss and friend Aron says, “We can’t outsource our future.” By choosing Caspio, you’re dependent upon them to add features, and while they do, they add them for all users, too. So much for differentiation.

And here’s a bonus quote from Jacob Kaplan-Moss, one of Django‘s lead developers, who admittedly has a bias in this area. But still, it’s a very telling quote: “I’ve actually stopped being all that concerned about Caspio: each new Caspio customer is one more competitor my paper doesn’t have to worry about.”

Now, I’m sure there are six reasons to use Caspio, but I don’t think they stack up in the long term. I think they leave you with more work, not less, and with apps that you have to spend valuable time making look different from everybody else who uses Caspio.

There’s a tendency among some folks in the industry to see CAR and other technological tools as just that – blunt instruments. Helpful, sure, but not ultimately necessary to the task of creating journalism. And for a segment of what journalism does, that’s probably ok. When we report on people and institutions that aren’t using technology to guide their decisions or actions, then an understanding of how data is used or certain technologies isn’t a necessity.

I suppose a music critic needn’t understand much about databases, for example, but reporters covering government, business, college or professional sports, to name a few, should be able to assess their subjects the way that people inside those sectors do. And increasingly, that means understanding the use of data. Many local governments base their police staffing – who covers where – on a non-stop flow of crime data. Sports teams pour over tape, logging their opponents’ tendencies in preparation for upcoming games. Businesses are all about the numbers, too.

And then there’s politics. Winning elections these days is very often about putting together enough voters to crack 50%. There’s microtargeting based on consumer data and door-to-door canvassing so that volunteers can input demographic data into centralized servers. They’re not doing that just for fun – it’s valuable information. But if journalists can’t really grasp how organizations are using data, we’re liable to miss the effects, and thus miss some fuller explanations of events. Yes, we can rely on people to tell us what’s happening – and we should – but if data plays a big part in the life of an organization, the reporter covering it should have some basis to evaluate that role.

So how does that relate to Dave’s story about the quads? Well, after reading it, I noticed that there were some subtle bits of detail that I never would have thought to include or been able to describe as well – about how the NICU operates, the details of the births. That’s because Dave has been there with his twin boys. A parent of a child born without complications or a single person would have been hard-pressed to write as good a story. I sure wouldn’t have been able to do so.

It’s the same idea when it comes to understanding the basis for decisions that come from, at least in part, the collection and consumption of data. It’s can mean the difference between telling a story and telling a better story. I’m sure plenty of organizations that we cover would be happy to have reporters who are in the dark about these things. But that doesn’t help our readers any.

So, technology and data as a tool? Yes. But when the tools become a crucial part of the world we cover, understanding how they work and being able to use them makes us better journalists.