First, I stand by what I said about how little we understand about the way that elections are won or lost these days. It’s not that political journalism has strayed from its roots, or stopped covering important elements of a modern campaign. It’s that the elements of a modern campaign have changed, and as journalists, we have not kept pace.

You might respond that campaigns still involve quite a lot of stuff that we do understand, such as debates and visits to state fairs and town hall meetings. True. But the nature of media and technology has brought extensive changes to the electoral system, and I don’t believe that we as journalists devote enough attention to understanding those changes. Remember the Dean campaign in 2003? Most of the coverage was on the, for then, staggering online fundraising managed by some doctor from Vermont. But that Wired piece I referenced had it right; Dean’s accomplishment was less a mastery of the Internet but a willingness to embrace its fundamental aspect: you give up some control by bringing other people in, and you gain a host of possibilities. You may, of course, choose badly or falter in some other way, but the lessons and possibilities are becoming clear. At the time, as a Web geek who loved politics, I felt that journalists couldn’t really explain the Dean campaign, because it was so alien to us. Today’s campaigns make me long for the simplicity of 2003.

But let’s stick with fundraising for a bit. Political fundraising can be hugely expensive, because campaigns need to amass large number of donors. Unless you’re the President, it’s hard to repeatedly gather the wealthiest Americans and have them fork over $2,500 or more for the pleasure of your company. So a smart campaign sticks with what works: direct mail is costly, for example, but it’s also effective. Telemarketing takes time and money, but it also works pretty well. Let’s not mess with the script too much. But what if you can mess with the script? Now it’s possible, even trivial, to experiment with Web site design or even advertisements in order to gauge their effectiveness and improve upon them. President Obama had a Director of Analytics for his 2008 campaign, and has been hiring data scientists experienced in predictive modeling.

White men smoking cigars in cramped rooms making gut calls is how we’ve usually understood campaign decision-making. This? Whole new ballgame. Yes, there is still a mass audience that is shaped by the media and big events. But there are now thousands and thousands of “small” audiences – or rather, they always were there. Now campaigns can identify them and deliver precision messages to them. And they can find them online in different ways; an hour after posting on Twitter about the Obama’s campaign use of Github, the campaign’s Digital Director was following me. And that’s the easy part.

While campaigns have a public presence that is mostly recorded and observed, the stuff that goes on behind the scenes is so much more sophisticated than it has been. In 2008 we were fascinated by the Obama campaign’s use of iPhones for data collection; now we’re entering an age where campaigns don’t just collect information by hand, but harvest it and learn from it. An “information arms race,” as GOP consultant Alex Gage puts it.

For most news organizations, the standard approach to campaign coverage is tantamount to bringing a knife to a gun fight. How many data scientists work for news organizations? We are falling behind, and we risk not being able to explain to our readers and users how their representatives get elected or defeated.

None of this is to say that we need to completely abandon our ways of covering elections. Horse-race coverage is and should be a part of campaign coverage, because in many respects elections are like horse races. Things can change rapidly, and small things can have big impact. We still should be on the ground, talking to voters, showing up at town halls and covering debates. We still need to show up and do the legwork.

But if we can’t appreciate, much less understand, what modern campaigns are doing to win elections, how can we hope to explain elections? If we don’t collect at least some of the information available to us – realizing that we can’t get our hands on everything that the campaigns do – we’ll miss the story. Elections will become even bigger surprises to us, and then how long will it be before readers start to ask whether we actually know the people and places we cover?

Surprises make the news. Some of my favorite stories from the 2004 presidential election are in a book by my friends Peter Wallsten and Tom Hamburger, then of the Los Angeles Times. Here’s one anecdote from the key state of Ohio:

One suburban African American woman in Ohio, for example, told us that though she tends to vote Democratic, she was deluged in 2004 with calls, e-mail messages and other forms of communication by Republicans who somehow knew that she was a mother with children in private schools, an active church attendee, an abortion opponent and a golfer.

Think about what this kind of thing means. It means that we cannot assume that the campaign visible to the mass audience is the same campaign that’s being pitched to individuals and groups around the nation, and that winning coalitions can be built not just by harnessing large groups (unions, religious voters, etc.) but also by piecing them together in small units. President Bush’s margin in Ohio in 2004? About 2.5 percent. The only thing that I don’t like about this anecdote is that Wallsten and Hamburger’s book appeared nearly two years later. Is there any evidence that we as journalists have closed the gap since then?

To understand how elections are now being waged, we need to have as many of the tools as do the campaigns. We need to build our own storehouses of data – voter registration, voter history, Census, campaign finance, advertisements and more. We need to be able to tap into the rich stream of material that’s being created and disseminated every day. We need to be able to see the value in small data points that can lead to bigger things.

Elections are great stories. They deserve to be told from a position of confidence and knowledge. We have work to do.

As part of our participation, panelists were asked to write a 1,000-word comment on the topic of their panel. Mine is “What are the common technical challenges journalists face in making sense of government documents and analyzing government actions, and how could those be overcome?” Faithful readers will know that I could probably go on for hours on this, but here is what I sent to the conference organizers (sans links):

I very much doubt that the conference organizers intended this, but the fact that our responses to these questions were requested in either PDF or MS Word formats is an excellent example of one of the technical challenges for journalists when dealing with government documents. So in the spirit of openness, I wrote this in Google Docs.

Both journalists and government employees who create and manage information need to know about more than the usual options for the collection and dissemination of information. Part of the technical failure rightly belongs to journalists — too often, we don’t ask or don’t know how to ask for information in a way that makes it easy to use. But far too often, government officials are either unaware of their format options or, more perniciously, all too aware and resort to distributing documents in, for example, a locked PDF.

I have been told many times that to release information in a format that would allow it to be copied is not the policy of a government agency. Those government agencies fail to understand what public information is. I also have been the recipient of records that clearly were stored in spreadsheet software but, for purposes of public release, have been printed out, scanned into images and stored as a PDF. Obfuscation and paranoia are not technical challenges, but they contribute to them, forcing journalists to acquire costly software or spend additional time overcoming an artificial roadblock.

This challenge is not due to deficiencies in software produced by any particular company but rather in the understanding of how information can and should be made available to citizens. To the greatest extent possible, government information should be made available in formats that allow its users to copy, sift and reorganize it as they please. In practice, this means favoring text-based and open formats over images, PDFs and closed formats. I am less concerned about how agencies store their data, as long as they are able to export it in common formats or reliable workarounds exist, but there are exceptions to this.

Map data, for example, is commonly stored at the government level in ESRI’s shapefile format, which, from the point of view of a journalist, has advantages and disadvantages. ESRI is a large, well-known company with products in use at most government agencies that have geographic data, so it makes sense that GIS data would be provided in that format. But not all newsrooms, and certainly not all journalists, are able to obtain ESRI’s software or have access to someone who can easily convert from shapefiles to other formats, such as the KML standard now owned by Google. Some government agencies already produce useful geographic data in KML format, but many others could join that list.

Fixing this situation will require education of journalists and government employees of the benefits and ease of working with open formats. The benefits for journalists are apparent: faster access to information that they can immediately put to use. Training journalists is a time-consuming and inefficient process, but journalists must break out of the mindset that government information only comes in documents.

Doing so means that journalists need to become as comfortable interviewing data as they are interviewing people. The benefits for government may need some more time to explain, but they exist. The Federal Election Commission is a case in point.

Thanks to a commitment to maintaining stable, available and well-documented data, the FEC makes it possible for its users to obtain and analyze information when they want to, even on a late-night deadline. This isn’t new-fangled technology; the FEC’s FTP site has been operating for years. But the agency operates as if it trusts its data users, not from a defensive standpoint. As a result, the FEC is rightfully seen as an agency that makes it possible for reporters to do their jobs, not as an impediment to that goal.

A more recent, but increasingly significant, technical challenge is that too many government agencies fail to make better use of the best information distribution platform they have: the Internet. In a digital age, some agencies continue to treat all records either as documents, or when they do make data available, it is done as a single dump. In many cases, journalists do not need an entire dataset; they are more likely to want to answer a single question or small set of questions. In those cases where government agencies make this possible, it is usually through a Web form of their own design – one which often is tailored to heavy users such as the regulated community.

Providing Application Programming Interfaces (APIs) to government data via XML or JSON feeds would make it possible for journalists and Web developers to take advantage of government data without having to download and process enormous files. And while adding an API will incur an up-front cost, it will also save agencies employee time handling requests that could be done computer-to-computer.

Yet such APIs are very rare in government, even though they would make it easier for users and journalists to combine disparate data, and would make it possible to build more useful applications from government data. We know this to be true, because in the absence of any meaningful government approach to disseminating legislative data, several outside organizations, including my own, have developed APIs to help spur the use and spread of congressional data.

But in order to do this, we have had to essentially reverse-engineer the Thomas site operated by the Library of Congress, writing fragile HTML parsers that can break should the LoC change the structure of individual pages. So, in order to answer anything beyond the most basic question on legislative matters, a journalist must either spend hours looking up information one page at a time or be able to write a computer program to parse those pages. There has to be a better way.

I’ve written before about the crucial role that news librarians can play in the life of the newsroom, but it takes people willing to embrace the shift in attitude and habits that these new roles demand. We’ve got a core group of folks who want to learn – the folks who paid $300 for the session, some of it out of their own pockets, were eager to dive in. The next day we heard from news librarians who have completely turned their jobs upside down (not always voluntarily), but I’m not sure that it didn’t just frighten some of the attendees.

At the News Division’s business meeting it was great to hear Jim Hunter, the outgoing division chair, say that the division’s website was a key element in serving our members and attracting more. Hopefully there will be more news on that front soon.