First, I stand by what I said about how little we understand about the way that elections are won or lost these days. It’s not that political journalism has strayed from its roots, or stopped covering important elements of a modern campaign. It’s that the elements of a modern campaign have changed, and as journalists, we have not kept pace.

You might respond that campaigns still involve quite a lot of stuff that we do understand, such as debates and visits to state fairs and town hall meetings. True. But the nature of media and technology has brought extensive changes to the electoral system, and I don’t believe that we as journalists devote enough attention to understanding those changes. Remember the Dean campaign in 2003? Most of the coverage was on the, for then, staggering online fundraising managed by some doctor from Vermont. But that Wired piece I referenced had it right; Dean’s accomplishment was less a mastery of the Internet but a willingness to embrace its fundamental aspect: you give up some control by bringing other people in, and you gain a host of possibilities. You may, of course, choose badly or falter in some other way, but the lessons and possibilities are becoming clear. At the time, as a Web geek who loved politics, I felt that journalists couldn’t really explain the Dean campaign, because it was so alien to us. Today’s campaigns make me long for the simplicity of 2003.

But let’s stick with fundraising for a bit. Political fundraising can be hugely expensive, because campaigns need to amass large number of donors. Unless you’re the President, it’s hard to repeatedly gather the wealthiest Americans and have them fork over $2,500 or more for the pleasure of your company. So a smart campaign sticks with what works: direct mail is costly, for example, but it’s also effective. Telemarketing takes time and money, but it also works pretty well. Let’s not mess with the script too much. But what if you can mess with the script? Now it’s possible, even trivial, to experiment with Web site design or even advertisements in order to gauge their effectiveness and improve upon them. President Obama had a Director of Analytics for his 2008 campaign, and has been hiring data scientists experienced in predictive modeling.

White men smoking cigars in cramped rooms making gut calls is how we’ve usually understood campaign decision-making. This? Whole new ballgame. Yes, there is still a mass audience that is shaped by the media and big events. But there are now thousands and thousands of “small” audiences – or rather, they always were there. Now campaigns can identify them and deliver precision messages to them. And they can find them online in different ways; an hour after posting on Twitter about the Obama’s campaign use of Github, the campaign’s Digital Director was following me. And that’s the easy part.

While campaigns have a public presence that is mostly recorded and observed, the stuff that goes on behind the scenes is so much more sophisticated than it has been. In 2008 we were fascinated by the Obama campaign’s use of iPhones for data collection; now we’re entering an age where campaigns don’t just collect information by hand, but harvest it and learn from it. An “information arms race,” as GOP consultant Alex Gage puts it.

For most news organizations, the standard approach to campaign coverage is tantamount to bringing a knife to a gun fight. How many data scientists work for news organizations? We are falling behind, and we risk not being able to explain to our readers and users how their representatives get elected or defeated.

None of this is to say that we need to completely abandon our ways of covering elections. Horse-race coverage is and should be a part of campaign coverage, because in many respects elections are like horse races. Things can change rapidly, and small things can have big impact. We still should be on the ground, talking to voters, showing up at town halls and covering debates. We still need to show up and do the legwork.

But if we can’t appreciate, much less understand, what modern campaigns are doing to win elections, how can we hope to explain elections? If we don’t collect at least some of the information available to us – realizing that we can’t get our hands on everything that the campaigns do – we’ll miss the story. Elections will become even bigger surprises to us, and then how long will it be before readers start to ask whether we actually know the people and places we cover?

Surprises make the news. Some of my favorite stories from the 2004 presidential election are in a book by my friends Peter Wallsten and Tom Hamburger, then of the Los Angeles Times. Here’s one anecdote from the key state of Ohio:

One suburban African American woman in Ohio, for example, told us that though she tends to vote Democratic, she was deluged in 2004 with calls, e-mail messages and other forms of communication by Republicans who somehow knew that she was a mother with children in private schools, an active church attendee, an abortion opponent and a golfer.

Think about what this kind of thing means. It means that we cannot assume that the campaign visible to the mass audience is the same campaign that’s being pitched to individuals and groups around the nation, and that winning coalitions can be built not just by harnessing large groups (unions, religious voters, etc.) but also by piecing them together in small units. President Bush’s margin in Ohio in 2004? About 2.5 percent. The only thing that I don’t like about this anecdote is that Wallsten and Hamburger’s book appeared nearly two years later. Is there any evidence that we as journalists have closed the gap since then?

To understand how elections are now being waged, we need to have as many of the tools as do the campaigns. We need to build our own storehouses of data – voter registration, voter history, Census, campaign finance, advertisements and more. We need to be able to tap into the rich stream of material that’s being created and disseminated every day. We need to be able to see the value in small data points that can lead to bigger things.

Elections are great stories. They deserve to be told from a position of confidence and knowledge. We have work to do.

If you’re a fan of the useful CSVKit, a command-line tool written in Python, but you’re a Rubyist, then please do yourself a favor and take a look at RemoteTable, a gem by Seamus Abshere. The two libraries aren’t entirely identical, and CSVKit has been rightly praised by others, so let me dive a little deeper into why RemoteTable is a data parser’s friend.

RemoteTable is essentially a set of useful wrappers around the common process I outlined above: grab a structured file and use it contents. Except that when I say “structured file”, I don’t just mean a CSV. I mean CSVs inside of zip files. Excel spreadsheets of the .xls and .xlsx varieties. Google Spreadsheets and Open Office spreadsheets. Web pages with HTML tables in them. XML. And, what the hell, fixed-width files, too. You’ll want to see the examples.

To those options it adds some useful utilities, much as CSVKit does. You can cut certain columns using the Unix cut utility, skip or crop rows using tail, make the entire file UTF-8 or remove “useless” characters.

It’s simple to try out. If you have Ruby installed and are not on Windows (sorry!), then gem install remote_table and wait a minute or two, as it installs quite a few dependencies for managing all this stuff. Then, in a console session, try grabbing a CSV file:

Each row is converted into an Ordered Hash so that you can refer to columns by name instead of position.

Now, what about a fixed-width file? Maybe one in which I’m only interested in some of the columns? Easy, thanks to RemoteTable’s nice DSL:

Notice that :cut option I passed? That’s where you include arguments to cut. In this case, I removed the financial columns from this file. You can also pass a :select option to it to do on-the-fly filtering of certain columns based on a matcher such as a regular expression or plain text.

RemoteTable can tell you when you’re not being very smart about its use, too, such as causing it to re-fetch the source file multiple times (it warns you when it does this). It also handles local files, so if you want to avoid the downloading bit (or your files aren’t on the Web), that’s fine, too. Honestly, I have no idea why this library isn’t more popular, but you can help make it so on Github.

Swizec Teller’s entry, “Measuring vocabulary richness with Python“, described an algorithm by George Udny Yule in a 1944 paper entitled “The statistical study of literary vocabulary.” Yule created a way to quantify the diversity of vocabulary in a given text, and Teller translated that formula into straightforward Python code.

So I made a simple Django app that accepts text via a form and uses Teller’s code to calculate Yule’s I score of vocabulary richness. It uses the really useful Natural Language Toolkit; the only oddity is that when developing locally on a Mac, the standard installation of NLTK via pip is borked, so you need to specify a file to download in your requirements.txt. You can find the demo app here.

I’m not offering a judgment on using Heroku or other instant deployment-type services; most of them seem pretty easy to use but out of my price range for anything significant. But it’s nice to know that services like Heroku, ep.io and others offer enough flexibility to do stuff like natural language parsing.

Done snickering? Let’s move on.

The Tools Team built mostly internal applications and services that helped the Web site run better. I mainly got to work on front-facing projects like the Congress Votes Database, the 2008 presidential campaign and an innovative series on lobbyist Gerald Cassidy. But I did work on a few internal tools, and since I joined The Times in late 2007 I’ve built a few more. I’ve found that such tools are not so different from what we now consider to be journalism by Web development. Chosen wisely and done well, they can have impacts that go far beyond a single story or series. We should not dismiss them as “not journalism.”

If you’re at the geekier end of the journalism spectrum, then chances are your colleagues know about the stuff you can do. They may not understand it or be able to explain it; a former managing editor of mine, when told about the various technical steps to accomplish something useful, would invariably respond with a touch of wonder: “Fuckin’ Internet!” You can explain your work to a decent percentage of your colleagues by invoking Harry Potter or the Lord of the Rings and leave it at that.

But that doesn’t mean that building tools that can be used by broad segments of the newsroom is a one-way street or has to lead to a divide between you and the other journalists. There will be people in every newsroom who mainly take and rarely give, and in those situations being a technologist is no different from being a clerk. Good tools, like good apps, are a product of collaboration and improve the ability of the newsroom in general. They also make for more and better apps.

Case in point: At The Times we have an Inside Congress app that displays information about votes and bills in Congress. The tool that underlies that app is enormous – it has tons more information, and we’re working to surface more and more of it. But the tool – an internal interface – has uses for our congressional reporters, our graphics editors and for me as a developer. I can point a reporter to the vote record comparison tool instead of having to run a database query or, worse, asking someone else to manually recreate something. We use the tool as a sort of canary in the mine to alert us to odd or interesting events, from committee assignment changes to bill sponsorship withdrawals to unusual voting patterns. In some cases, having the data internally has led to improvements in the app itself, such as our “key amendments” pages for certain bills. I didn’t think of that, but someone else who saw the internal tool did, and we built it together.

Perhaps most important to me as a developer, building the internal tool has broadened the number of people I work with and has given me a range of ideas for making apps easier to build and better. Not all of them pan out, but some of them do. Put another way: the tool actually helps me develop closer working relationships with my colleagues.

A good tool doesn’t just make it easier for a reporter to create a story. It actually seeds the story, or makes it possible for more people in a newsroom to collaborate. When you have data but no tool, you become a gatekeeper of a sorts – which is appropriate in many circumstances, but not all. I can’t possibly know what my colleagues are thinking about, considering or being alerted to, but I can make it easier for them to test out theories and do some exploration on their own. Some of them prefer to do their own work, and we certainly miss some opportunities for apps that way. But others consult with me quite a bit, since they now have a much better idea of what we have and what we might be able to do with it.

Skeptics might respond that there is a difference between tools built around journalistic content, like the Congress app, and those that “merely” solve a technical problem. This is a short-sighted argument. What we do as builders of Web applications (external or internal) is informed by everything we touch. Pulling a piece of one tool for use someplace else is a useful technique because it reinforces the value of not repeating yourself and because it sometimes enables you to look at an old problem or situation from a new vantage point.

Back at washingtonpost.com, my former colleague Adrian Holovaty liked to say that we didn’t build internal versions of our apps because the public version was the internal version. Fair enough, to a point, but I think that line can veer into the data ghetto when not rigidly policed.

Most of my colleagues, I’m confident, have very little idea what it is that I specifically do. Sometimes I spend the time educating, and sometimes I let our tools help with the evangelization process. However they see my work, I’m pretty happy as long as it contributes to our journalism together. App developer? Sure. Tool maker? Why not. Labels don’t interest me much, and most of my colleagues don’t seem to care. The results – the journalism – are what matter.

These are complicated questions, made moreso by the options now available for teaching database skills. When I attended an IRE database bootcamp in 1997 (taught by my now-colleague Jo Craven McGinty), there were basically three options: the then-young Access, FoxPro or Paradox. Hard to believe, but back then I worked in a newsroom that had FoxPro and Paradox, but not really Access (Note: if you are under 30 and reading this, you may not even know what FoxPro and Paradox are. That’s ok. They, an in particular FoxPro, were wonderful database managers in their day.)

Not only do we now have open source options (SQLite, MySQL, Postgres) and SQL Server, but we also have a variety of “database-like” Web applications, like Fusion Tables and Google Refine, that can do some of the things that only desktop software used to do. And let’s face it, Excel is a very powerful tool for data analysis. Many of the things a reporter might want to do to a data file, such as sorting and filtering, are arguably a lot easier in Excel or another spreadsheet.

So why even teach SQL, then? The reasons I do it, and will continue to, are these:

1. SQL is an excellent and relatively simple way to enhance your data interviewing skills. When you have to write out your questions, you tend to think about them a little more than if you’re just pointing and clicking around. This is why when I had to teach Access, I bypassed the visual query builder. Yes, SQL queries involve writing more than doing an Excel filter, but those syntax errors also make you consider what you’re doing, and that’s a good thing.
2. SQL is still common enough on the Web that teaching it provides an additional branch, if you will, of learning, or at least makes it easier. When I explain how Facebook assembles all your friend’s posts, comments and pictures, I usually do so by pointing out the existence of FQL. If you already know SQL, it’s a very small leap to understanding, at a basic level, how Facebook works.
3. There are some times when you will absolutely need to use a SQL database. Or, at least, something that’s not Excel. Multi-million-row tables. Regular expression-based pattern matching. Intensive, complicated queries. If you haven’t explored SQL, you might not know these are even possible, and you might give up.

As to what to use when teaching SQL, I stick with SQLite despite Sarah Cohen‘s completely valid point that date and time support is much more complicated than it should be. Perhaps a new installment of Troy Thibodeaux’s excellent tutorial will help address that issue. In the meantime, let’s keep teaching SQL – and asking questions.

You might ask, “How is it that you’ve managed to keep a job in journalism, then?”

There’s no single answer to that, although the majority of it would have to be everything I learned from being a member of Investigative Reporters & Editors. And of that part, what I’ve really learned to love and work at is the other kind of interviewing. The one you don’t hear much about in journalism school: interviewing data.

To be fair, you really don’t hear all that much about the craft of interviewing people at journalism school, either. There is the occasional class, but the way that most people I know get better at it is simply by doing. When people ask me how I can approach complete strangers and ask them detailed and occasionally personal questions, I’m quick to reply that I spent four summers delivering breakfast in bed to newlyweds in the Poconos. When you’ve had a naked man answer the door at 8 a.m. and tell you to put the trays down next to the tripod-mounted video camera, talking to evenly partially-clothed strangers gets pretty easy.

Interviewing data takes practice, too, although I can’t really find a parallel from my days waiting tables. Both kinds of interviewing have much in common: you want to be as prepared as possible so as to better evaluate the results and be able to adapt your questions to the situation. Both require you to place a solid block of skepticism, even suspicion, on your shoulders as you embark. And both, if done well, can result in an unexpected admission – something even the subject of the interview didn’t really “know”.

This is why I continue to teach spreadsheets in classes, because they make for excellent initial interview tools. Looking at some data in a spreadsheet, you can easily size it up with basic sorting and filtering. That’s kind of the “getting-to-know-you” phase of the data interview. What are the ranges of this data? What looks unusual? Just as you get first impressions upon meeting someone, you get similar feelings about data.

With data you have to ask all the basic questions you do with a person, just so you know exactly what you’re dealing with. Questions like: “How old are you?”, “Where were you born?”, “Who do you report to?” work for both people and data (although I suppose “made” is a better word than “born”). And then, once you’ve got a solid foundation, you ask the trickier questions, the ones that you need to really think about. The ones that, when you’re planning a big interview with the subject of your investigation, you game-plan and write out as if they were lines in a soap opera.

And that’s where the big difference is: with data, you can ask a lot of potentially embarrassing questions, and the data won’t complain, walk out or threaten to sue. You can ask variations of the same question 20 times and the data won’t mind. When I say that I prefer interviewing data to people, this is why. Data will only lie to you if it’s just bad data or if you misunderstand the question. Unfortunately, almost every data set is “bad” in some way. But once you find that out, you usually can deal with it.

With the increased availability of information in structured forms, the skill of interviewing data is even more valuable now that it has been in the past. And yet it’s still considered a niche, a specialty skill. It’s odd, because what makes a good interviewer is not whether she uses a digital recorder or a pen. The technology itself is a tool. The crucial factor is the skill in being an interviewer – preparation, knowing what questions to ask and knowing when something isn’t right.

You wouldn’t stumble into an interview with a source having done no research, no preparation. Why in the world should journalists treat data sources any differently?

As part of our participation, panelists were asked to write a 1,000-word comment on the topic of their panel. Mine is “What are the common technical challenges journalists face in making sense of government documents and analyzing government actions, and how could those be overcome?” Faithful readers will know that I could probably go on for hours on this, but here is what I sent to the conference organizers (sans links):

I very much doubt that the conference organizers intended this, but the fact that our responses to these questions were requested in either PDF or MS Word formats is an excellent example of one of the technical challenges for journalists when dealing with government documents. So in the spirit of openness, I wrote this in Google Docs.

Both journalists and government employees who create and manage information need to know about more than the usual options for the collection and dissemination of information. Part of the technical failure rightly belongs to journalists — too often, we don’t ask or don’t know how to ask for information in a way that makes it easy to use. But far too often, government officials are either unaware of their format options or, more perniciously, all too aware and resort to distributing documents in, for example, a locked PDF.

I have been told many times that to release information in a format that would allow it to be copied is not the policy of a government agency. Those government agencies fail to understand what public information is. I also have been the recipient of records that clearly were stored in spreadsheet software but, for purposes of public release, have been printed out, scanned into images and stored as a PDF. Obfuscation and paranoia are not technical challenges, but they contribute to them, forcing journalists to acquire costly software or spend additional time overcoming an artificial roadblock.

This challenge is not due to deficiencies in software produced by any particular company but rather in the understanding of how information can and should be made available to citizens. To the greatest extent possible, government information should be made available in formats that allow its users to copy, sift and reorganize it as they please. In practice, this means favoring text-based and open formats over images, PDFs and closed formats. I am less concerned about how agencies store their data, as long as they are able to export it in common formats or reliable workarounds exist, but there are exceptions to this.

Map data, for example, is commonly stored at the government level in ESRI’s shapefile format, which, from the point of view of a journalist, has advantages and disadvantages. ESRI is a large, well-known company with products in use at most government agencies that have geographic data, so it makes sense that GIS data would be provided in that format. But not all newsrooms, and certainly not all journalists, are able to obtain ESRI’s software or have access to someone who can easily convert from shapefiles to other formats, such as the KML standard now owned by Google. Some government agencies already produce useful geographic data in KML format, but many others could join that list.

Fixing this situation will require education of journalists and government employees of the benefits and ease of working with open formats. The benefits for journalists are apparent: faster access to information that they can immediately put to use. Training journalists is a time-consuming and inefficient process, but journalists must break out of the mindset that government information only comes in documents.

Doing so means that journalists need to become as comfortable interviewing data as they are interviewing people. The benefits for government may need some more time to explain, but they exist. The Federal Election Commission is a case in point.

Thanks to a commitment to maintaining stable, available and well-documented data, the FEC makes it possible for its users to obtain and analyze information when they want to, even on a late-night deadline. This isn’t new-fangled technology; the FEC’s FTP site has been operating for years. But the agency operates as if it trusts its data users, not from a defensive standpoint. As a result, the FEC is rightfully seen as an agency that makes it possible for reporters to do their jobs, not as an impediment to that goal.

A more recent, but increasingly significant, technical challenge is that too many government agencies fail to make better use of the best information distribution platform they have: the Internet. In a digital age, some agencies continue to treat all records either as documents, or when they do make data available, it is done as a single dump. In many cases, journalists do not need an entire dataset; they are more likely to want to answer a single question or small set of questions. In those cases where government agencies make this possible, it is usually through a Web form of their own design – one which often is tailored to heavy users such as the regulated community.

Providing Application Programming Interfaces (APIs) to government data via XML or JSON feeds would make it possible for journalists and Web developers to take advantage of government data without having to download and process enormous files. And while adding an API will incur an up-front cost, it will also save agencies employee time handling requests that could be done computer-to-computer.

Yet such APIs are very rare in government, even though they would make it easier for users and journalists to combine disparate data, and would make it possible to build more useful applications from government data. We know this to be true, because in the absence of any meaningful government approach to disseminating legislative data, several outside organizations, including my own, have developed APIs to help spur the use and spread of congressional data.

But in order to do this, we have had to essentially reverse-engineer the Thomas site operated by the Library of Congress, writing fragile HTML parsers that can break should the LoC change the structure of individual pages. So, in order to answer anything beyond the most basic question on legislative matters, a journalist must either spend hours looking up information one page at a time or be able to write a computer program to parse those pages. There has to be a better way.

Most folks in journalism know about the Wall Street Journal’s famed ‘A-Hed’ stories – the ones that used to be in the middle column of the paper that are off-beat, sometimes funny and extremely well-written. It turns out that the Journal has a landing page for those pieces, which is a great idea. My Northwestern colleague Matt Mansfield pointed it out to me after reading one such piece by Danny Yadron, a former student.

It’s nice to have a single spot for the A-Heds, but alas, there is no feed, as my former colleague Amanda Zamora pointed out. Problem solved. Thanks to Google App Engine and Python, there is now. It’s updated daily and the code (such as it is) can be found on Github. Add it to your feed reader today!

Here’s a basic example of a chart using the WordPress plugin; it charts the percentage of drives that ended in touchdowns for six not-quite-randomly-selected college football teams from the 2010 season (data from College Football Reference).

Note: This post contains a interactive data presentation that may not show up in your feed reader. For the full experience, visit this article in your web browser.

• drive outcomes - Google 文档

The simple answer is that APIs are an extension of what reporters do every day: ask questions. The difference is that instead of forcing reporters to gather data from multiple sources, format it to fit your local database needs and then update that database when new releases are available, APIs allow reporters to query live data from all over the Web. If you have experience working with, say, Microsoft Access and setting up an ODBC connection to a remote database, APIs are kind of like that – except that you have near-instant access to more sources of data, more useful tools (like geocoders) and more timely information than ever before.

My path working with data went something like this: spreadsheets came first, which I routinely describe as the “gateway drug” of computer-assisted reporting. Some people become such Excel wizards that it almost doesn’t make sense for them to move beyond that expertise; there is so much you can do in a spreadsheet that alone it would be worth the time to learn. But there were things about spreadsheets that annoyed and frustrated me. Pivot tables were a clumsy fit for me – they got me close to what I wanted in many instances but never quite there. And so I moved onto databases.

Databases are still one of my favorite things. They are powerful, relatively flexible and range in utility from the ultra-portable SQLite to the transactional goodness that is PostgreSQL. But they take time and effort to build, maintain and – perhaps most importantly in the long run – connect to additional sources of information. APIs are not a complete solution to these problems, but they provide a very good one that data journalists should be familiar with and consider incorporating into their work.

A simple example is the reporter who wants to track the votes of his or her state’s delegation in Congress. There are several APIs for this data, including the one I work on and another by OpenCongress. The reporter could build a database of these votes by hand or write scripts to parse the House and Senate vote data and insert them into it. But why, when the data is freely available via HTTP?

It can’t be that simple, can it? Well, no. But it can be simpler. The data you get from APIs usually comes in XML or JSON. Data journalists have, for better or worse, been dealing with XML for awhile now. JSON may be less familiar, but it is quite nice to deal with and there are plenty of libraries with which to do so. But even better than that is the fact that other people have already solved that problem for you. Not long after we released the NYT Congress API I noticed a Ruby client library for it on Github. I had never met the author; he had never contacted me. Just the same, he made it easier for people using Ruby to query the API and get back data. There’s also an excellent Python library for it, written by NPR’s Chris Amico.

Thus can you, the data journalist, benefit from other people who need and use APIs. Check out GovKit, a Ruby wrapper to multiple government and political APIs, created by the folks at the Participatory Politics Foundation. Go play with it, and figure out what sorts of things you can do when the number of data sources you’re able to tap into multiplies overnight. The possibilities for journalists are only limited by the kinds of questions we can imagine and try to answer. APIs can make it easier to act on that greatest of questions: What if?