Although I didn’t finish graduate school, I have written my fair share of academic papers, so I’m not a complete novice in the area. And despite the fact that it is a draft, as Usher points out, the idea that you would post even a draft on the web and then profess surprise when someone links to it or reads it is a little off, particularly for someone writing about a) journalism and b) open source. If you weren’t expecting critics, maybe it’s a good idea not to let them see the material. Update: in fairness, Nikki Usher was not aware that the paper was online at all. (I should also note that criticism is my full-time non-job; I probably should have sought work in the field, not due to any real talent on my part but based purely on personal enthusiasm.)

There is a good bit to like in the paper, which will be presented this month at the International Communication Association’s annual convention. There is a tendency for those working at the intersection of technology and journalism to focus on the tools – the stuff that actually exists now – rather than systemic changes to journalism itself (there are ahem, some exceptions to this tendency). In part that’s because systemic changes are a hard problem too easily pushed to the background by the demands of doing journalism today. But it’s also in part because incremental changes, as the authors note, can be valuable in changing the whole. But, ok, I get it, and in many ways agree (except for the utopian one-open-source-CMS-to-rule-them-all idea – can we just kill that fantasy and focus on making information portable?).

Then I read the part about Fech. As a contributor to the project, I am admittedly biased in its favor, but I do not think my reaction is solely or even mainly based on that fact. Here’s what they wrote about the project:

The New York Times developed Fech, a tool that helps journalists crawl financial disclosures by political candidates just by knowing a filing number (Strickland, 2011). Just as the discourse around open source tools emphasizes their pro-social benefits, Fech’s creators note that more access to these filings will lead to better journalism. But Fech also gives one more tool to journalists eager for the horse-race style journalism that is divisive and counterproductive for democracy (Patterson, 1993; Cappella & Jamieson, 1997). There is another problem with what could be a strength of Fech: While the source code is posted on Github for other developers, the tool has been built to help people in the newsroom, not to encourage participation by ordinary people.

Where to begin? First, and perhaps least important, it vastly understates what Fech enables journalists and developers to do, particularly in regards to what can be done programmatically with disclosure data. I do not think that it is a stretch to say that the easier it is to examine and search campaign finance disclosures, the easier it will be for reporters and the public to discover interesting and useful pieces of information. Indeed, the use of Fech by news organizations like ProPublica, Reuters and The Associated Press – to say nothing of our use at The Times – has borne that out.

But here’s the line that got my back up: “But Fech also gives one more tool to journalists eager for the horse-race style journalism that is divisive and counterproductive for democracy.” The evidence for that? There isn’t any. This is pure speculation; I would argue that it is refuted by the examples I just cited. Can the authors — or anyone — cite an example of where Fech has been used to enable more horse-race style journalism (in its pejorative sense, which is what I assume the authors meant)? I’m 17 years out of graduate journalism school, but I’m pretty sure that assertions like that need a bit more than a citation to work that says horse-race journalism is bad for democracy. In theory, Fech could be used to run nuclear reactors, I guess, but since there is no evidence of that actually happening, I’m going to discount that as a possibility.

And finally, the authors write: “While the source code is posted on Github for other developers, the tool has been built to help people in the newsroom, not to encourage participation by ordinary people.” Well, yes and no. I would be hard-pressed to describe those people interested in campaign finance data as “ordinary,” but we open sourced Fech so that it could be used in newsrooms and in any other situation. I don’t understand how exactly the authors presume that we built it only to help people in the newsroom, or to discourage participation by non-newsroom folks. The fact that the contributors to Fech come mainly (but not exclusively) from newsrooms who cover campaigns is understandable to me (and to Jeff Larson). I’m not entirely clear how what we’ve done makes it harder for non-newsroom people to participate. I’d love to read about that, but there are no examples or further discussion in the paper (nor was any user of Fech contacted by the authors, from what I can tell. I wonder if they installed it and tried it themselves).

Usher, to her credit, offered several explanations as to why this passage was in the draft. They include the fact that this paper, like many, undergoes blind review. Fair enough, but it’s worth asking whether the reviewers are able to evaluate these claims, since most of them could be debunked by reading posts on Open. It also, like many of my own projects, seemed to have been a bit of a rush job. I know all about that, but was it really so difficult to talk to anyone involved in Fech? Finally, my objections, however valid, don’t damage the overall point of the paper, but reflect the possibility that I “may be getting lost in the weeds.”

That’s the tricky thing about journalism, data and even open source. Weeds matter. If you get the weeds wrong, the eventual result usually suffers. If I’m lost in the weeds, maybe the garden needs some attention.

CBS’s Mark Knoller, who keeps detailed notes on Obama’s travels, recently told The New York Times that since the president filed for re-election, he’s taken 60 domestic trips and 26 of them involved fundraisers.

But Mark Knoller of CBS, the unofficial keeper of presidential work schedules, reported that President George W. Bush had taken more time off than Obama at this point in his first term.

According to presidential watcher Mark Knoller of CBS, George W. Bush, at this time of his presidency, had made 30 visits to his Texas ranch spanning all or part of 220 days. The Obama’s vacation day count is less than half of that.

This isn’t about Knoller as a person or as a reporter. It’s just that this situation – where one person has become the official source of public knowledge about the travels of the President of the United States – is far from ideal. Forget that the government is occasionally off-base on presidential travel statistics; how is it that other news organizations, including my own, have relied on a system in which one person – however diligent and generous – holds such important information?

From an information management standpoint, having Knoller be the keeper of presidential travel information is not only inefficient – what happens if Knoller is on vacation, or busy? – but makes it harder to regularly review the data or incorporate it into other inquiries. In reality, this is our problem, not Knoller’s, and his generosity has enabled us to carry on as if we’d been collecting this information all the time. But we haven’t. It’s easy enough to just ask Knoller, especially since we don’t use the information all that often.

We’re not talking about uncovering classified information here, but the daily whereabouts of the President of the United States. And yet somehow, every other news organization has decided that it’s perfectly ok not to have this information at its fingertips. It probably won’t happen as long as Knoller remains in his job, but what happens if someday CBS decides not to share that information anymore? Or Knoller decides he’s tired of doing this and retires? In the “weak link in the chain” scenario, the rest of us are the weak links, not him. He’s doing his part. Why are we shirking ours?

I finally have something tangible from work to show to my mother: an A1, above-the-fold story in today’s New York Times. It doesn’t really help explain what I do, but it’s something that’s a bit easier to understand than, say, a listing of git commits.

A colleague of mine at The Times, Michael Strickland, responded on Twitter: “Enough about the designer-programmer. More about the programmer-writer.”

As usual, he’s onto something, particularly when it comes to news organizations. Literate programming has been around quite a while, and I’m lucky enough to work with people who approach code in a way that seems closer to artistry than to engineering. I’m no expert on such things. But I was a full-time reporter for nearly a decade, and I still do my share of reporting, and that’s where I see the greater potential of applying programming: the Programmer-Reporter.

Let’s stipulate right up front that I, like a lot of folks, am a sucker for an Anne Hull story, the kind built on hours and hours of listening, watching and reflecting. What follows is no knock on what my former Palm Beach Post colleague Ron Hayes used to call “notebook-assisted reporting” built on talking to people, writing down what they say and then turning that into a great story. The only reason this post is not about that kind of journalism is because I was never really much good at it.

There are other stories to tell, and other ways to find them. I usually tell classes that I teach that if any of them can write like Hull or has the source development skills of, say, Bob Woodward, then they probably don’t need to learn what I’m teaching them. But the rest — the vast majority, from my experience — may want to pay attention.

A lot of daily beat reporting – from sports to government to business – relies heavily on reporters knowing the habits and schedules of the people and institutions they cover. Certain events happen in a relatively predictable pattern and a lot of the reporting revolves around keeping tabs on them. But news, the stuff we talk about, often consists of things that go against that pattern, the unusual event in a sea of regularity.

It follows then, that journalists should prize methods that would help unearth such anomalies, those needles in haystacks that we hold dear. Some do, and in other cases there are few real methods other than examining every document or attending every meeting. But way too often, across topics and beats, we remain unaware of or ignore practices that could help us spot news and make sense of the larger picture. If we have a system of story development, it’s a system that seems to value serendipity and entropy. Meanwhile, Donald Rumsfeld’s line about “known unknowns” remains stubbornly in effect.

This is true even in areas where reporting relies heavily on data, such as political campaigns. Many of the stories relating to campaign contributions, for example, are a result of reporters meticulously poring over pages of filings, applying the Potter Stewart test: “I’ll know it when I see it.” This is, in too many cases, a waste of time, since we often do have an idea of what we’re looking for, but believe in this idea of “data serendipity” when practice shows us that asking specific questions, or at least about specific ideas, is a better way to go. The easiest question for any source to deflect is, “Anything interesting going on?” Unfortunately, it’s also the easiest one to ask.

One thing that I’ve learned from writing software is that you don’t really want to “make news,” as it were. Predictability is a good thing, and edge cases – when things get weird or different – are what you want to avoid. Reporting, on the other hand, seeks out the edge cases, the departures from the norm. How to make those two come together? Here’s a way: make it possible to expect the unexpected when it comes to analyzing patterns in data. What we need are easily configurable systems that enable reporters to ask questions of data in a consistent manner and then provide results in a way that makes sense for journalists.

I’m not talking about bland TPS reports but something that turns a piece of data into a potential story. For example, if a local congressman has received donations from executives of XYZ Corp. every March in previous election years, but not this year, then that’s potentially newsworthy; maybe XYZ isn’t giving as much, but maybe they no longer support the local politician, or are a bellwether of a lack of business support. In this case, the absence of data – something that’s very hard for people to spot in pages of filings – is the trigger event that can cause a reporter to follow up.

As much as journalists love to wax about serendipity, much of life is based on our habits and patterns, which are predictable enough to be tested against data. The same idea applies to scenarios that may not have happened in the past but could be defined and applied to the data. Reporters are testing out theories all the time, often by calling up sources and putting a question or theory out there. There’s no reason why we can’t enable the same ease of inquiry with data. In fact, it seems possible to do it on a much broader scale and in more precise ways.

Will this lead directly to stories? Not in many cases. Again, this is a typical process for reporters: float an idea, test its merits and polish. Rinse and repeat. If we could build interfaces that would assist in that process by making it easier to ask questions (or define patterns), how many more stories could we find? How many known unknowns could be crossed off the list?

A lot, I think. And despite the thrill of seeing your name on the front page, this is the thing that keeps me energized. This is where programmers can make an impact – either by diving into a subject area or pairing with reporters who have that expertise – by making it easier to find better stories. The news is out there, in most cases. Finding it, nailing it down, that’s the challenge. It is, in programming terms, a hard problem. But the programmers I know love to tackle hard problems.

The working question going into the story was that despite the headlines garnered by the freshman class of Republican lawmakers in the House, it wasn’t clear that they were mainly responsible for opposing the GOP leadership, particularly House Speaker John Boehner. Jennifer first outlined a series of votes that attracted Republican opposition (in the House, the majority’s first obligation is to satisfy most, if not all, of its caucus. A significant number of “No” votes from the majority’s ranks is a clear sign that there are some issues to resolve). Then I pulled the vote data from our API, although the same data is available from other sources, too.

We looked at the freshmen Republicans as a voting bloc, and then at members of the Republican Study Committee, the largest sub-group within the House GOP. Since such organizations aren’t technically legislative committees, we had to rely upon its list of members (which turned out to add a wrinkle to the story). Here’s what we found:

But an analysis of voting patterns on the most contentious bills in the 112th Congress shows that House members of the Republican Study Committee — a group of both veterans and newcomers that meets weekly to hammer out a conservative agenda — have cast the bulk of “no” votes on big bills, including those important to Speaker John A. Boehner of Ohio.

The freshmen who have joined the study committee — which was founded in 1973 — play an important role in its renewed clout, having increased its membership to 163 from roughly 110 two years ago. As a group, however, the freshmen are less homogenous and less apt to buck the leadership than the study committee itself is as a whole.

It was relatively straightforward, as data analysis goes, since all we needed to do was pick the universe of votes, pull the data and identify freshmen and RSC members. Identifying freshmen lawmakers seems pretty easy, but there are a few questions to bear in mind. For example, are lawmakers who were elected in special elections before November 2010 freshmen? Are former members who reclaimed their seats in 2010 freshmen? What about people elected after 2011? (We said yes in all cases, which enlarges the usual number cited as the 2010 freshman class).

With RSC members, it was a little more involved, in that an earlier version of the RSC member page indicated that membership had changed somewhat since last July. That became an element of the story:

Since then, over a dozen members have resigned from the study group, including a few freshmen. In interviews, some said that the episode had angered them and that they had tired of the committee’s attempts to define who is worthy of being called a conservative. Most were afraid to tackle the group on the record.

Vote analysis can seem like a dry exercise – there are so many votes, and most of them are fairly lopsided affairs. But voting is important not only for the individual decisions that lawmakers take, but for what it can tell us about the collective behavior of members.

Being a former Congressional Quarterly staffer, I have an innate fondness for House Admin, one of the lesser-known committees but one with a large influence over what kinds of information the public can see about the House side of the legislative process. The committee has jurisdiction over the Library of Congress, which by extension means Thomas, the online home of so much Congressional information.

There are many other posts about the general desires of what those folks committed to transparency want when it comes to Congress, but Daniel Schuman of the Sunlight Foundation sums them up pretty well: “To the maximum extent possible, legislative information must be available online, in real time, and in machine readable formats.”

I don’t disagree, and I am sympathetic to complaints that Congress has been slow to address the availability of bulk data. People such as Josh Tauberer have been screen-scraping Thomas since 2004, and I joined in the process a year later at washingtonpost.com. In 2012, we’re both still doing it, now joined by Sunlight, OpenCongress and who knows how many others (speaking of OpenCongress, if you want a less patient restatement of Schuman’s thoughts, OC’s David Moore has a stem-winder of a post for you).

I, too, long for the day when I don’t have to wonder when my HTML parsers will break after a seemingly innocuous change to Thomas’ styles, or when I don’t have to enter three different IDs for a new Senator (Bioguide, LIS and Thomas’ own unique sequential number). But my presentation on Thursday concentrated on a more fundamental need. Before bulk data can become really useful, it has to be more consistent, understandable and accurate. Right now, if you’re not willing to put in a lot of time studying the quirks of Congress, you will always face the likelihood that your data, however lovingly collected, has plenty of errors.

For example, in the Senate it is possible for the Majority Leader and Minority Leader to alter the rules of math when it comes to how many senators constitute a three-fifths majority. The death of Sen. Ted Kennedy in 2009 reduced the number of Democrats in the chamber at that time to 59, and the total number of senators “duly elected and sworn” to 99. For votes requiring a three-fifths majority (thanks, Malcolm), a 99-member Senate would need 59.4 senators for passage, or at least 59. But the party leaders agreed to keep the three-fifths threshold at 60 votes throughout the period when the Senate had 99 senators, not 100. For much of that period, any two-thirds vote displayed on nytimes.com had the wrong number of votes required for passage, because I was relying on math. I could not find any place in the Congressional Record or anywhere else where this was documented.

An edge case, you might say. But when it comes to Congress, there are loads of them. A reporter called me several weeks ago to ask why a seemingly simple question about three members of her state’s delegation was maddeningly hard to answer. All three had been elected to the House the same year, and had served since then. But each of them had a different number of total votes he or she was eligible to vote on. How could that be?

It took me a little while, but the only explanation I could find was that their dates of service had to differ in some way, and my guess was that not all of them were sworn in for each session on the same day. It happens. Unfortunately, neither Thomas nor the Clerk of the House provides an easy way to find out when a particular member was sworn in, despite the fact that it is a basic element of what makes someone a Member of Congress. At the conference, I heard someone say that it would be possible to provide a list of swearing-in dates for every lawmaker. That’s good, and needed, but it’s not good enough. I need, and the data demands, timestamps in this case. That’s the only way I can be sure of what votes a member was or was not eligible to vote on.

You might think that you could find the total number of House votes for a given year by looking at the Clerk’s votes site. In 2011, the last vote was roll call 949. Alas, officially, there were 948 votes that year, because roll call 484 was vacated and replaced by vote 485, and thus never really happened.

In my presentation I cited a few other examples, but they mostly boil down to this: unless we can make congressional information easier to use and understand by people outside the small circle of legislative wonks, bulk data access by itself won’t solve our problems. Today the most creative uses of congressional information, such as Sunlight’s Capitol Words project, suffer from this limitation. I love Capitol Words, but right now the Congressional Record – the source for it – cannot reliably tell me in a machine-readable form whether a particular word or phrase or speech was even spoken out loud on the floor of the House or Senate. That’s kind of a big deal, for reporters, historians and the public.

If we can’t use congressional data to answer what should be straightforward questions, or can’t agree on what the answers should be, providing immediate access to that data in bulk form may not be as helpful as we would think, and in some cases risks adding to the confusion. It may expose more of those problems, which is of some usefulness, but if the ultimate goal is not just access but understanding, we need to address the fundamental issues of accuracy and consistency before we switch on the firehose.

First, I stand by what I said about how little we understand about the way that elections are won or lost these days. It’s not that political journalism has strayed from its roots, or stopped covering important elements of a modern campaign. It’s that the elements of a modern campaign have changed, and as journalists, we have not kept pace.

You might respond that campaigns still involve quite a lot of stuff that we do understand, such as debates and visits to state fairs and town hall meetings. True. But the nature of media and technology has brought extensive changes to the electoral system, and I don’t believe that we as journalists devote enough attention to understanding those changes. Remember the Dean campaign in 2003? Most of the coverage was on the, for then, staggering online fundraising managed by some doctor from Vermont. But that Wired piece I referenced had it right; Dean’s accomplishment was less a mastery of the Internet but a willingness to embrace its fundamental aspect: you give up some control by bringing other people in, and you gain a host of possibilities. You may, of course, choose badly or falter in some other way, but the lessons and possibilities are becoming clear. At the time, as a Web geek who loved politics, I felt that journalists couldn’t really explain the Dean campaign, because it was so alien to us. Today’s campaigns make me long for the simplicity of 2003.

But let’s stick with fundraising for a bit. Political fundraising can be hugely expensive, because campaigns need to amass large number of donors. Unless you’re the President, it’s hard to repeatedly gather the wealthiest Americans and have them fork over $2,500 or more for the pleasure of your company. So a smart campaign sticks with what works: direct mail is costly, for example, but it’s also effective. Telemarketing takes time and money, but it also works pretty well. Let’s not mess with the script too much. But what if you can mess with the script? Now it’s possible, even trivial, to experiment with Web site design or even advertisements in order to gauge their effectiveness and improve upon them. President Obama had a Director of Analytics for his 2008 campaign, and has been hiring data scientists experienced in predictive modeling.

White men smoking cigars in cramped rooms making gut calls is how we’ve usually understood campaign decision-making. This? Whole new ballgame. Yes, there is still a mass audience that is shaped by the media and big events. But there are now thousands and thousands of “small” audiences – or rather, they always were there. Now campaigns can identify them and deliver precision messages to them. And they can find them online in different ways; an hour after posting on Twitter about the Obama’s campaign use of Github, the campaign’s Digital Director was following me. And that’s the easy part.

While campaigns have a public presence that is mostly recorded and observed, the stuff that goes on behind the scenes is so much more sophisticated than it has been. In 2008 we were fascinated by the Obama campaign’s use of iPhones for data collection; now we’re entering an age where campaigns don’t just collect information by hand, but harvest it and learn from it. An “information arms race,” as GOP consultant Alex Gage puts it.

For most news organizations, the standard approach to campaign coverage is tantamount to bringing a knife to a gun fight. How many data scientists work for news organizations? We are falling behind, and we risk not being able to explain to our readers and users how their representatives get elected or defeated.

None of this is to say that we need to completely abandon our ways of covering elections. Horse-race coverage is and should be a part of campaign coverage, because in many respects elections are like horse races. Things can change rapidly, and small things can have big impact. We still should be on the ground, talking to voters, showing up at town halls and covering debates. We still need to show up and do the legwork.

But if we can’t appreciate, much less understand, what modern campaigns are doing to win elections, how can we hope to explain elections? If we don’t collect at least some of the information available to us – realizing that we can’t get our hands on everything that the campaigns do – we’ll miss the story. Elections will become even bigger surprises to us, and then how long will it be before readers start to ask whether we actually know the people and places we cover?

Surprises make the news. Some of my favorite stories from the 2004 presidential election are in a book by my friends Peter Wallsten and Tom Hamburger, then of the Los Angeles Times. Here’s one anecdote from the key state of Ohio:

One suburban African American woman in Ohio, for example, told us that though she tends to vote Democratic, she was deluged in 2004 with calls, e-mail messages and other forms of communication by Republicans who somehow knew that she was a mother with children in private schools, an active church attendee, an abortion opponent and a golfer.

Think about what this kind of thing means. It means that we cannot assume that the campaign visible to the mass audience is the same campaign that’s being pitched to individuals and groups around the nation, and that winning coalitions can be built not just by harnessing large groups (unions, religious voters, etc.) but also by piecing them together in small units. President Bush’s margin in Ohio in 2004? About 2.5 percent. The only thing that I don’t like about this anecdote is that Wallsten and Hamburger’s book appeared nearly two years later. Is there any evidence that we as journalists have closed the gap since then?

To understand how elections are now being waged, we need to have as many of the tools as do the campaigns. We need to build our own storehouses of data – voter registration, voter history, Census, campaign finance, advertisements and more. We need to be able to tap into the rich stream of material that’s being created and disseminated every day. We need to be able to see the value in small data points that can lead to bigger things.

Elections are great stories. They deserve to be told from a position of confidence and knowledge. We have work to do.

If you’re a fan of the useful CSVKit, a command-line tool written in Python, but you’re a Rubyist, then please do yourself a favor and take a look at RemoteTable, a gem by Seamus Abshere. The two libraries aren’t entirely identical, and CSVKit has been rightly praised by others, so let me dive a little deeper into why RemoteTable is a data parser’s friend.

RemoteTable is essentially a set of useful wrappers around the common process I outlined above: grab a structured file and use it contents. Except that when I say “structured file”, I don’t just mean a CSV. I mean CSVs inside of zip files. Excel spreadsheets of the .xls and .xlsx varieties. Google Spreadsheets and Open Office spreadsheets. Web pages with HTML tables in them. XML. And, what the hell, fixed-width files, too. You’ll want to see the examples.

To those options it adds some useful utilities, much as CSVKit does. You can cut certain columns using the Unix cut utility, skip or crop rows using tail, make the entire file UTF-8 or remove “useless” characters.

It’s simple to try out. If you have Ruby installed and are not on Windows (sorry!), then gem install remote_table and wait a minute or two, as it installs quite a few dependencies for managing all this stuff. Then, in a console session, try grabbing a CSV file:

Each row is converted into an Ordered Hash so that you can refer to columns by name instead of position.

Now, what about a fixed-width file? Maybe one in which I’m only interested in some of the columns? Easy, thanks to RemoteTable’s nice DSL:

Notice that :cut option I passed? That’s where you include arguments to cut. In this case, I removed the financial columns from this file. You can also pass a :select option to it to do on-the-fly filtering of certain columns based on a matcher such as a regular expression or plain text.

RemoteTable can tell you when you’re not being very smart about its use, too, such as causing it to re-fetch the source file multiple times (it warns you when it does this). It also handles local files, so if you want to avoid the downloading bit (or your files aren’t on the Web), that’s fine, too. Honestly, I have no idea why this library isn’t more popular, but you can help make it so on Github.

Swizec Teller’s entry, “Measuring vocabulary richness with Python“, described an algorithm by George Udny Yule in a 1944 paper entitled “The statistical study of literary vocabulary.” Yule created a way to quantify the diversity of vocabulary in a given text, and Teller translated that formula into straightforward Python code.

So I made a simple Django app that accepts text via a form and uses Teller’s code to calculate Yule’s I score of vocabulary richness. It uses the really useful Natural Language Toolkit; the only oddity is that when developing locally on a Mac, the standard installation of NLTK via pip is borked, so you need to specify a file to download in your requirements.txt. You can find the demo app here.

I’m not offering a judgment on using Heroku or other instant deployment-type services; most of them seem pretty easy to use but out of my price range for anything significant. But it’s nice to know that services like Heroku, ep.io and others offer enough flexibility to do stuff like natural language parsing.

Done snickering? Let’s move on.

The Tools Team built mostly internal applications and services that helped the Web site run better. I mainly got to work on front-facing projects like the Congress Votes Database, the 2008 presidential campaign and an innovative series on lobbyist Gerald Cassidy. But I did work on a few internal tools, and since I joined The Times in late 2007 I’ve built a few more. I’ve found that such tools are not so different from what we now consider to be journalism by Web development. Chosen wisely and done well, they can have impacts that go far beyond a single story or series. We should not dismiss them as “not journalism.”

If you’re at the geekier end of the journalism spectrum, then chances are your colleagues know about the stuff you can do. They may not understand it or be able to explain it; a former managing editor of mine, when told about the various technical steps to accomplish something useful, would invariably respond with a touch of wonder: “Fuckin’ Internet!” You can explain your work to a decent percentage of your colleagues by invoking Harry Potter or the Lord of the Rings and leave it at that.

But that doesn’t mean that building tools that can be used by broad segments of the newsroom is a one-way street or has to lead to a divide between you and the other journalists. There will be people in every newsroom who mainly take and rarely give, and in those situations being a technologist is no different from being a clerk. Good tools, like good apps, are a product of collaboration and improve the ability of the newsroom in general. They also make for more and better apps.

Case in point: At The Times we have an Inside Congress app that displays information about votes and bills in Congress. The tool that underlies that app is enormous – it has tons more information, and we’re working to surface more and more of it. But the tool – an internal interface – has uses for our congressional reporters, our graphics editors and for me as a developer. I can point a reporter to the vote record comparison tool instead of having to run a database query or, worse, asking someone else to manually recreate something. We use the tool as a sort of canary in the mine to alert us to odd or interesting events, from committee assignment changes to bill sponsorship withdrawals to unusual voting patterns. In some cases, having the data internally has led to improvements in the app itself, such as our “key amendments” pages for certain bills. I didn’t think of that, but someone else who saw the internal tool did, and we built it together.

Perhaps most important to me as a developer, building the internal tool has broadened the number of people I work with and has given me a range of ideas for making apps easier to build and better. Not all of them pan out, but some of them do. Put another way: the tool actually helps me develop closer working relationships with my colleagues.

A good tool doesn’t just make it easier for a reporter to create a story. It actually seeds the story, or makes it possible for more people in a newsroom to collaborate. When you have data but no tool, you become a gatekeeper of a sorts – which is appropriate in many circumstances, but not all. I can’t possibly know what my colleagues are thinking about, considering or being alerted to, but I can make it easier for them to test out theories and do some exploration on their own. Some of them prefer to do their own work, and we certainly miss some opportunities for apps that way. But others consult with me quite a bit, since they now have a much better idea of what we have and what we might be able to do with it.

Skeptics might respond that there is a difference between tools built around journalistic content, like the Congress app, and those that “merely” solve a technical problem. This is a short-sighted argument. What we do as builders of Web applications (external or internal) is informed by everything we touch. Pulling a piece of one tool for use someplace else is a useful technique because it reinforces the value of not repeating yourself and because it sometimes enables you to look at an old problem or situation from a new vantage point.

Back at washingtonpost.com, my former colleague Adrian Holovaty liked to say that we didn’t build internal versions of our apps because the public version was the internal version. Fair enough, to a point, but I think that line can veer into the data ghetto when not rigidly policed.

Most of my colleagues, I’m confident, have very little idea what it is that I specifically do. Sometimes I spend the time educating, and sometimes I let our tools help with the evangelization process. However they see my work, I’m pretty happy as long as it contributes to our journalism together. App developer? Sure. Tool maker? Why not. Labels don’t interest me much, and most of my colleagues don’t seem to care. The results – the journalism – are what matter.

These are complicated questions, made moreso by the options now available for teaching database skills. When I attended an IRE database bootcamp in 1997 (taught by my now-colleague Jo Craven McGinty), there were basically three options: the then-young Access, FoxPro or Paradox. Hard to believe, but back then I worked in a newsroom that had FoxPro and Paradox, but not really Access (Note: if you are under 30 and reading this, you may not even know what FoxPro and Paradox are. That’s ok. They, an in particular FoxPro, were wonderful database managers in their day.)

Not only do we now have open source options (SQLite, MySQL, Postgres) and SQL Server, but we also have a variety of “database-like” Web applications, like Fusion Tables and Google Refine, that can do some of the things that only desktop software used to do. And let’s face it, Excel is a very powerful tool for data analysis. Many of the things a reporter might want to do to a data file, such as sorting and filtering, are arguably a lot easier in Excel or another spreadsheet.

So why even teach SQL, then? The reasons I do it, and will continue to, are these:

1. SQL is an excellent and relatively simple way to enhance your data interviewing skills. When you have to write out your questions, you tend to think about them a little more than if you’re just pointing and clicking around. This is why when I had to teach Access, I bypassed the visual query builder. Yes, SQL queries involve writing more than doing an Excel filter, but those syntax errors also make you consider what you’re doing, and that’s a good thing.
2. SQL is still common enough on the Web that teaching it provides an additional branch, if you will, of learning, or at least makes it easier. When I explain how Facebook assembles all your friend’s posts, comments and pictures, I usually do so by pointing out the existence of FQL. If you already know SQL, it’s a very small leap to understanding, at a basic level, how Facebook works.
3. There are some times when you will absolutely need to use a SQL database. Or, at least, something that’s not Excel. Multi-million-row tables. Regular expression-based pattern matching. Intensive, complicated queries. If you haven’t explored SQL, you might not know these are even possible, and you might give up.

As to what to use when teaching SQL, I stick with SQLite despite Sarah Cohen‘s completely valid point that date and time support is much more complicated than it should be. Perhaps a new installment of Troy Thibodeaux’s excellent tutorial will help address that issue. In the meantime, let’s keep teaching SQL – and asking questions.