One of the common responses to such entreaties is exemplified in this comment, which includes this plea: “I understand the need to bolster one’s skill set. But what happened to the days when we actually, you know, worried about reporting rather than slavishly trying to master every piece of technology?”

If only that was the real problem.

The real problem is the way that we as journalists manage information, because that determines so much else: the kinds of stories we’re able to envision and construct, the amount of context we’re able to bring to bear in a short amount of time and our ability to connect the dots. In general, and this is my scientific conclusion, we suck at managing information.

That’s nothing new, you might say, and you’d be right. But what has changed is that a lot of the people and institutions we cover are now getting smarter about this stuff, and are using better tools to help them manage information. From tracking crime to measuring customer loyalty, the sophisticated use of information is a crucial factor in many modern activities. Us? We’re still knocking rocks around hoping to generate a spark.

I’m not knocking learning skills like how to maintain a blog. I’m just saying that if all we do is teach new tools and skills, we’re making the underlying problem harder to solve, not easier, because we’re just encouraging the production of even more separate and disconnected piles of information. More photos over here, more spreadsheets over there. We’re still drowning in information and we can’t figure out how to use it to our best advantage, like finding undiscovered patterns and coming up with definitive explanations instead of the ol’ three-person anecdote story.

So yeah, teach those CAR and multimedia skills. Have everybody Twitter. But please, let’s find a way to address the fact that for many journalists, Microsoft Word is the primary tool for organizing any and all kinds of information. Let’s make sure that our silos of content (text archives, photo archives, databases, etc.) can at least be made to talk to each other, if not naturally, then through APIs or metadata or something. And let’s start talking about how a news organization’s information belongs to the organization, not just to individual reporters and editors, and how our products could be so much better if we adhered to that principle before a story/photo/slideshow is published, not just after.

The Case for Access

Before I offer the case against using it, let’s look at why we would use Access to teach database concepts. First, it’s widely available as part of Microsoft’s popular Office suite. A lot of news organizations have it already installed, or can do so without much trouble. Second, it has a familiar look and feel for people who have used Excel - sorting and filtering work nearly the same way, for example - and imports and exports Excel files with ease. Third, the query grid that Access has a default makes it easy to get started on actually getting answers from your data.

That’s about it, as far as I’m concerned. I guess you could throw in the ability to generate reports and construct forms, but these are less of an advantage as web-based apps have become more popular and added features. In fact, the last reason I cited, the query grid, isn’t really an advantage at all, as I’ll explain below.

The Case Against Access

Access costs money. In SQLite, MySQL and PostgreSQL, there are superior database programs that are free and open-source. If you’re asking your students, many of whom may be buying Mac laptops, to get Access, you’re putting an additional burden on them. And if that’s all they know once they graduate and manage to land a job, if that place doesn’t have Access, they may need to get it (or have nothing at all).

The Access query grid hides the fact that underneath, Access runs SQL queries. So a user is able to construct and execute a SQL query without writing any SQL whatsoever. This is, imho, a bad thing, as it makes it possible to get results without actually knowing what you are doing. When we teach the query grid, we’re teaching behavior over understanding, or at the very least we’re allowing behavior to compete with understanding. And that doesn’t even begin to address the issue that the query grid doesn’t do everything that SQL can. In terms of teaching, this is critical; we’re not properly equipping students for the opportunities and challenges they could face.

Another issue is data portability: Access databases don’t support dumping to a .sql file, which is a great way to transfer SQL data without losing data types. Access does export to many formats, including Excel, CSV and XML, but the lack of SQL dump ability is a pain for transferring data. If you want to send somebody an Access database, you can either send them the entire file (providing they have Access installed), or you can export each of the tables and have them re-import them. And if you do email that .mdb (or now, .accdb) file, be warned that they do get quite big. To demonstrate this, I loaded the same three tables into Access 2007 and SQLite and the Access file was nearly 3 times the size of the SQLite database.

Finally, there’s the Web. Know many popular Web sites that run off an Access database? Me neither. If all you know is Access and not the underlying SQL concepts, your transition to a popular server software like MySQL is going to be more difficult. Sure, you say, but it’s better than nothing. But as far as the Web goes, Access is almost nothing itself. So why would you teach a program that has very little future on the Web - the platform of today and tomorrow?

The Case for SQLite

SQLite is my choice for the candidate to replace Access in journalism education. In addition to the advantages listed above, it’s also easy to “install.” If you can download files, unzip them and move them to a location on your hard drive, you can “install” SQLite. If you can install a Firefox add-on, you can manage it in the browser. And you can take your database files home with you or email them around. The add-on supports importing CSV files, SQL dumps and XML (although all databases can have issues with importing XML). It looks and works the same on a PC or a Mac. Most importantly, it demands an understanding of SQL that you can avoid when learning Access.

When I first learned SQL at an IRE bootcamp, we were using FoxPro and we learned how to type the SQL commands. That knowledge only becomes more valuable as you learn the limits and possibilities of SQL. Journalism educators and trainers should commit to teaching SQL on the broadest platform possible and with an emphasis on the syntax and meaning of the language itself, not on which buttons to click. Otherwise we risk sending students out into this new journalism world even less-prepared to handle data intelligently, and I don’t think we can afford that.

What follows is my entirely subjective, data-heavy and document-light version. It’s Congress-heavy, because the executive branch has done much, much better in many ways. No, really, show us the data:

1. Congressional committee votes. As far as I know, only commercial companies like CQ possess this information in the aggregate. Most committees publish them in committee reports (House Judiciary is one of the better ones) without a standard format and in such a way as to make their gathering prohibitively expensive. And yet these are some of the most telling public actions lawmakers make.
2. Earmarks. If you don’t think the Appropriations Committees have a database of earmarks, you’re naive. Of course they do - it’s valuable information. Now, about sharing it in anything but an image PDF format… well, let’s just say that Keith Ashdown and the folks at Taxpayers for Common Sense probably aren’t going away soon.
3. Foreign Travel Reports (Codels). The House publishes PDFs and text files of this data, but they are formatted for reading, not analysis. It would not be hard to change this.
4. Legal Defense Funds. It’s utterly ridiculous that while House members now file their campaign reports electronically, legal defense fund reports are still filed on paper. This is a no-brainer.
5. Senate Votes in XML. Go ahead, view source on this page. See where the HTML comment says “****** vote_111_1_00110.xml … “? They already generate these files; but the public can’t have them. They’re only for the use of Senators. There’s absolutely no reason the Senate cannot join the House in doing this, so why won’t they? Update: they have!
6. Senior Executive Service. This one is particularly egregious, in that the information on senior-level political appointees in the executive branch previously was made available in database-friendly formats, but now is only available via PDF. So OPM chose to make the information less useful.
7. High-Level Diplomatic Visits. Another “I can’t believe it’s not a database” entry. The State Department offers a list of visits by foreign leaders and lists of visits by the president and secretary.
8. The CIA World Factbook. Oh, you can download the PDF, but (and I am not making this up): “the search software resides on our server and cannot be distributed with the World Factbook.” Thanks!

That’s eight, and I can already think of some more. What’s on your list? Actual federal data, please, as opposed to documents that are valuable for their full-text content. I’m sure I’m missing some that should be on here.

It’s very basic, and owes much to the example put forth by James Turk’s python wrapper for the Sunlight API. At the suggestion of Sean Gillies, I’ve made the calls more in the style of Django’s ORM by using get() and filter() functions. So far, I’ve got functions for members and roll call votes, and I’ll be adding committees soon. Please feel free to fork the code or send suggestions my way.

The questions we got about Represent and our plans for it were excellent; many had good suggestions for expanding the content outside the walls of the NYT. I spent a good portion of the time extolling the virtues of GeoDjango, demonstrating that it could enable even a relative GIS newbie such as myself to help make a good geo app.

It’s a shame that we don’t have a Django-focused group in the DC area, as I’d love to attend more gatherings like we did last week. My thanks to Loren Davie, Kevin Fricovsky and the folks at HUGE, who host django-nyc meetings. Hopefully I can make some future gatherings when I’m in New York.

Today (well, technically last night) at work we launched a beta Web application called Represent that helps New York City residents keep tabs on what their elected officials are doing. It’s the product of an idea that my colleague Andrei Scheinkman suggested when we sat down in the late Spring to kick around ideas for the Times’ internal technology challenge. We ended up being a finalist with our entry, which became the app you see today, thanks to design work by Stephan Weitberg and advice from a lot of folks.

Since Andrei and I both liked Python, and the contest asked for a working prototype, we built Represent using GeoDjango, the part of Django that makes spatial work easy enough that even I can manage it. This is my first time really dealing with GIS data, and probably the hardest part was getting all the dependencies installed the first time (after you do it once, the install becomes fairly routine, and on the latest Ubuntu version it’s simple).

So if you’re thinking about going the GeoDjango route, or trying it out, I’d offer these suggestions:

• Go with Ubuntu Linux as your OS. It’s just easier. You can run GeoDjango on Windows, but I think I hate the Postgres Windows install and configure process more than just about anything I’ve tried to do on Windows.
• When installing, do what the docs say. These are written by people who have already installed it. Once you start freelancing, your chance of success goes down dramatically.
• Make sure that when you install Postgres, you test it out via the psql shell.
• When in doubt, try the #geodjango IRC channel on FreeNode
• When things just work, do not be surprised or await a message saying they worked.

Once you get GeoDjango installed - and it might take awhile the first time - you’ll be really impressed with how easy it is to use. I’ve written before about the power of LayerMapping, which is a huge gift to people who want to move from using Shapefiles to using GeoDjango. And there are other goodies, too, like the ability to plug into the GeoIP library to place your site visitors on a map (or configure their content based on that information). For Represent, we needed to produce KML files of each district that would display on the results pages. Turns out GeoDjango does that, too. To my eternal shame, I never really got the hang of working with Arc. Now I have a tool that suits me better.

Liz Donovan, one of the best news researchers ever, died of lung cancer in Gainesville, Ga. Truly a pioneer among news librarians, she was instrumental in everything from research for Woodward and Bernstein during Watergate to embracing (and leading other researchers to adopt) the use of blogs, wikis and other new technologies in newsrooms. She worked at the Washington Post and Miami Herald before retiring to the mountains of North Carolina where she continued to learn and spread her knowledge (and some beautiful photographs of the area).

Liz and Margot Williams were the first to welcome me to the News Division and to encourage me to participate in conferences, the mailing list and other activities. Liz encouraged me to write about the technological challenges and opportunities faced by newsrooms, and was generous to a fault with her colleagues seeking help. She was a constant innovator, a skilled researcher and a wonderful person. Among the people in journalism, it is rare to find someone about whom no one has a bad thing to say. Liz was one of those people. I’ll miss her friendship, advice and leadership.

If you haven’t heard of Capistrano, it and similar utilities save you the trouble of typing a series of commands to login to remote servers, update your code, bounce Apache, clean up files, etc. You put all that stuff into a single file that just runs those tasks when you need them. If you have an app that you’re regularly updating, it’s a godsend.

If you’ve used Capistrano before, Fabric is pretty similar in terms of its philosophy; there’s a fabfile in your project (like a capfile for Capistrano) that contains the configuration details and your commands for deploying code, restarting servers and the like. What I like about Fabric is that it makes virtually no assumptions about what you want to do. It doesn’t default to Subversion, or git, as a VCS. It doesn’t assume much of anything, which can be a little daunting if you’re used to a bit of code generation.

But that’s actually a good thing, because you end up writing fabfiles that only contain the stuff you need to have and nothing else. There’s a Django app that ties Fabric more tightly to Django’s manage.py utility, but I’m not sure I see the need for it. Without it, you type something like “fab deploy” instead of “python manage.py fab deploy”. But it’s there.

One thing I’d like to see from Fabric is better handling for sudo tasks on remote servers. This has been raised on the mailing list, and hopefully it gets resolved soon. Otherwise I’m really enjoying its simplicity and will use it for my Django apps going forward.

I’ve added team and player statistics for games in 2006, 2007 and 2008 (example), and will build out individual game stats back to 2001 (example). I’ve fixed several outstanding data issues, including bowl games, overtime games and at the suggestion of Chris McComas, moved conference affiliation from the College model to the CollegeYear model. I’ve added an ever-diminishing list of unbeaten teams.

There is, of course, so much more to do. I’ve got a list of feature ideas a mile long, plus an urgent need for some graphics (using the Google Chart API), team rankings and poll data. But now that the elections are over, I can spend some time working on the site. My long-term goal is to have Fumblerooski be a resource for people seeking to identify that one stat, to study a trend (either on the site or via APIs) or just to lose themselves in college football data. What would you like to see?