Tell Them What You Think

TTWYT’s development blog and related musings

See here for the background — today, there are two more DRM’d PDFs from Defra, and another one from Ofgem. What’s going on?

TellThemWhatYouThink tries to get a list of documents for each consultation it finds. When it finds documents, it indexes them, so that they can also be checked when you perform a search, or when new consultations are matched up against email alerts.

These documents come in a variety of formats. One of those formats — PDF — allows its creator to set a “no-copy” bit. This prevents users from selecting and copying text from the document, and also prevents TellThemWhatYouThink’s indexing tool from converting the document into a form it can read. This is a crude form of Digital Rights Management.

I settled down this morning to figure out why all of the documents from one of Defra’s current consultations were unable to be indexed. Imagine my surprise when I discovered that all of them have the no-copy bit set. I can’t index them. You can’t copy & paste any of the text into your response as a quote. You can’t copy an excerpt to a discussion board.

This is all fantastically irritating, of course, but more to the point: what on earth is government doing copy-protecting its consultation documents?

I added comments to TellThemWhatYouThink over the weekend. As an anti-spam measure, I included a text box asking people to complete a simple sum to prove they were a person. This was for two reasons:

• It’s very quick to implement
• It’s more accessible than an image captcha — the sum will be read out by screen readers

Unfortunately, though, it doesn’t work. Be warned! Spam comments started appearing on the site within hours of the new functionality going live.

I’ve replaced that system with a reCAPTCHA test. It has an audio test for blind users — I hope it’s sufficient. We shall see how it fares!

Some splendid people in the US have released a paper recommending that Government should focus on releasing its data in reusable ways, rather than designing better user experience — their thesis being that third parties will do this for them if the data is available. Funnily enough, I find myself agreeing.

The paper is a little US-centric but the principles are sound and entirely applicable to the UK. Ed Felton has posted an excerpt if you don’t fancy reading the whole thing.

The paper hits all the right buttons: let innovative private bodies come up with the best ways to display government data and compete for each other’s audiences. Make structured data available first, and then produce a government site to display it, if required — the data should be a priority, not an afterthought. Make sure that such data feeds exist in known, permanent locations. Make government sites operate on the same data they provide to others — TellThemWhatYouThink is powered by its own API. We eat our own dogfood. I wonder how many government departments can say the same^*?

I made a lot of these points when I spoke at Tower08 earlier in the year. It’s definitely true that there are people in Government who agree, but there’re still plenty that don’t — hence the unfortunate need for campaigns like Free Our Bills.

Hopefully, reports like these will continue to be written, good examples of the reuse of public data will continue to be found, and Government will eventually see the light.

¹ Ok, that’s not quite true. I know the answer, as does all of sentient life.

I had a totally awesome time at the Barcamp over the weekend.

Matthew Somerville did two great talks — one on the new video feeds on TheyWorkForYou.com, and another on Bach Chorales. Stuart Langridge did a splendid talk on HTTP codes,  which led to quite a bit of pleasant ferreting around in TellThemWhatYouThink’s logs and code to see if things could be improved (they can). The various Yahoo people demonstrated how to do their code reviews, and allowed us to “ask them anything”, which was excellent fun. I did a talk on TellThemWhatYouThink, predictably enough, and some great ideas came out of that — thanks to all who came. I had long conversations with Rob McKinnon, of TheyWorkForYou.co.nz, who had some great ideas about how to involve people more with the site and each other.

In short, a splendid time was had all round — much thanking and hat-tipping due to Ross Brugies, who sorted the whole thing out — food, beer, swag, ideas and new people. Who could want more? I can’t wait for the next one.

TellThemWhatYouThink now supports comments. This is a very small first step towards the site supporting proper discussion.

Also among the new additions: I’ve added a tabbed interface at the bottom of each consultation page, so that more stuff can go there in the future without the pages getting really huge. Feedback welcome. At the moment, the other tabs are for incoming links — which now displays links from Bloglines, which seems to produce better results than Technorati — and “Spread the word” which has links to all the popular social bookmarking tools. I hope there’ll be more stuff there eventually, but I’m not sure what: what do you think?

Eventually, there’ll be options for keeping track of a consultation, being notified when one is published (or isn’t!) by email or rss, and widgets that you can put on your own site. Watch this space!

TellThemWhatYouThink.org now has an official twitter feed. Each government consultation discovered by the site is now posted to Twitter. Enjoy!

I’ve been working with Mark Horrell and Steph Gray at DIUS over the last couple of months to improve TellThemWhatYouThink’s support for their consultations.

It has been a splendid experience. DIUS’s consultations are now provided in an XML feed — lovely, structured ATOM goodness with custom elements galore, available to all. DIUS are the first (hopefully of many!) to provide their data in such a useful format, for which they deserve some serious kudos.

Not all has gone perfectly, though. I discovered that the library I was using to parse this feed did not understand custom ATOM elements at all, and as a result, the consultations appearing on the site were pretty garbled. I’ve fixed the problem now, but the consultations I had gathered before today were pretty messed up, so I’ve removed them from the site. The consultations which were currently live were gathered from the ATOM feed — correctly, this time — during the scrape last night and are now all present and correct. Apologies to anyone who got a duplicate email alert or who was inconvenienced by the sudden loss of content!

Thanks very much to Justin Kerr-Stevens for meeting me, to the Open Rights Group for the hook-up, and to Mark & Steph for all their work.

The MCGA have recently deployed a new version of their website. It has been significantly overhauled, and is much improved. Unfortunately, this has created some problems:

First, all links from TellThemWhatYouThink to content on the MCGA website have broken. Many government departments do not bother to ensure that their URIs remain alive after a consultation is completed. Often, they disappear quite quickly after a consultation concludes, or they are changed from something like:

www.department.gov.uk/open/someconsultation

to:

www.department.gov.uk/closed/someconsultation

This is really quite annoying. To deal with this problem, or at least, to lessen its impact, TellThemWhatYouThink checks all outgoing links to ensure that they are still alive. If they aren’t, a page is displayed with some (hopefully) useful suggestions — search for it on Google, and similar.

Unfortunately, this isn’t working with the MCGA, because they’re not returning the correct error code (404) when someone tries to access a dead link. Instead, they return a code indicating that the page has moved (302), and provide the URI of a custom error page as the new location. This is really broken: a 302 redirect should be used when content at a particular URI has moved to a new one, not when it has been removed completely.

Second, their new consultations does not exclusively contain consultations. It also contains awful, incomprehensible mess. Doubtless this is useful and meaningful to some people, but it is certainly not a consultation in the normal sense of the word: it it presumably a response. Whether or not the response itself is open to further comment, I do not know. The page doesn’t say.

Of the links on the consultation page,  only one looks to me like an actual consultation. Its structure does not bode well.

I shall review the MCGA website every so often to see if a new consultation has emerged to which the current one could be compared, and to see if any useful structure is present in the document. For now, though, I think it is broken, so I’m removing it from TellThemWhatYouThink until it can be supported again.

I had the pleasure of speaking to the assembled great and good at the Tower08 Transformational Government conference on Monday this week. I hope that video will be available at some point, and I’ll link to it if it is.

I talked, reasonably predictably, about the resusability of public data, and about why it’s important to embrace the idea that data should be made available in ways that allow people to use it, reuse it, combine it in new and clever ways and produce new, useful tools.

I also pointed out that there is an incredible amount of value to be generated from this data if it can be published in ways that allow more collaboration, and that it’ll be much cheaper in the long run if Government doesn’t try to solve all the problems. I drew a comparison between DirectGov’s fairly awful search facilities and the results produced by DirectionlessGov, which drew both heckles and laughs — an odd response. I am rather surprised to find that there actually are people out there who think that DirectGov’s search is better than Google’s. It’s a strange world we live in!

Being fairly new to the scene, I was most struck by the huge differences in people’s interpretations of what transformational government should be about. In fairness, this shouldn’t have been that surprising: everyone is interpreting it according to their vested interests, which is predictable enough.

At one end, there are people saying that everyone should own their own data, that public data is public property and should be disseminated in ways that make it as useful as possible, that massive data sharing and joined-up delivery of public services through one site is a dangerous folly.

At the other, you have people saying that we need to make identity card systems to share everyone’s data throughout government, that we should make public services usable online by having ultra-secure identification methods, that we need one place to find everything anyone might want from government, and that web 2.0, sharing and mass collaboration are merely the whimsical trends du jour.

I think it’s probably easy to tell where I stand! I’m happy to say that there is a cadre of people in government who also tend towards the former view, and that it is larger than one might think. These ideas are gaining some traction, at least, and that is quite something.