Publishing the data is the easy half: what an open data portal needs after the files
If your organisation publishes data for the public to reuse, hosting the files is rarely what makes it hard. A spreadsheet on a server is a solved problem. The interesting questions all start after publication.
The same 3 questions come up on every portal.
- Someone needs a dataset you have not released yet. How do they ask, and how does anyone else see that the request was made?
- Someone spots an error in a column, or cannot work out how a figure was calculated. Where does that go, and does it end up attached to the data or lost in a generic contact form?
- A developer weighing up whether to build on your data has to judge, before investing any time, whether it is well maintained and licensed for their purpose.
Each of those is a real product problem, and none of them is solved by publishing more files.
Queensland's open data portal: CKAN and the extensions a government adds to it
The Queensland Government's open data portal, at data.qld.gov.au, publishes state datasets for public reuse. It is the practical expression of open data policy: anyone can find, download and reuse what a government agency holds, under a licence that permits it. Its readers are the public, journalists, researchers and businesses building products on public data, alongside the agency staff publishing it.
It runs on CKAN, the open-source data portal platform behind many national and state open data sites around the world. CKAN handles the catalogue itself. Everything a particular government needs on top of that is an extension, and Queensland maintains its extensions as public repositories anyone can read.
Alongside the portal sits the Queensland business services catalogue, the directory of services government offers to businesses. That one is Drupal(Opens in a new tab/window).
Our contribution, in the open: public CKAN repositories and a Drupal catalogue build
Between 2019 and 2021, engaged through Salsa Digital(Opens in a new tab/window), we contributed to the CKAN extensions behind the portal:
- the Queensland-specific behaviour that shapes the portal itself
- its theme
- the extensions that answer those 3 questions directly
Our changes sit alongside those of a much wider group of contributors, in repositories the Queensland Government owns and publishes itself.
That is a different working posture from a private client repository. Changes are visible, attributable and reviewed in public, by a community that did not hire you and has no particular reason to be polite about it. For a client, that posture reads as verifiable work: you can read the changes we made without taking our word for any of it.
Contributing successfully in that setting, in Python and CKAN rather than the PHP and Drupal we work in most days, is a credential in its own right.
On the business services catalogue we carried more of the load. We were the leading contributor to its Drupal build.
The 3 problems open data publishers keep hitting: requests, comments and certification
The extensions we worked on are worth understanding even if you never run CKAN: they are well-considered answers to the 3 questions above.
- Data requests turn a portal from one-way publishing into a demand signal. The public asks for datasets that have not been released, in the open, so agencies can see what people actually want rather than guessing at it.
- Dataset comments give quality feedback a home. A threaded discussion attached to the data itself means an error report or a methodology question lands where the next reader will see it.
- Third-party certification is the trust signal. Open Data Institute certificates rate a dataset's quality, licensing and sustainability against a published standard, so a developer can judge it without having to ask anyone.
Most organisations that publish data publicly run into the same 3 problems: a council, a regulator, a research body, a university. CKAN's answers to them are a useful reference even when the build in front of you is Drupal. For a client, that reads as time saved: the hard questions are already named, so a portal project starts at your own data rather than at first principles.
What the work left us with: maintained extensions and a second platform ecosystem
The portal is live, and the extensions are public, active and still maintained by the Queensland Government. For work of this shape that is the outcome that matters: the code stayed in service and other people kept building on it.
The extensions we contributed to between 2019 and 2021 are still public, active and maintained by the government that owns them.
For us it settled a question clients ask often. CKAN is Python, with its own community, its own extension model and its own deployment shape. Merged contributions in it answer "can you work outside Drupal?" concretely rather than with an assurance.
It also left us with a working understanding of open data as a domain. Those 3 problems are the first things we raise when a client tells us they are about to start publishing.
For organisations opening their data to the public
Everything on this page is one of our services in an unfamiliar setting: development, carried out in public repositories a state government owns, in Python and CKAN alongside the Drupal we work in most days.
If you run a council, a regulator, a research body or a university about to publish data for the public to reuse, this is the shape the engagement takes: the 3 questions answered before the build starts, work done in whatever ecosystem your portal already runs on, and changes anyone can read. The same engineering is available with AI-assisted delivery where it suits the work, at the same tested standard.
If you are standing up an open data portal, or running one where the datasets go out and nothing comes back, tell us what you publish and who needs it.