Data Package Standard | Blog

Data Package (v2) is released!

Wed, 26 Jun 2024 00:00:00 GMT

:::note Originally published on: https://frictionlessdata.io/blog/2024/06/26/datapackage-v2-release/ :::

We are very excited to announce the release of version 2.0 of the Data Package standard (previously known as Frictionless Specs). Thanks to the generous support of NLnet, starting from November last year, we were able to focus on reviewing Data Package in order to include features that were often requested throughout the years and improve extensibility for domain-specific implementations.

Data Package is a standard for data containerisation, which consists of a set of simple yet extensible specifications to describe datasets, data files and tabular data. It is a data definition language (DDL) and data API that enhances data FAIRness (findability, accessibility, interoperability, and reusability). Since its initial release in 2007, the community has suggested many features that could improve or extend the standard for use cases that weren't initially envisioned. Those were sometimes adopted, but there wasn't a versioning or governance process in place to truly evolve the standard.

We started with the issues that had accumulated in the GitHub repository to build our Roadmap for v2. Many of the requested features are now adopted, making Data Package the answer for even more use cases.

In parallel we assembled an outstanding Data Package Working Group composed of experts from the community. We carefully selected a diverse group of people who brought different use-cases, formats, and data types that we would need the Standard to support. Together with them, we crafted a governance model that is explicit, in order to create an environment that adequately supports new contributions and ensures project sustainability.

We would like to thank each one of them for their remarkable contribution and for the incredibly insightful conversations we had during these months. Thank you to my colleague Evgeny Karev, Peter Desmet from the Research Institute for Nature and Forest (INBO), Phil Schumm from CTDS - University of Chicago, Kyle Husmann from the PennState University, Keith Hughitt from the National Institutes of Health, Jakob Voß from the Verbundzentrale des GBV (VZG), Ethan Welty from the World Glacier Monitoring Service, Paul Walsh from Link Digital, Pieter Huybrechts from the Research Institute for Nature and Forest (INBO), Martin Durant from Anaconda, inc., Adam Kariv from The Public Knowledge Workshop, Johan Richer from Multi, and Stephen Diggs from the University of California Digital Library.

If you are curious about the conversations we had during the Standard review, they are all captured (and recorded) in the blog summaries of the community calls. Alternatively you can also check out the closed issues on GitHub.

So what is new in version 2?

During these months we have been working on the core specifications that compose the Standard, namely: Data Package – a simple container format for describing a coherent collection of data in a single ‘package’, Data Resource to describe and package a single data resource, Table Dialect to describe how tabular data is stored in a file, and Table Schema to declare a schema for tabular data.

During the update process we tried to be as little disruptive as possible, avoiding breaking changes when possible.

We put a lot of effort into removing ambiguity, cutting or clarifying under-defined features, and promoting some well-oiled recipes into the Standard itself. An example of a recipe (or pattern, as they were called in v1) that has been promoted to the Standard is the Missing values per field. We also added a versioning mechanism, support for categorical data, and changes that make it easier to extend the Standard.

If you would like to know the details about what has changed, see the Changelog we published.

To increase and facilitate adoption, we published a metadata mapper written in Python. We have also worked on Data Package integrations for the most notable open data portals out there. Many people from the community use Zenodo, so we definitely wanted to target that. They have recently migrated their infrastructure to Invenio RDM and we proposed a Data Package serializer for better integration with the Standard (more info on this integration will be announced in an upcoming blog!). We also created a pull request that exposes datapackage.json as a metadata export target in the Open Science Framework system, and built an extension that adds a datapackage.json endpoint to every dataset in CKAN.

If you want to know more about how to coordinate a standard update, we shared our main takeaways at FOSDEM 2024. The presentation was recorded, and you can watch it here.

And what happens now?

While the work on Data Package 2.0 is done (for now!), we will keep working on the Data Package website and documentation together with the Working Group, to make it as clear and straightforward as possible for newcomers. In parallel, we will also start integrating the version 2 changes in the software implementations).

Would you like to contribute? We always welcome new people to the project! Go and have a look at our Contribution page to understand the general guideline. Please get in touch with us by joining our community chat on Slack (also accessible via Matrix), or feel free to jump in any of the discussions on GitHub.

Funding

This project was funded through NGI0 Entrust, a fund established by NLnet with financial support from the European Commission's Next Generation Internet program. Learn more at the NLnet project page.

Original image available in the blog post.

Data Package (v2) work started

Wed, 15 Nov 2023 00:00:00 GMT

:::note Originally published on: https://frictionlessdata.io/blog/2023/11/15/frictionless-specs-update/ :::

We are very pleased to announce that thanks to the generous support of NLnet we have kickstarted the Data Package (v2) update.

After a first discussion with the community in the last call, we are setting up a working group to help us with the v2 release.

Taking into account the group's concerns about the shortness of the initial timeline we proposed, we were able to work out an extension of the initial time frame until mid-2024. We are now aiming at releasing the v2 of the Frictionless specs by June 2024.

Goal

Our overarching goal is to make the Frictionless specs, namely the Data Package, Data Resource, File Dialect, and Table Schema standards, a finished product, establishing a sustainable mechanism for future maintenance extension.

The modular approach will of course still be the cornerstone of the Frictionless specs v2, and we won’t introduce any breaking changes.

Additional deliverables

Together with the working group, we will start a reference collection of data types we want the v2 to support in a GitHub issue.
We will work with leading data publishing portals, namely CKAN and Zenodo, to provide native Data Package export integration.
The Frictionless core team at OKFN, will work on a new small Python library, a metadata mapper, with the objective of making it simpler to create standard extensions, and being able to use it in the data publishing systems integrations (Zenodo/CKAN/etc).
A new website and new documentation will be published, with improved technical language and better accessibility.

Roadmap

The project roadmap will be mainly guided by the list of issues submitted by the community throughout the years, and collected here.

Social Aspect

We will hold monthly update calls for the working group, which are of course not compulsory, and which will be open to the broader community too. In parallel we will work asynchronously with the working group, using a review model for any changes in the specifications themselves.

Decision-making

For every GitHub issue on the specs submitted by the community throughout the years, the Frictionless core team or any working group member will propose a recommended action to the working group. The action proposed will be accepted if consensus with the working group is reached, meaning we have arrived at a decision, or at least a compromise, that everyone can live with.

The working group will be invited to share their view in a devoted GitHub issue. If a broader conversation is needed, the proposal discussion can be elevated to the monthly call for deliberation. The working group will be given a reasonable amount of time to review the proposed action.

Consensus is reached and the issue is closed if at least ⅔ of the working group members participate in the discussion and express their favourable opinion. In case of serious and explicitly stated concerns, working group members who are core library investors (at the moment: OKFN, INBO, Datopian) may veto a proposed action.

The community manager at OKFN will reach out to working group members who did not participate in the discussion to make sure their opinion is also captured. We want to ensure that reminders of your participation are handled with care. You can expect a gentle and considerate approach, such as receiving an email once every two weeks highlighting any issues where your vote is pending. Our goal is to keep you informed without causing any unnecessary inconvenience, so you can feel confident and at ease with the process

Decision-making on the technical maintenance of the specs will be centralised by OKFN.

Diversity

Leveraging diversity is an opportunity that we must embrace for the benefits it brings. Extensive research consistently demonstrates that diverse participation leads to better outcomes and more resilient technologies — diverse communities are unequivocally stronger communities.

We acknowledge the need for greater diversity in our community, and we understand that fostering diversity, equity, and inclusion requires substantial effort. We cannot underestimate the challenges before us. Confronting the deep-rooted and centuries-old racist components of our system is a huge challenge, and we understand that achieving racial equity is a continual journey with no predefined endpoint.

Our mission is to build equity within our community, fostering inclusion and amplifying diversity across all aspects of the Frictionless project. For our specs update, we are proactively encouraging every member of the community involved in the working group to identify and invite candidates interested in contributing to the update work, with a particular emphasis on enhancing the diversity of our group.

We welcome contributors from diverse backgrounds, recognising that individuals with varied experiences bring forth new and innovative ideas that help create an inclusive and welcoming ecosystem. Our goal is to create an inclusive and friendly environment based on mutual respect and exchange, ensuring that no one feels isolated, and everyone is motivated to actively contribute to the project's development.

Acknowledging the ethical imperative, we understand that recruiting individuals into a community historically designed to exclude them is neither ethical nor effective without adequate support. We therefore prioritise transparency with new community members, providing a contributor's guide, a published roadmap, and comprehensive documentation to ensure clarity and support throughout their engagement. Our Code of Conduct applies to all activities linked to the Frictionless Data project, and it is enforced by our community manager. As part of this project, we will make it a priority to enhance the accessibility of our documentation and website, aiming to make them as inclusive as possible.

We welcome any other suggestions that would help us enhance diversity, equity, and inclusion in our work. Please let us know if you have a good idea to share!

Funding

This project is funded through NGI0 Entrust, a fund established by NLnet with financial support from the European Commission's Next Generation Internet program. Learn more at the NLnet project page.

Original image available in the blog post.