How to Publish Open Data

Digital Enterprise Research Institute www.deri.ie

How to publish Open Data

Richard CyganiakOpening Up Government Data – Galway, 8 Nov 2011

Stefan.Decker@deri.orghttp://www.StefanDecker.org/

TimBL’s 5-star plan for open data

★ Make your stuff available on the Web

★★ Make it available as structured data(e.g., an Excel sheet instead of image scan of a table)

★★★ Use a non-proprietary format(e.g., a CSV file instead of an Excel sheet)

★★★★ Use linked data format(i.e., URIs to identify things, and RDF to represent data)

★★★★★ Link your data to other people’s data to provide contextSource: http://inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/

Five-shamrock scheme

1. Publish data on the web

2. Publish data in a machine-processable format

3. Use an open standard format

4. Publish under an open license

5. List your data in a data catalog

The web is where people look for it first Google can index it Less phone calls and emails (and FoI requests) to

answer

Lots of data is already there

Databases Reports Spreadsheets Maps

2. Publish data in a machine-processable

format

Allow others to do their own processing, analysis and visualisation of your data

New services, new ideas

Examples

CSO Quarterly National Household Survey http://cso.ie/qnhs/calendar_quarters_qnhs.htm

EPA enforcement files and ScraperWiki http://www.epa.ie/whatwedo/enforce/lic/info/ https://views.scraperwiki.com/run/irish-epa-visuals/

Galway and Fingal planning applications http://lab.linkeddata.deri.ie/2010/planning-apps/ Getting the data: 210 lines of code vs. 30 lines of code

Symptom: screenscraping

People use tools like ScraperWiki to get at data that isn't machine-readable https://scraperwiki.com/tags/ireland

Scraping is not the right way of doing this Expensive Brittle Strain on computing resources

Formats

Good: MS Excel, CSV, XML, JSON, Microdata Not so good: Pure websites, MS Word Bad: PDF Really bad: Only charts/maps without numbers

Good practices

Publish in multiple formats, at least one machine-readable

Publish Excel files alongside large PDF reports Publish CSV alongside database-backed web

applications

Not all formats are created equal Some formats bring many tools and applications

that people can already use

Quick tour of formats

CSV – Comma-Separated Values More open (and simpler) alternative to Excel format Can be opened in and exported from Excel, Google

Spreadsheets, Google Refine, … KML – Keyhole Markup Language

Simple format for presenting geographic data Can be opened in Google Maps

RSS – Really Simple Syndication Notifications of updates of any kind Can be opened in RSS readers and many email clients

Developer-oriented formats

XML – Extensible Markup Language W3C (World Wide Web Consortium) standard, 1997 established, reliable, ubiquitous

JSON – Javascript Object Notation IETF (Internet Engineering Task Force) standard, 2006 great for web APIs very simple; very fashionable right now

RDF – Resource Description Framework W3C standard, 2004 great for data integration steeper learning curve

Also: standard classifications

Within your data, use the same categories as everybody else

CSO http://www.cso.ie/surveysandmethodologies/

classifications_stan.htm StatCentral list of classifications

http://www.statcentral.ie/classifications.asp

Also: standard identifiers

Example: School roll numbers Department of Education publishes an Excel file with all

school roll numbers Can be used to Google the same school on other

websites, school evaluation reports etc Example: Ordnance Survey UK geo identifiers

Uses URIs (web addresses) as identifiers http://data.ordnancesurvey.co.uk/doc/7000000000037256 Great for use in RDF

Linked Open Data Cloud

Summary

Prefer open, widely used standards But: also prefer what you know best Support multiple formats for different audiences

where it makes sense Great: CSV, KML, RSS, XML, JSON

4. Publish under anopen license

Regulates what others can and cannot do with the data

For re-users, uncertainty about rights is a major concern

A good way to ensure that your organisation gets acknowledged

You need some non-discriminatory policy for giving rights to the data anyway (PSI directive)

Complex topic

Destroying a potential income stream? Content licenses vs database licenses Mixing and compatibility of licenses

Wikipedia, OpenStreetMap

Irish PSI License

Created in response to PSI Directive Available at http://psi.gov.ie/ Problems: Documents may not be used “for the

principal purpose of advertising or promoting a particular product or service” Can't be combined with Wikipedia or OpenStreetMap

Not an open license according to Open Definition http://opendefinition.org/

Open database licenses

http://opendefinition.org/licenses/

License features

You're allowed to do pretty much anything, provided you…

Attribution (“By”) – give credit ShareAlike (“SA”) – adapted data must be

published in the same way

Does Open Data have to be free?

Many would say yes A matter of terminology and definitions Either way there is nothing wrong with charging

for certain data

Data protection

Personal information is not open data Freedom of Information legislation

http://foi.gov.ie/

Summary

Stating an explicit license is important Irish PSI License: It's readily available, but not

“open enough” for some applications Open Data Commons licenses with various

constraints

5. List your data in adata catalog

So that people know it exists This is how the world learns about available data This is how you learn what they do and need

Some key information about a dataset

What data is being published? What's the license? When was the data collected? When will it be updated, if at all? How was/is this data collected? What was/is the data used for? Contact person? Where to give feedback?

How to do this in practice?

Have a simple page on your website Use an open community data catalog Set up your own catalog Use a national Irish data catalog???

Open community catalogs

The Data Hub http://thedatahub.org

Irish CKAN http://ie.ckan.net

Set up your own catalog

Requires a budget Roll your own software?

data.fingal.ie Use open source, e.g., CKAN?

data.gov.uk Berlin Open Data …

National Irish data catalog?

CSO's StatCentral? Marine Institute's ISDE? Who publishes the catalog in other countries?

UK: Cabinet Office US: White House Australia: Dept of Finance and Deregulation New Zealand: Dept of Internal Affairs

Summary

Data catalogs make it easy to find data Basic metadata, how to give feedback etc Important: How often are datasets accessed? “Request a dataset” feature Also: Open Data Ireland Google Group

http://groups.google.com/group/open-data-ireland

5. List your data in a data catalog

How to Publish Open Data

Technology

Transcript of How to Publish Open Data

FOSSASIA'16: Exploring How Designer Can Contribute to Open Source Software

제대로배워보자 제대로배워보자 공개SW 솔루션설치& 활용가이드 Open Source Software Installation & Application Guide How to Use Open Source Software 시스템SW

Reventing How America Votes Through Open Source Solutions - Gregory Miller OSCON 2010

OPEN ACCESS Publish? Or Perish? 开放存取 : 出版 ? 还是死亡 ? Group 5 第五组.

Open how-we-will-work-live-and-learn-in-the-future

[OSS Upstream Training] 3 how open stack is made

Unlocking Technology How Open Source Drives Innovation

SAXO Publish

【Reproduced】How to publish in top journals

How to make Successful Open APIs for Startups (2012)

How to Write A Good Paper and Publish in International ...library.md.chula.ac.th/guide/ThomsonReuters25Nov2015.pdf · How to Write A Good Paper and Publish in International Journals

How to Use the PowerPoint Template - Oracle · original data sets in Hadoop –Publish blended data sets back to HDFS •Leverage results in other tools –Publish data to Hadoop

Open Lesson How We Built Guide Me Right - Open Campus Tiscali

제대로배워보자CentOS... · 2018-11-12 · 제대로배워보자 공개SW 솔루션설치& 활용가이드 Open Source Software Installation & Application Guide How to Use Open

How to publish in an isi journal حنان القرشي

how to publish you article in Elsevier كيف تنشر بحثك في الزفير والدوريات الشهيرة

How To Build The Open Mesh 09

Presentation publish

How to use Open Source in Software Development

제대로배워보자Cubrid...제대로배워보자 공개SW 솔루션설치& 활용가이드 Open Source Software Installation & Application Guide How to Use Open Source Software