2019 | The Year in Review
2019 has been a massive year for the progress of Open Data and its many innovative uses; both in Public Policy and within the Corporate environment. In several countries, Open Data has taken its first steps; while in other more mature settings, Open Data continues to showcase opportunities for growth, greater efficiency and above all else, transparency and trust. The following are some of the most memorable Open Data stories of 2019.
Open Government, United Kingdom | March 2019 Edition
To kick off this year in review, we begin by looking back at our March 2019 edition. Focusing on a UK Minister by the name of Nusrat Ghani investing £4 million into a platform to provide location information about services; giving greater certainty to passengers about when public transport will arrive. This transport revolution was designed to put the power back into consumers' hands and increase productivity and help the economies in rural and regional areas to thrive. Backed by research showing the ever-increasing demand for readily available and mobile information regarding fare prices, and departure and arrival times, the Minister for Buses was also exploring the concept of mobility-as-a-service and on-demand public transport systems.
We can only hope in the new year other countries can follow in the UK's footsteps and provide readily available information regarding public transportation to the public and improve the service of the industry as a whole.
Data Giants, Global | April 2019 Edition
Over the years, Amazon executives have announced and implemented many sustainability practices. However, in 2019 more than 6000 Amazon employees agreed that despite the milestones and accolades put in place, Amazon was not committing enough to global sustainability goals and their corporate responsibility as a whole. Milestones included the enriching and enabling of research and innovation integral to both the Sustainability Initiative and Amazon's Open Data programme.
As a result, such employees signed a letter urging the company to increase Sustainability Initiatives using Open Data and furthermore, adopt six initiatives;
Consistency of public goals and timelines with science and the IPCC report,
Complete transition away from fossil fuels,
Prioritisation of climate impact when making business decisions,
Reduction of harm to the most vulnerable communities first,
Advocacy for local, federal, and international policies, and
Fair treatment of all employees during climate disruptions and extreme weather events.
This was considered a giant leap of global sustainability practise, and with increasing conversations regarding climate change and sustainability measures across the globe, let's incorporate these necessary initiatives into more organisations through the use of open data.
Data Smart Cities, India | May 2019 Edition
In 2015, Indian Prime Minister Narendra Modi launched and persisted his 100 Smart City Mission despite a lack of uniformity and standardisation. With appreciation for data growing and the requirement to involve and integrate digital technologies, the original goal evolved from developing "Smart Cities" to "Data-Smart Cities". The implementation of a rigid and repeatable system by the Ministry of Housing and Urban Affairs measuring progress and providing a roadmap for each city to follow and contribute the overall data smart goal has removed any potential ambiguity and enabled cities to achieve accreditation.
The Data Maturity Assessment Framework of 2019 played a significant role in this process. The framework was designed to tackle many challenges for all contributors, the operation aimed to drive practical data usage and to assist in structuring the approach to building collaborative data ecosystems. The Indian authorities have included several facets to their DataSmart Cities planning, including Institutional structures across all tiers of governance, Planned policies and standards, and technology platforms to implement planned policies.
Open Data and Food Security, Ghana | June 2019 Edition
According to the Data for African Development Working Group, the challenges for the substantial sharing of Open Data across Sub-Saharan Africa include funding for data collection. Also, the data collection process is often unstable and inadequate. Data accuracy is rarely checked with Donors’ priorities sometimes overtaking national ones. And finally, National statistical offices lack clear incentives to improve. These systemic failures have resulted in a reliance on data sharing organisations to lead the open data community in Africa as opposed to government agencies.
Since its creation in 2004, African technology company Esoko has passionately advocated for the free sharing of data to improve the lives of African farmers and their communities. Starting as a messaging service to help the agricultural sector and farmers share critical intel, as technology has expanded, so have their ability to share data. Throughout Africa, companies like Esoko have contributed significantly to the impact that Open Data can have on the farming communities by sharing data on market prices, weather and market intelligence.
Esoko’s work is an example of one of the many ways in which open data can help achieve the United Nations Sustainable Development Goals to fight global poverty. It is another example of where the private sector is leading the fight to access life-changing data when many governments are still inefficient in providing open datasets. Their platform illustrates the importance of mobile technology in getting Open Data into the hands of those that can benefit the most from it.
Shanghai to debut Open Data System, China | Spring 2019 Edition
At last, to finish our recap for 2019, we end with commending Shanghai for becoming one of the most open cities in the world in 2019 following the release of the city-level public open data system designed to increase digital economy and tech innovation, improve data usage and develop applications based on data processing. To further encourage the development and research of transferable technology such as artificial intelligence and cognitive services, the system provided records regarding online finance, smart transportation, health care, and culture and tourism data in its initial stage.
To emphasise the importance of data security, the Shanghai government incorporated a variety of standards and a rating system to meet the concerns of the people and protect their privacy and data safety.
Looking ahead | Resolutions
As one year draws to a close, and the year of seemingly perfect vision commences, the topic of Open Data is beginning to clear before our very eyes. In Australia, 2020 may see the launch of a world's first Consumer Data Right. In certain other countries, there appears to be a race to the top with rankings being hotly contested. Open Data continues to assist in the eradication of poverty and the enhancement of inclusiveness to some of the world's most needy. The team at Open Data Australia look ahead at the top 3 data issues looming on the horizon in 2020.
Counting back from 3 to 1....
3. The Introduction of the Consumer Data Right
In May 2018, the Australian Competition and Consumer Commission (ACCC) announced the pending adoption of the Consumer Data Right (CDR). The CDR intended to enable customers to share their transaction, usage and product data with service competitors and comparison services if they choose to do so.
According to the ACCC, “This new right will improve consumers’ ability to compare and switch between goods and services on offer. We expect the scheme to encourage competition between service providers, leading not only to better prices for customers but also more innovation of products and services.”
The element that has made the Consumer Data Right so unique is the intention of a multi-phased or sectoral adoption of the legislation over several years. Open Banking was subsequently chosen to be the first sector designated. But the path to an Open Banking launch has been slightly rocky, with the timelines and the elements of the regime changing several times.
At the latest update, the implementation timetable has been adjusted so that the obligations for the four major banks to share consumer data will now commence on 1 July 2020.
- major banks will be required to share consumer data relating to credit and debit cards, deposit accounts and transaction accounts from 1 July 2020
- major banks will be required to share consumer data relating to mortgage and personal loan accounts from 1 November 2020.
Major banks will be obliged to share certain more complex data sets including relating to joint accounts, closed accounts, direct debits and scheduled payments from 1 November 2020.
Obligations to share Product Reference Data (PRD) will remain unchanged. That is:
- major banks will be required to share PRD for credit and debit cards, deposit accounts, transaction accounts, mortgage and personal loan accounts from 1 February 2020
- non-major banks will be required to share PRD for credit and debit cards, deposit accounts and transaction accounts from 1 July 2020.
In addition to the changes in go-live dates, the roles and responsibilities of third-parties and intermediaries are under consideration. Intermediaries may participate in the collection of CDR data, or offer 'services' for the collection and processing of CDR data as a product, complementary to the accredited parties in providing goods or services to consumers. The ACCC has also recognised that the scheme may realise situations where CDR data is being accessed or processed by non-accredited CDR parties; such as financial advisers or accountants, and the potential risks that this access may introduce especially if these non-accredited parties are off-shore.
While Open Banking is the latest craze to grip the planet, with more than 16 countries either introducing or working towards adding this form of Data Portability and Digital adoption, Australia has paved the way for subsequent sectors to join the banking sector. Open Energy will be the next cab off the rank, with consultation currently underway and a launch tentatively scheduled for late 2020/2021.
Additional sectors have been unveiled as telecommunications and potentially other utilities and a long-shot of the Superannuation industry that has largely remained untouched by any new regulations or visibility for quite some time.
Regardless of the delays, there still seems to be quite a lot of work left to be done by both the participants and the regulators, with no intention of providing Public Education and Public Awareness around the new schemes echoing the mistakes of the slow United Kingdom launch. And no definitive solution to privacy concerns around non-registered participants accessing Australia's consumer data.
2. The Changing Landscape of Data Consent
The topic of Data Consent is a globally moving feast as we see new and often competing regulatory models created to protect the rights of the citizen.
In May 2018, the European Union (EU) introduced the application of the General Data Protection Regulation (GDPR), as one set of data protection rules for ALL companies that operate within the EU to observe. The stronger rules were created to allow citizens to have more control over their personal data and to reduce business benefits to that of a level playing field.
The most significant changes were around the qualification of the terms Consent, Personal Data, Processing and Control. Any business that operates in Europe, either physically, digitally, or attracts customers that reside in Europe must now conduct their data operations in line with a unified set of rulings or face significant penalties. Companies found to have breached the GDPR can be fined €20million, or 4% of their annual revenue, WHICHEVER IS HIGHER! Global giants such as British Airways (€183million fine in Sept 2018) and Marriott (£123million fine in July 2019) have already experienced fines at the hands of the regulators. The regulation further depicts the uses of data, the different consent models, and what activities can be undertaken within each, and the requirements surrounding data breaches.
Often considered to be the most robust data regulations in the world, several other countries are currently considering adopting the GDPR, an element of the GDPR or a version of the GDPR depending on the existence of complementary or opposing data laws. Amongst those countries are both the United States of America, soon adopt their own Data Regulations, the CCPA, and Australia who has recently adopted the Consumer Data Right.
For more information on the GDPR - Learn More.
The California Consumer Privacy Act (CCPA) is a State Statute as opposed to a National Law, intended to enhance the privacy rights and consumer protections of the residents of California, United States of America. Finally coming into effect from the 1st January 2020, the Senate passed Bill 1121, way back on the 13th September 2018.
The Act intends to provide California residents with the right to:
- Know what personal data is being collected about them.
- Know whether their personal data is sold or disclosed and to whom.
- Say no to the sale of personal data.
- Access their personal data.
- Request a business to delete any personal information about a consumer collected from that consumer.
- Not be discriminated against for exercising their privacy rights.
And similar to the prescriptions of the GDPR, the Act applies to any business, including any for-profit entity that does business in California, physically, digitally, or derives customers from that state. For the Act to apply, the company must make annual gross revenues in excess of $25 million, buy or sell personal information of 50,000 or more consumers/households, or earn more than half it's annual revenue from dealing in consumers personal information.
So that covering most of the Digital Platform giants, but what about the rest of the services or smaller businesses that may profit from consumer data? There are a number of questions over the lightness of the Act. Another point of contention seems to be around the levels of penalties for breaching the Act;
The following sanctions and remedies can be imposed:
For more information on the CCPA - Learn More.
- Companies, activists, associations, and others can be authorised to exercise opt-out rights on behalf of California residents.
- Companies that become victims of data theft or other data security breaches can be ordered in civil class action lawsuits to pay statutory damages between $100 to $750 per California resident and incident, or actual damages, whichever is greater, and any other relief a court deems proper, subject to an option of the California Attorney General's Office to prosecute the company instead of allowing civil suits to be brought against it.
- A fine up to $7,500 for each intentional violation and $2,500 for each unintentional violation.
- Privacy notices must be accessible and have alternative format access clearly called out.
Consumer Data Right
You would be forgiven, given the timing of its introduction, for thinking that the Consumer Data Right (CDR) was Australia's comparison to the GDPR or CCPA, but sadly you would be wrong. While CDR is being introduced to give consumer more freedom, greatest competition and data portability, it is Australia's Privacy Regulations that most closely mimic the GDPR in regards to consent.
Privacy was governed under the Privacy Act 1988, until amended via addenda in the Privacy Regulation Act 2013, and then further in the latest addition regarding security breaches reporting, the Privacy Amendment (Notifiable Data Breaches) Act 2017. Is anyone else a little confused???
While the GDPR and Australia's Privacy Laws are moving somewhat in a similar direction, some significant differences may catch some businesses out.
- The issue of Serious Harm
- The requirement for breach notification and respective timeframes
- Business Sizing
- General Reporting Requirements
Another significant separation is around the concept of consent as a topic. Terms such as Implied, Expressed, Bundled, Packaged and Individual Consent means the difference between a consumer has full understanding and clarification over the use of their personal data, and even if a secondary or derived use may be permitted. The practice of bundling consent to an opt-out model with implied consent by merely utilising a service or downloading an application may become a thing of the past. With the majority of consumers unclear as to the legalities entombed within each Term and Condition policy that they choose to browse over, it will be up to the regulators to reduce the ability for companies to profit through ignorance.
At the end of the day, it will be up to all companies to be aware of major regional data privacy legislation, including foreign subscribers and customers outside of their jurisdiction. The GDPR affects virtually everyone and is considered to be the most restrictive of any current-day regulation. Commercial operations, security and privacy professionals will need to assess each environment and ensure if they have any business or staff within the European Union. They comply with GDPR or risk significant penalty.
2020, will be a massive year on the data privacy spectrum, with more and more countries adopting stricter frameworks and regulations. I wonder if Cambridge Analytica could have happened in the year 2020?
1. The Democratisation of Data
Academically, the Democratisation of Data is the process or system of making data accessible to as many people as possible. From that access, data-driven decisions can be made from data that is tangible, able to be understood and suitable for the need of the participant. Data Democratisation requires the data to be made available in the right format and through the correct process to satisfy the needs of the user.
Sure, that sounds simple, right? The concept certainly makes sense, so why then is data democratisation so elusive? In its purest form, the need for a portal or platform to satisfy the needs of the many in unison is a large part of the barrier to creation. If you are a corporate client, versus a government agency, for instance, the format, the timeliness, even the level of detail required may vary significantly. What are the two parties going to use the data for? Would they even want to access the same raw data, or may they wish to access a processed version to achieve their separate goals?
Another issue is around the technology delivery system, or the actual platform/portal utilised. Human beings tend to favour charts, maps and tables as these formats can assist in the human-understandability portion of the data and increase the usability of the data. Still, a computer may prefer Application Programming Interface (APIs) or a less attractive code variety (to humans anyway). Computers don't need to like the data, as much as receive the data in a format that they can efficiently process and consume.
And where does this leave us with the increase in Artificial Intelligence and Machine Learning ecosystems? Could it be possible that humans gaining access to the outflows of these systems will become preferential to the absorption of the raw data? Possibly. But with the adoption of Artificial Intelligence and Machine Learning, humans are becoming less trusting. They crave transparency of logics and demand assurances that the ethical creation of the code will alleviate biases and truly represent diversity. Is there a place for the logics and ethics to be showcased in these platforms and processes?
As our needs seem to increase, so too, do the technological breakthrough, as if in lock-step. We require proof of providence in data, enter Blockchain. We desire personalisation of services and recommendation ala Netflix, enter Cognitive Search. We wish to summarise a seriously massive data dump in virtually real-time, enter Artificial Intelligence. There is a place for these technologies to augment our desires and to compliment the common-sense, governance-based, value-adds that truly democratised data requires if the needs of the many are ever going to be satisfied in a one seriously amazing co-location.
Enter, the world of Data Democratisation, which can be so much more than we ever hoped it would be!
The Expert's Predictions
As the 2020 year kicks off, the team at Open Data Australia thought they would ask a group of Data Professional from the 4 corners of the globe to share where they think the world of data is going. Some of their predictions are mind-blowing; while other predictions are downright scary! The following is just an exert of the conversation thus far....
Interview with TWO Data Rock-Stars
| Dominic Rebello & Ramesh Nagarajan |
In this segment, the team at DATAFIED ask the Data Rock-Stars the hard questions. Why they chose this field? What are the greatest challenges facing the industry? What risks do they fear the most? In this edition, we turn the microphone on and Interview not one Data Rock-Star, but two bonafide Data Champions! Dominic Rebello is the Managing Director (APAC) of the award-winning company, CluedIn and Ramesh Nagarajan is a Serial Entrepreneur and the Industry Lead, Data & Analytics at Fujitsu.
Dominic - What is your current role with Data?
I’m launching CluedIn into APAC, working as a Solution Architect / Product Manager - Working with some of the largest companies within Australia across the Financial, Insurance, Health and Travel industry to build a successful data foundation and enable them to become truly data-driven.
Ramesh - What is your current role with Data?
In my role, I provide strategy and advice on creating a coherent data creation and management strategy for both internal stakeholders and external customers. This includes working with customers on designing data catalogues and master data management for data exchange, migration of data from disparate data sources, creating a bridge between data management in the cloud and on-premise through a virtual data hub.
Dominic - Was your role with CluedIn your first job with Data?
Nope, I’ve always been involved with data and found it the most exciting part of any role, whether when I’m running a company or managing teams, having access to data I can trust to make my decisions off is better than my gut.
Ramesh - Was your role with Fujitsu your first job with Data?
No. I have been working with solutions that involve Data management and analytics for the past 20 years.
Ramesh - What do you think are the greatest challenges facing the producers or users of Open Data?
While the uses of opening up data for the benefit of consumers away from organisational silos is new, the concept of using data within the enterprise has been there for a long time. Supply chain, inventory management, sales forecasting, resource planning, demand forecasting all have been using data within the enterprise boundaries. The shift towards consumers having access to their data that benefits are something new and exciting and from that point of view, the "Open" data is very much relevant.
The shift towards open data still requires a) regulations and policies that govern what kinds of customer data can be shared, b) whether it opens up any security issues within the organisation boundaries and c) are the organisations are fully on-board to share all the data, even if customers consent, as there might be derived data that is seen as intellectual property of an organisation not to mention if customers obtain any meaningful service from sharing their data.
Dominic - What do you think are the greatest challenges facing the producers or users of Open Data? (Or is it time to remove the word Open and just talk about the sharing of data?)
Working with this problem for the past few years I think there are many challenges and not just the ‘greatest’ as you can’t share something you don’t know what it is or if you have consent. There are 9 main parts of a successful data foundation to make Open Data work. The hardest parts are; Data Quality - Data Literacy - Data Governance
There is a pandemic happening in the enterprise if you are unaware of the quality of data you are dealing with, how can they even think of sharing when you have regulations like GDPR hanging above you or the fact it takes 80% of an Analyst's time to structure. Fixing Data Quality starts with understanding what the damage is first. So, for those that are fixing data quality without knowing what benchmarks are in place, you are essentially going to be left behind or give up. Data Quality metrics is actually one of the more exciting parts of using CluedIn, the moment where you integrate data and CluedIn tells you what quality you are dealing with. Using over 19 different Data Quality metrics you will see Accuracy, Validity, Completeness, Relevance, Uniformity, Stewardship, Timeliness, Consistency, Accountability, Connectivity, Reliability, Quality, Integrity, Conformity, Flexibility, Staleness, Availability and Usability
Data literacy is imperative for any organisation desiring to derive value from data or share data. It is required across all industries, business domains and geographies, and will benefit any business process, role and decision where there is the opportunity to measure, manage and monetise data. Data literacy will impact all employees, from the board room to the break room.
If you don’t understand the data you have and what you can do with it then you are heading for problems. Nearly all meetings I walk into when I scratch the surface on Data Governance it’s not solved, they might have brought a solution claiming to help or have policies in place but nothing the policing data on entry and exit. This is due to the fact companies have too much data and don’t have it organised and structured with a policies engine on-top.
Ramesh - What is the most exciting use case of Open Data that you have seen lately?
While open data is talked about in the Banking and Finance Industry in the form of an open banking regime driven through Consumer Data Rights standards, the greatest benefit to the customer happens once the data and the analytics are contextualised to provide relevant products and services. Say, a customer is transacting with a bank and if the derived data and analytics are available for the customer to share it with another entity of their choice (say a travel agency or an energy company) then that entity can provide better services as it would be in the best interest of that entity to onboard a new customer or to retain their current customer.
Open Shareable Data increases the competition amongst the various providers to ensure that they provide the best in class products and services to the end customer. As a result, there is a shift away from monopolistic entities towards a shareable ecosystem where everyone wins.
Dominic - What is the most exciting use case of Open Data (Shared Data) that you have seen lately?
Working with one of Australia’s largest companies we ingested company data from 184 different source systems, enriched with 34+ external source systems including Government Open Data and gave a true 360 view on they’re customers but also potential customers while staying fully compliant. Internally each employee has access to company data in a Goggle-like search (Enterprise Search), with the ability to steam a data into there BI solution with Quality Metric’s all while the user can only see the data they have access to.
Ramesh - What is the number 1 mistake that you see people make with Data when working with it?
Creating too many data silos through lakes, warehouses, hubs results in ungovernable data.
Dominic - What is the number 1 mistake that you see people make with Data when working with it?
They think data 'joins' well with other data! Due to the nature and amount of data companies have collected over the years from multiple data sources they don’t hold a magic ID for merging data and data won’t merge point to point. We are in a lucky space as for many years this has been done manually and with a low level of trust so when we show a company how we merge data based on a pattern called Eventual Connectivity they are blown away.
CluedIn provides the industry’s most comprehensive, end to end Data Hub for helping large enterprise companies become data-driven. With support for integrating, preparing, governing and proliferating data throughout the business, CluedIn is the technical leader in making data ready to use and easily accessible throughout the enterprise.