Estonia wins 3rd prize in the European Big Data Hackathon

22 teams, each consisting of three members, from 21 countries took part in the competition. The Estonian team included Innar Liiv, an Associate Professor of Data Science at Tallinn University of Technology and a Cyber Studies Visiting Research Fellow at the University of Oxford, as a university representative, Rain Öpik, CTO at DeltaBid, as an industry representative and Toomas Kirt, a Principal Analyst of Statistics Estonia.

The task of the competition was to prototype a product within limited time that would analyse big data and other possible data sources and answer the question about how to use data and analytics to support policy makers of the European Union in reducing the mismatch between jobs and skills at regional level. In Europe, there are many countries with labour markets demanding people with certain skills, while in other countries, people with these skills are looking for work. The competition aimed to find out whether data would help to better understand the problem or whether it is possible to develop data visualisation tools that would allow a quick overview and discovery of interesting relationships and patterns.

“Our team concentrated on understanding the roots of the problem and on achieving a clearly distinguishable solution,“ said the team leader Innar Liiv. "Before we started programming, we read through a large number of documents, starting with President Juncker’s letter to Marianne Thyssen, strategic documents and budgets of the directorate dealing with employment issues to the suggestions and reports of the European Parliament committees which deal with the same subject,” Liiv explained the approach of the Estonian team.

The Estonian prototype was given the name "Megatrend and Intervention Impact Analyzer for Jobs" and the aim was to visualise the inner structure of the labour market based on the similarities of job skills and to visualise as one image both the labour market demand and supply and the affecting so-called megatrends. As a distinguishing factor, the Estonian team used as an external source the sensational article by the Oxford Martin School, published some months ago, where those jobs were named, which are at biggest risk of disappearing if the trend of automation and robotisation continues. The list of jobs in the scientific publication had to be transformed into European standards and comparatively analysed. This work resulted in images displaying together the labour market demand and supply, the mismatch and as additional information, also the impact of robotisation on the labour market. The last item enabled to also answer the question about whether robotisation targets the current gap between the labour market supply and demand or, instead, further increases unemployment.

The teams were evaluated by a panel of 20 members, consisting of policy makers from many directorates and divisions and industry representatives from Oracle, Amazon, Microsoft, IBM, SAP, SAS, Accenture, etc.

Mariana Kotzeva, the Acting Director-General of Eurostat, who presented the award pointed out that the Estonian team went beyond the expectations and tried to look to the future.

„I am delighted about the success of the Estonian team in the hackathon. This confirms the good level of Estonia as an e-country and our scientists in the field of data analytics and big data. Big data is the topic of the future and as a leading e-country, we wish to discuss this matter also at a high-level conference on the free movement of data in the single digital market on 17 July during the Estonian presidency of the Council of the European Union,” Urve Palo, the Minister of Entrepreneurship and Information Technology, commented on the success in the hackathon.

The Estonian team used Python, Amazon RDS Postgres, Hadoop cluster with Hive, Spark and other technology.