Ulrich Matter

This book is devoted to the art and science of web mining — showing how the world's largest information source can be turned into structured, research-ready data. Drawing on many years of teaching graduate courses on Web Mining and on numerous large-scale research projects in web mining contexts, the author provides clear explanations of key web technologies combined with hands-on R tutorials that work in the real world — and keep working as the web evolves. Through the book, readers will learn how to - scrape static and dynamic/JavaScript-heavy websites - use web APIs for structured data extraction from web sources - build fault-tolerant crawlers and cloud-based scraping pipelines - navigate CAPTCHAs, rate limits, and authentication hurdles - integrate AI-driven tools to speed up every stage of the workflow - apply ethical, legal, and scientific guidelines to their web mining activities Part I explains why web data matters and leads the reader through a first “hello-scrape” in R while introducing HTML, HTTP, and CSS. Part II explores how the modern web works and shows, step by step, how to move from scraping static pages to collecting data from APIs and JavaScript-driven sites. Part III focuses on scaling up: building reliable crawlers, dealing with log-ins and CAPTCHAs, using cloud resources, and adding AI helpers. Part IV looks at ethical, legal, and research standards, offering checklists and case studies, enabling the reader to make responsible choices. Together, these parts give a clear path from small experiments to large-scale projects. This valuable guide is written for a wide readership — from graduate students taking their first steps in data science to seasoned researchers and analysts in economics, social science, business, and public policy. It will be a lasting reference for anyone with an interest in extracting insight from the web — whether working in academia, industry, or the public sector.

Successfully navigating the data-driven economy presupposes a certain understanding of the technologies and methods to gain insights from Big Data. This book aims to help data science practitioners to successfully manage the transition to Big Data.Building on familiar content from applied econometrics and business analytics, this book introduces the reader to the basic concepts of Big Data Analytics. The focus of the book is on how to productively apply econometric and machine learning techniques with large, complex data sets, as well as on all the steps involved before analysing the data (data storage, data import, data preparation). The book combines conceptual and theoretical material with the practical application of the concepts using R and SQL. The reader will thus acquire the skills to analyse large data sets, both locally and in the cloud. Various code examples and tutorials, focused on empirical economic and business research, illustrate practical techniques to handle and analyse Big Data.Key Features:Includes many code examples in R and SQL, with R/SQL scripts freely provided onlineExtensive use of real datasets from empirical economic research and business analytics, with data files freely provided onlineLeads students and practitioners to think critically about where the bottlenecks are in practical data analysis tasks with large data sets, and how to address themThe book is a valuable resource for data science practitioners, graduate students and researchers who aim to gain insights from big data in the context of research questions in business, economics, and the social sciences.

Successfully navigating the data-driven economy presupposes a certain understanding of the technologies and methods to gain insights from Big Data. This book aims to help data science practitioners to successfully manage the transition to Big Data.Building on familiar content from applied econometrics and business analytics, this book introduces the reader to the basic concepts of Big Data Analytics. The focus of the book is on how to productively apply econometric and machine learning techniques with large, complex data sets, as well as on all the steps involved before analysing the data (data storage, data import, data preparation). The book combines conceptual and theoretical material with the practical application of the concepts using R and SQL. The reader will thus acquire the skills to analyse large data sets, both locally and in the cloud. Various code examples and tutorials, focused on empirical economic and business research, illustrate practical techniques to handle and analyse Big Data.Key Features:Includes many code examples in R and SQL, with R/SQL scripts freely provided onlineExtensive use of real datasets from empirical economic research and business analytics, with data files freely provided onlineLeads students and practitioners to think critically about where the bottlenecks are in practical data analysis tasks with large data sets, and how to address themThe book is a valuable resource for data science practitioners, graduate students and researchers who aim to gain insights from big data in the context of research questions in business, economics, and the social sciences.

?Die Verbreitung des Internets und die zunehmende Digitalisierung in der öffentlichen Verwaltung und Politik haben über die letzten Jahre zu einer starken Zunahme an hochdetaillierten digitalen Datenbeständen über politische Akteure und Prozesse geführt. Diese big public data werden oft über programmatische Schnittstellen (Web APIs; programmable Web) verbreitet, um die Einbettung der Daten in anderen Webanwendungen zu vereinfachen. Die Analyse dieser Daten für wissenschaftliche Zwecke in der politischen Ökonomie und Politologie ist vielversprechend, setzt jedoch die Implementierung einer data pipeline zur Beschaffung und Aufbereitung von Daten aus dem programmable Web voraus. Dieses Buch diskutiert die Chancen und Herausforderungen der praktischen Nutzung dieser Datenbestände für die empirische Forschung und zeigt anhand einer Fallstudie ein mögliches Vorgehen zur systematischen Analyse von big public data aus dem programmable Web auf.