Yong Cheng

How is it possible to allow multiple data owners to collaboratively train and use a shared prediction model while keeping all the local training data private? Traditional machine learning approaches need to combine all data at one location, typically a data center, which may very well violate the laws on user privacy and data confidentiality. Today, many parts of the world demand that technology companies treat user data carefully according to user-privacy laws. The European Union's General Data Protection Regulation (GDPR) is a prime example. In this book, we describe how federated machine learning addresses this problem with novel solutions combining distributed machine learning, cryptography and security, and incentive mechanism design based on economic principles and game theory. We explain different types of privacy-preserving machine learning solutions and their technological backgrounds, and highlight some representative practical use cases. We show how federated learning can become the foundation of next-generation machine learning that caters to technological and societal needs for responsible AI development and application.

Gödel's true-but-unprovable sentence from the first incompleteness theorem is purely logical in nature, i.e. not mathematically natural or interesting. An interesting problem is to find mathematically natural and interesting statements that are similarly unprovable. A lot of research has since been done in this direction, most notably by Harvey Friedman. A lot of examples of concrete incompleteness with real mathematical content have been found to date. This brief contributes to Harvey Friedman's research program on concrete incompleteness for higher-order arithmetic and gives a specific example of concrete mathematical theorems which is expressible in second-order arithmetic but the minimal system in higher-order arithmetic to prove it is fourth-order arithmetic.This book first examines the following foundational question: are all theorems in classic mathematics expressible in second-order arithmetic provable in second-order arithmetic? The author gives a counterexample for this question and isolates this counterexample from the Martin-Harrington Theorem in set theory. It shows that the statement “Harrington's principle implies zero sharp" is not provable in second-order arithmetic. This book further examines what is the minimal system in higher-order arithmetic to prove the theorem “Harrington's principle implies zero sharp" and shows that it is neither provable in second-order arithmetic or third-order arithmetic, but provable in fourth-order arithmetic. The book also examines the large cardinal strength of Harrington's principle and its strengthening over second-order arithmetic and third-order arithmetic.

This book presents four approaches to jointly training bidirectional neural machine translation (NMT) models. First, in order to improve the accuracy of the attention mechanism, it proposes an agreement-based joint training approach to help the two complementary models agree on word alignment matrices for the same training data. Second, it presents a semi-supervised approach that uses an autoencoder to reconstruct monolingual corpora, so as to incorporate these corpora into neural machine translation. It then introduces a joint training algorithm for pivot-based neural machine translation, which can be used to mitigate the data scarcity problem. Lastly it describes an end-to-end bidirectional NMT model to connect the source-to-target and target-to-source translation models, allowing the interaction of parameters between these two directional models.