Graph Neural Networks (GNNs) are the first choice methods for graph machine learning problems thanks to their ability to learn state-of-the-art level representations from graph-structured data. However, centralizing a massive amount of real-world graph data for GNN training is prohibitive due to user-side privacy concerns, regulation restrictions, and commercial competition. Federated Learning is the de-facto standard for collaborative training of machine learning models over many distributed edge devices without the need for centralization. Nevertheless, training graph neural networks in a federated setting is vaguely defined and brings statistical and systems challenges. This work proposes SpreadGNN, a novel multi-task federated training framework capable of operating in the presence of partial labels and absence of a central server for the first time in the literature. SpreadGNN extends federated multi-task learning to realistic serverless settings for GNNs, and utilizes a novel optimization algorithm with a convergence guarantee, Decentralized Periodic Averaging SGD (DPA-SGD), to solve decentralized multi-task learning problems. We empirically demonstrate the efficacy of our framework on a variety of non-I.I.D. distributed graph-level molecular property prediction datasets with partial labels. Our results show that SpreadGNN outperforms GNN models trained over a central server-dependent federated learning system, even in constrained topologies. The source code is publicly available at this https URL

Graph Neural Network (GNN) research is rapidly growing thanks to the capacity of GNNs to learn representations from graph-structured data. However, centralizing a massive amount of real-world graph data for GNN training is prohibitive due to user-side privacy concerns, regulation restrictions, and commercial competition. Federated learning (FL), a trending distributed learning paradigm, aims to solve this challenge while preserving privacy. Despite recent advances in vision and language domains, there is no suitable platform for the federated training of GNNs. To this end, we introduce FedGraphNN, an open research federated learning system and the benchmark to facilitate GNN-based FL research. FedGraphNN is built on a unified formulation of federated GNNs and supports commonly used datasets, GNN models, FL algorithms, and flexible APIs. We also contribute a new molecular dataset, hERG, to promote research exploration. Our experimental results present significant challenges from federated GNN training: federated GNNs perform worse in most datasets with a non-I.I.D split than centralized GNNs; the GNN model that performs the best in centralized training may not hold its advantage in the federated setting. These results imply that more research effort is needed to unravel the mystery of federated GNN training. Moreover, our system performance analysis demonstrates that the FedGraphNN system is affordable to most research labs with a few GPUs. FedGraphNN will be regularly updated and welcomes inputs from the community.

We study spatiotemporal forecasting of high-dimensional rectangular grid graph structured data, which exhibits both complex spatial and temporal dependencies. In most high-dimensional spatiotemporal forecasting scenarios, deep learningbased methods are widely used. However, deep learning algorithms are overconfident in their predictions, and this overconfidence causes problems in the human-in-the-loop domains such as medical diagnosis and many applications of th generation wireless networks. We propose spatiotemporal extensions to variational autoencoders for regularization, robustness against out-of data distribution, and incorporating uncertainty in predictions to resolve overconfident predictions. However, variational inference methods are prone to biased posterior approximations due to using explicit exponential family densities and mean-field assumption in their posterior factorizations. To mitigate these problems, we utilize variational inference & learning with semi-implicit distributions and apply this inference scheme into convolutional long-short term memory networks(ConvLSTM) for the first time in the literature. In chapter 3, we propose variational autoencoders with convolutional long-short term memory networks, called VarConvLSTM. In chapter 4, we improve our algorithm via semi-implicit & doubly semi-implicit variational inference to model multi-modalities in the data distribution . In chapter 5, we demonstrate that proposed algorithms are applicable for spatiotemporal forecasting tasks, including space-time mobile traffic forecasting over Turkcell base station networks.