FLARE: Efficient Distributed Large-Scale Graph Neural Networks Training With Adaptive Latency-Aware Probabilistic Caching

Since the emergence of Graph Neural Networks (GNNs), researchers have extensively investigated training on large-scale GNN training because of their success and wide usage in various domains including biological networks, finance, and recommendation systems. This work focuses on training large-scale distributed GNNs, where partitioning massive graphs across multiple machines creates remote communication overhead that becomes a major scalability bottleneck. We introduce a policy-driven caching mechanism that prioritizes node features and embeddings based on access frequency and cross-partition fetch cost, significantly minimizing communication overhead without sacrificing accuracy. Our policies are based on analysis of Node Affinities (NAFs) during multi-hop neighborhood sampling that extend substantially beyond the graph partition boundaries. Analyzing NAFs not only alleviates the communication bottleneck but also provides a systematic mechanism to manage in-memory data effectively, prioritizing GPU storage for node features with high fetch latency. We present FLARE, a system designed to handle partitioned feature data while leveraging the NAF-based caching policy. FLARE substantially reduces both communication overhead and training convergence time. Extensive experiments on benchmark datasets show that training FLARE on a three-layer GCN, GAT, and GraphSAGE across eight GPU machines achieves up to 12.04× (8.12× on average) speedup over DistDGLv2, demonstrating substantial performance gains compared to state-of-the-art methods.