Federated Fine-Tuning of Mixture of Experts Vision Transformers
Abstract
Mixture-of-Experts (MoE) architectures decouple representational capacity from per-token computational cost: each input activates only a few specialised experts, yielding far greater capacity than dense models at equal activation cost. Federated Learning (FL) trains models on edge devices—aggregating local updates, never centralising raw data—preserving privacy and unlocking edge-resident data. Their integration is hard: cross-device FL faces tight bandwidth and memory budgets and local datasets are heterogeneous (non-IID); MoE sparsity, however, can overcome these barriers. Adapting the partial-replica paradigm of the FoMoE framework—where each client hosts only a subset of the experts—this thesis investigates the federated fine-tuning of a Vision MoE (V-MoE-S/32) on edge devices holding severely non-IID data, combined with Low-Rank Adaptation (LoRA) and Sparsity-aware Model Aggregation. Under a static expert assignment, restricting each client to half or fewer of the expert pool costs only 1–2 percentage points of Top-1 accuracy versus the full-replica baseline, while cutting the per-round communication payload by up to 28% and static on-device memory by up to 36%. A static assignment, however, mismatches the experts a client needs most with those initially allocated. To close this gap, this work designs and evaluates a novel server-orchestrated dynamic expert reassignment protocol that periodically migrates experts towards the clients demanding them most. Rather than raising the accuracy ceiling, it mainly stabilises the federation against sub-optimal initial placements, at a communication overhead of 1–3% of total traffic. Partial expert replication, parameter-efficient adaptation, and sparsity-aware aggregation thus compose into a practical recipe for fitting sparse vision models within edge memory, compute, and bandwidth budgets; by decoupling local hardware limits from model size, this design lets edge networks train far larger architectures.
Keywords
Not available
Specialisations
Similar theses
- On the limits of privacy in federated learning : personalization and adversarial analysis Sargam Gupta · KTH Royal Institute of Technology · 2026
- Designing for Privacy in Tabular Federated Learning Ivo Osterberg Nilsson ; Maximilian Birr Engvall · Jönköping University · 2026
- Addressing data heterogeneity, scarcity, and training efficiency in privacy-preserving federated learning Chen, Haokun · Ludwig-Maximilians-Universität München · 2025
- Studies on biomolecular structure learning and federated privacy protection Hanlin Gu · 2022
- A Comprehensive Framework for Targeted Transfer Learning: Privacy-Preserving, Secure, and Efficient Domain Adaptation Sayyed Farid Ahamed · Old Dominion University · 2026
- Multi-Scale Deep Learning Frameworks for Trajectory Prediction, Scene Understanding, and Perception in Autonomous Driving Xingnan Zhou · Concordia University · 2026
