Massive parallelism and extreme heterogeneity are key to enabling futuristic exascale high-performance computing (HPC). Parallelism usually involves a shared-memory model with hardware-based cache-coherence mechanisms that enforce atomicity, ensuring transparent data movement and memory consistency. However, as the levels of parallelism (up to 100M cores) and heterogeneity increase, the scalability of cache coherence protocols is compromised due to (i) extensive protocol-related traffic, (ii) unique memory requirements of specialized architectures/accelerators, and (iii) the high latency (+100 clock cycles) associated with atomic operations. We propose to use hardware message queues (HMQs) which might be the key for practical massive parallelism and extreme heterogeneity. First, HMQs offer a low-latency direct path for inter-node communication that bypasses expensive cache-coherence protocols. Second and contrary to general-purpose cache-coherence systems, the same HMQ mechanisms can be used for general-purpose cores such as RISCVs or kick-start computation in specialized accelerators such as Fast Fourier Transform. In this work, we propose MOSAIC, a full-stack platform to facilitate the evaluation and design space exploration of HMQs in heterogeneous architectures. Since field programmable arrays (FPGAs) provide a cost-effective testbed for hardware exploration, we aim at an extremely lightweight, flexible architecture optimized for FPGA. However, MOSAIC could also target chiplets or SoC/ASIC.