In this paper, we explore the possibility of building a quantum memory that is robust to thermal noise using large N matrix quantum mechanics models. First, we investigate the gauged SU(N) matrix harmonic oscillator and different ways to encode quantum information in it. By calculating the mutual information between the system and a reference which purifies the encoded information, we identify a transition temperature, Tc, below which the encoded quantum information is protected from thermal noise for a memory time scaling as N^2. Conversely, for temperatures higher than T_c, the information is quickly destroyed by thermal noise. Second, we relax the requirement of gauge invariance and study a matrix harmonic oscillator model with only global symmetry. Finally, we further relax even the symmetry requirement and propose a model that consists of a large number N^2 of qubits, with interactions derived from an approximate SU(N) symmetry. In both ungauged models, we find that the effects of gauging can be mimicked using an energy penalty to give a similar result for the memory time. The final qubit model also has the potential to be realized in the laboratory.