We have studied the magnetic field (<50T) dependence of the photoluminescence (PL) of self-assembled InAs/GaAs quantum dots as a function of temperature (T). As the temperature is raised from 4.2 up to 80 K, thermal redistribution causes the PL to be increasingly dominated by dots with a lower PL energy. Magneto-PL demonstrates that these low-energy dots are larger in size only in the growth direction and not in the plane of the sample. At high temperatures (T>100 K), a different physical phenomenon emerges: we see an anomalous decrease of the PL shift in magnetic field, which is attributed to field enhancement of the quantum dot barrier potential. This mechanism strongly favors excitons in small dots with a weak PL shift in magnetic field, hence laterally smaller dots increasingly dominate the PL at high temperatures and high fields.