We report photoluminescence measurements on stacked self-assembled InP quantum dots in magnetic fields up to 50 T. For triply stacked layers the dots become strongly coupled when the layer separation is 4 nm or less. In contrast, doubly stacked layers show no sign of coupling. We explain this puzzling difference in coupling by proposing a model in which the holes are weakly confined in the GaxIn1-xP layers separating the layers of dots, and are responsible for the coupling. Since only one such intervening layer exists in the doubly stacked dots coupling is excluded. Our model is strongly supported by the exciton masses and radii derived from our experimental results, and is consistent with available theory.