The dependence of the vortex penetration and expulsion on the geometry of mesoscopic superconductors is reported. Hall magnetometry measurements were performed on a superconducting Al square and triangle. The stability of the vortex patterns imposed by the sample geometry is discussed. The field-temperature H-T diagram has been reconstructed showing the transitions between states with different vorticity. We have found that the vortex penetration is only weakly affected by the vortex configuration inside the sample while the expulsion is strongly controlled by the stability of the vortex patterns. A qualitative explanation for this observation is given.