We report on detailed low temperature scanning tunneling spectroscopy measurements performed on nanoscale Co islands on Au(111) films. At low coverages, Co islands self-organize in arrays of mono-and bilayer nanoscale structures that often have an hexagonal shape. The process of self-organization is induced by the Au(111) 'herringbone' reconstruction. By means of mapping of the local density of states with lock-in detection, electron standing wave patterns are resolved on top of the atomically flat Co islands. The surface state electrons are observed to be strongly confined laterally inside the Co nanosized islands, with their wavefunctions reflecting the symmetry of the islands. To complement the experimental work, particle-in-a-box calculations were performed. The calculations are based on a newly developed variational method that can be applied to '2D boxes' of arbitrary polygonal shape. The experimental patterns are found to fit nicely to the calculated wavefunctions for a box having a symmetry corresponding to the experimental island symmetry. The small size of the Co islands under study (down to 7.7 nm(2)) is observed to induce a strong discretization of the energy levels, with very large energy separations between the eigenstates up to several 100 meV. The observed standing wave patterns are identified either as individual eigenstates or as a 'mixture' of two or more energetically close-lying eigenstates of the cobalt island. Additionally, the Co surface state appears not to be limited to mono- and bilayer islands, but this state remains observable for multilayered islands up to five monolayers of Co.