The quality of thin thermal oxide layers are a primary concern for the yield and reliability of MOS-devices. In this paper it is demonstrated that the electrical characteristics of these layers are strongly influenced by the characteristics of the Si surface prior to the growth of the oxide layer. The effect of the Si surface cleanliness and topography on the gate oxide quality is discussed. Total Reflection X-Ray Fluorescene Analysis measurements, Scanning Tunneling and Scanning Optical Microscopy are used to trace the origin of the observed effects. It is suggested that a combination of a rough Si surface with the presence of (metallic) contamination strongly degrades the oxide breakdown properties.