We study the resistivity scaling in nanometer-sized metallic wires due to surface roughness and grain-boundaries, currently the main cause of electron scattering in nanoscaled interconnects. The resistivity has been obtained with the Boltzmann transport equation, adopting the relaxation time approximation of the distribution function and the effective mass approximation for the conducting electrons. The relaxation times are calculated exactly, using Fermi's golden rule, resulting in a correct relaxation time for every sub-band state contributing to the transport. In general, the relaxation time strongly depends on the sub-band state, something that remained unclear with the methods of previous work. The resistivity scaling is obtained for different roughness and grain-boundary properties, showing large differences in scaling behavior and relaxation times. Our model clearly indicates that the resistivity is dominated by grain-boundary scattering, easily surpassing the surface roughness contribution by a factor of 10.