When selecting materials for a building, architects are guided not only by structural and functional requirements but also consider aspects relating to the sensory effect and experience. The material information and selection tools available for architects are, however, often dominated by technical material data without providing systematic knowledge on experience aspects of materials. Hence, no comprehensive material information is available to the architect. This paper aims to identify, organize, and map the different elements contributing to the material selection process for buildings, and generate a schematic of basic material selection considerations for an architectural design project. The paper discusses the findings of three studies investigating architects’ materials selection process at different stages of the design process. As a result, architects’ materials selection considerations are identified and organized according to four categories: Context, Manufacturing, Material aspects, and Experience. These considerations are discussed in relation to each other as well as in reference to material selection research in architecture and industrial design. The results confirm the importance of aspects relating to experience in an architectural material context and emphasize the need for further research and information on the topic of material experience in architecture.