According to WHO, “As of 2010, more than half of all people live in an urban area and it is expected that by 2030, 6 out of every 10 people will live in a city, and by 2050, this proportion will increase to 7 out of 10 people”[1]. For an environment that is so heavily populated, so thoroughly engineered and so keenly observed, cities and the factors that can describe their performance remain poorly understood. Anecdotal evidence suggests that the major constraint for urban managers and researchers in driving innovative solutions to issues is a lack of data on the structure and composition of this existing urban fabric and its change over time. While such information may be available in various sources it is scattered and difficult to synthesise into an overall picture of a city. So a system relying on readily available information such as imagery would be of great benefit.   This paper presents an automated method aimed to build an Urban Inventory in a comprehensive, scalable form incorporating spatial context to monitor the trends in urban materials, land uses and vegetation. This method fuses the opportunities presented by computer vision, machine learning, pattern recognition and remote sensing to quantifying urban elements from the immensely large data base of geo-referenced imagery that is publically and privately available. The system can provide adaptation sciences with information on the urban environment for use in evaluating resilience to natural disasters, building stock management for energy efficiency and early identification of climate impacts.