The Hermitage mansion was built for banker Charles-Juste Bugnion between 1851 and 1853. The plans for it were developed by Swiss architect Louis Wenger. Charles-Juste Bugnion had purchased the vast Hermitage estate overlooking the city of Lausanne in 1841. The two already present houses were demolished in the spring of 1851 in order to make place for the new residence.
In 1976, the Bugnion family bequeathed the house and part of the estate to the City of Lausanne and a private foundation was created with the purpose of establishing a high-level visual arts museum. Since 1984, around 3 temporary art exhibitions are held each year inside the Hermitage.
The Hermitage residence was designed with large windows letting in plenty of light. Its south facade with views towards Lausanne and the Alps has a vast balcony.
Both sides of the building feature large loggias, combining cut stone and cast iron.
On the first floor, the "Salon bleu" is dedicated to the history of the Hermitage estate and its inhabitants. The furniture, curtains and the wallpaper are from the Napoleon III period. Until 1983, it was located on the ground floor and served as a smoking room.
A new entrance pavilion and underground exhibitions rooms were added in 2002 to plans by architectural practice Mondada Frigerio Dupraz.