In the 14th and the 15th century, the mansion was the residence of the Counts and Princes of Celje.
Following the fall of the Counts of Celje, the building was the office of the vice-dominus (or vice-count); in the mid-18th century, it was turned into barracks upon order by the Empress Maria Theresa. Although the numerous refurbishments robbed the building of its medieval appearance, it is among the most notable castles in Slovenia. Accordingly, it is protected as a cultural monument of national importance. The basement level houses the largest in situ display of the remains of the Roman Celeia, spanning over 1,000 square metres. There is also an exhibition on the Counts of Celje in the upper floors. In addition, the castle is also home to a permanent archaeological exhibition and the Celje Gallery of Contemporary Art.