It is located in the south-western corner of the mediaeval city walls, at the former confluence of Sušnica and Savinja. The castle, then consisting only of a tower, is first mentioned as pledged property in a document dated 1323. Later, the Counts of Celje refurbished the castle, extended it, and made it their residence. During the reign of the Counts of Celje, the Lower Castle was considered one of the most beautiful renaissance buildings in the Slovenian territory. In the feud for the Celje heritage, the building was sacked (Jan Vitovec). Later, it was home to the office of the vice-dominus (or vice-count). The castle was well-fortified and protected by walls. However, it would see the fate of its predecessor, the Old Castle, falling to neglect. In mid-16th century it was partly restored, only to be converted into barracks in 1748–1750 upon the orders of Maria Theresa. At first, it was used as accommodation for the prisoners of the Austro-Prussian War (1757). Then, it was used as barracks until the early 1980s.