The castle sits atop a ridge on a rocky overhang over Celje. The original castle was built in the early 13th century by the Vovberk (Heunburg) family, a dynasty of counts from the Carinthia region who held Celje and its surrounding as their estates until their extinction in 1322. After a decade of feud with the rival dynasties, the castle became in 1333 a property of their heirs, the Lords of Žovnek (Sanneck), later the Counts of Celje. They held the castle until the death of the last male heir of the dynasty in 1456. After that, it became the property of the landgraviate, occupied by a number of burgraves. It was sacked during the Peasants Revolt of 1515, only to be restored in the second half of the 16th century. Later, it lost its strategic importance and slowly started to decay. As early as in the 17th century, it was stripped of its rood; in the mid-18th century, the roof cladding was used to repair the Lower Castle, and the remaining usable construction material was used by the Count Anton Gaisruck to build the Novo Celje mansion. In the first half of the 19th century, the owners started using the castle as a stone quarry and it was only saved from total destruction by the Count Wickenburg who purchased the ruins and handed them to the Styrian territorial estates. All subsequent efforts for its preservation and restoration were more or less a failure.