The castle, located on a high hill (463 m above sea level) on the right bank of the Poprad River, north of the village of Sucha Struga (now within its administrative boundaries), was probably built at the turn of the 13th and 14th centuries; some 19th century authors suggested that the origins of the oldest part of the building, the stone tower, may date back to the 12th century; it is now dated to the 13th century.
There is no information in source documents about who built the castle. The first mention comes from a privilege granted by Władysław the Elbow-high on April 17, 1312, which allowed the Poor Clares Order from Stary Sącz to collect customs duties "under the castle Ritter" (Latin: prope castrum Ritter). The castle of Rytro is also mentioned in a 1331 document by Władysław Łokietek, who granted the burghers of Sącz a forest (Latin: silvam) ''next to the Rither castle'' (Latin: ultra Rither castrum sitam). In the 15th century the castle in Rytro was the residence of the aldermen of Sącz and was rebuilt. Since the 13th century the castle certainly belonged to the royal domain. It remained in the hands of tenants: Jakusz from Boturzyn coat of arms Czewoja, the Toporczyk family coat of arms Topór, who took the surname Ryterski; in the 16th century Piotr Kmita became the administrator. Shortly afterwards, the castle fell into ruins, as testified by Marcin Bielski's account and vetting reports from the years 1616-1617. According to local legend, it was destroyed in 1657 during the invasion of George Rakoczi.
What remains of the castle are the ruins of a tower and the remains of a wall, probably a residential building
