He was the second son of King Władysław II Jagiełło, and the younger brother of Władysław III of Varna. As 13 years old boy, Kazimierz was invited by Lithuanian nobles under leadership of John Gasztold to come to Lithuania and on June 29, 1440 he was installed as Grand Duke of Lithuania without Polish consent. Kazimierz succeeded his brother after a three-year interregnum on June 25, 1447. In 1454, he married Elisabeth of Austria, daughter of the late Holy Roman Emperor, Albert II of Habsburg, and his also deceased wife Elisabeth II of Bohemia. Her distant relative Frederick III of Austria, became Holy Roman Emperor, and reigned in Germany even after Kazimierz's own death. The marriage strengthened the ties between the house of Jagiełło and the sovereigns of Hungary-Bohemia, and put Kazimierz at odds with the Holy Roman Emperor, in that internal Habsburg rivalry.
Elisabeth's only brother Ladislas, king of Bohemia and Hungary, died in 1457, and after that Casimir and Elisbeth's dynastical interests directed also towards his former kingdoms.
Kazimierz IV Jagiellon's son Kazimierz was to have married the daughter of Emperor Frederick III but instead chose a religious life, eventually being canonized as St. Casimir. Kazimierz IV Jagiellon's other son, Władysław, combined the thrones of Hungary and Bohemia. His younger sons, Jan I Olbracht, Alexander Jagiellon and Zygmunt I the Old, succeeded him in turns in Poland and in the Grand Duchy of Lithuania.