The mortality impact of COVID-19 has mainly been studied at the national level. However, looking at the aggregate impact of the pandemic at the country level masks heterogeneity at the subnational level. Subnational assessments are essential for the formulation of public health policies. This is especially important for federal countries with decentralised healthcare systems, such as Germany. Therefore, we assess geographical variation in the mortality impact of COVID-19 for the 16 German federal states in 2020 and 2021 and the sex differences therein. For this purpose, we adopted an ecological study design, using population-level mortality data by federal state, age, and sex, for 2005–2021 obtained from the German Federal Statistical Office. We quantified the impact of the pandemic using the excess mortality approach. We estimated period life expectancy losses (LE losses), excess premature mortality, and excess deaths by comparing their observed with their expected values. The expected mortality was based on projected age-specific mortality rates using the Lee-Carter methodology. Saxony was the most affected region in 2020 (LE loss 0.77 years, 95% CI 0.74;0.79) while Saarland was the least affected (-0.04, -0.09;0.003). In 2021, the regions with the highest losses were Thuringia (1.58, 1.54;1.62) and Saxony (1.57, 1.53;1.6) and the lowest in Schleswig-Holstein (0.13, 0.07;0.18). Furthermore, in 2021, eastern regions experienced higher LE losses (mean: 1.13, range: 0.85 years) than western territories (mean: 0.5, range: 0.72 years). The regional variation increased between 2020 and 2021, and was higher among males than among females, particularly in 2021. We observed an unequal distribution of the mortality impact of COVID-19 at the subnational level in Germany, particularly in 2021 among the male population. The observed differences between federal states might be partially explained by the heterogeneous spread of the virus in 2020 and by differences in the population’s propensity to follow preventive guidelines.