Proximate determinants theory considers infertility rates a risk factor for lower fertility rates, but the assumption that people who perceive infertility will have fewer children has not been tested. This study investigates the association of self-perceived infertility with the number of children people have had after 11 years. Infertility implies reduced chances of conception (rather than sterility), but people do not always consistently perceive infertility over time. If people who think they are infertile at one time can later report no infertility, then does self-perceived infertility necessarily lead to having fewer children? We answer this question by analyzing 11 waves of the German family panel (pairfam) data using negative binomial growth curve models for eight core demographic subgroups created by combinations of gender (men/women), parity (0/1+children), and initial age groups (25–27 and 35–37). Those who repeatedly perceived themselves to be infertile (three times or more) had fewer children than those who perceived themselves to be infertile once or twice in only four of eight gender by initial parity by age groups. Only in four groups did people who perceived themselves to be infertile once or twice have fewer children than those who never perceived themselves to be infertile in both the unadjusted and adjusted models. Thus, self-perceived infertility does not necessarily result in fewer children. Rather, the association depends upon life course context and gender.