In 2019 there were 490,000 children under five living with HIV. Understanding the dynamics of HIV suppression and rebound in this age group is crucial to optimizing treatment strategies and increasing the likelihood of infants achieving and sustaining viral suppression. Here we studied data from a cohort of 122 perinatally-infected infants who initiated antiretroviral treatment (ART) early after birth and were followed for up to four years. These data included longitudinal measurements of viral load (VL) and CD4 T cell numbers, together with information regarding treatment adherence. We previously showed that the dynamics of HIV decline in 53 of these infants who suppressed VL within one year were similar to those in adults. However, in extending our analysis to all 122 infants, we find that a deterministic model of HIV infection in adults cannot explain the full diversity in infant trajectories. We therefore adapt this model to include imperfect ART adherence and natural CD4 T cell decline and reconstitution processes in infants. We find that individual variation in both processes must be included to obtain the best fits. We also find that infants with faster rates of CD4 reconstitution on ART were more likely to experience resurgences in VL. Overall, our findings highlight the importance of combining mathematical modeling with clinical data to disentangle the role of natural immune processes and viral dynamics during HIV infection.