In this work we generalize the tools of factor analysis for the study of multivariate stochastic processes whose second order structure evolves over time. In particular, we introduce a new class of factor models with time-varying factor loadings. The assumption that loadings are smooth enables to estimate the models using nonparametric methods. To estimate these non-stationary factor models we generalize the properties of the principal components techniques to the time-varying framework. We discuss representation issues, identification and estimation of the models and derive the asymptotic theory for the estimated loadings, the factors and the common components. We provide simulation results and applications to real data.