We prove an explicit formula for the Poincaré polynomials of parabolic character varieties of Riemann surfaces with semisimple local monodromies, which was conjectured by Hausel, Letellier and Rodriguez-Villegas. Using an approach of Mozgovoy and Schiffmann the problem is reduced to counting pairs of a parabolic vector bundles and a nilpotent endomorphism of prescribed generic type. The generating function counting these pairs is shown to be a product of Macdonald polynomials and the function counting pairs without parabolic structure. The modified Macdonald polynomial $\tilde H_\lambda[X;q,t]$ is interpreted as a weighted count of points of the affine Springer fiber over the constant nilpotent matrix of type $\lambda$.