We measure all nonzero elements of the three-dimensional diffusion tensor D for clusters of colloidal spheres to a precision of 1% or better using digital holographic microscopy. We study both dimers and triangular trimers of spheres, for which no analytical calculations of the diffusion tensor exist. We observe anisotropic rotational and translational diffusion arising from the asymmetries of the clusters. In the case of the three-particle triangular cluster, we also detect a small but statistically significant difference in the rotational diffusion about the two in-plane axes. We attribute this difference to weak breaking of threefold rotational symmetry due to a small amount of particle polydispersity. Our experimental measurements agree well with numerical calculations and show how diffusion constants can be measured under conditions relevant to colloidal self-assembly, where theoretical and even numerical prediction is difficult.