We present a new, high-speed technique to track the three-dimensional translation and rotation of non-spherical colloidal particles. We capture digital holograms of micrometer-scale silica rods and sub-micrometer-scale Janus particles freely diffusing in water, and then fit numerical scattering models based on the discrete dipole approximation to the measured holograms. This inverse-scattering approach allows us to extract the position and orientation of the particles as a function of time, along with static parameters including the size, shape, and refractive index. The best-fit sizes and refractive indices of both particles agree well with expected values. The technique is able to track the center of mass of the rod to a precision of 35 nm and its orientation to a precision of 1.5°, comparable to or better than the precision of other 3D diffusion measurements on non-spherical particles. Furthermore, the measured translational and rotational diffusion coefficients for the silica rods agree with hydrodynamic predictions for a spherocylinder to within 0.3%. We also show that although the Janus particles have only weak optical asymmetry, the technique can track their 2D translation and azimuthal rotation over a depth of field of several micrometers, yielding independent measurements of the effective hydrodynamic radius that agree to within 0.2%. The internal and external consistency of these measurements validate the technique. Because the discrete dipole approximation can model scattering from arbitrarily shaped particles, our technique could be used in a range of applications, including particle tracking, microrheology, and fundamental studies of colloidal self-assembly or microbial motion.