Three-dimensionally extended dusty plasmas containing mixtures of two particle species of different size have been investigated on parabolic flights. To distinguish the species even at small size disparities, one of the species is marked with a fluorescent dye, and a two-camera video microscopy setup is used for position determination and tracking. Phase separation is found even when the size disparity is below 5%. Particles are tracked to obtain the diffusion flux, and resulting diffusion coefficients are in the expected range for a phase separation process driven by plasma forces. Additionally, a measure for the strength of the phase separation is presented that allows to quickly characterize measurements. There is a clear correlation between size disparity and phase separation strength. Molecular dynamics simulations of binary dusty plasmas have been performed and their behavior with respect to the phase separation process has been analyzed. Here as well, it is found that even the smallest size disparities lead to phase separation. The separation is due to the force imbalance on the two species and the separation becomes weaker with increasing mean particle size. In the second part of the thesis, Experiments on self-excited dust-density waves under various magnetic fields have been performed. For that purpose, different dust clouds of micrometer-sized dust particles were trapped in the sheath of a radio frequency discharge. The self-excited dust-density waves were studied ...