Recently it was found [Poedts et al., Phys. Plasmas 7, 3204 (2000)] that dusty plasma flows host nonperiodic modes-shear-dust-acoustic (SDA) vortices. These modes, interlaced with dust-acoustic (DA) waves, are able to exchange energy with the ambient flow. In this paper it is studied how these processes evolve in differentially rotating and self-gravitating flows of dusty plasmas. It is found that the presence of the self-gravity and of Coriolis forces makes both SDA vortices and DA waves transiently unstable. It is argued that the transient shear instability could be important for the formation of the fine structure of planetary rings, for the dynamics of charged dust masses and transition to dust-acoustic turbulence in galactic gaseous disks. (C) 2004 American Institute of Physics.