Diamonds containing nitrogen in different forms have been irradiated by 3-MeV electrons or Co-60 gamma photons and characterized by optical absorption (OA) and electron spin resonance (ESR). An unusually low production rate of vacancies (V) and interstitials (I) was observed in gamma-irradiated Ib-type diamonds (those containing isolated nitrogen) and pure IIa-type diamonds as compared to Ia diamonds (containing nitrogen clusters). Postirradiation annealing at temperatures above 300 degrees C strongly increased the V and I concentrations in Ib diamond, but not in IIa diamond. These results are explained as gamma irradiation of diamond predominantly produces V-I complexes instead of individual V and I defects. Strong effect of charge state on V-I recombination is revealed: In Ib diamond, V-I complexes are negatively charged and dissociate upon annealing. On the contrary, V-I pairs are neutral in IIa diamond, and they annihilate during irradiation. The OA, ESR, and positron annihilation signatures of the V--I pairs are identified.