As one of the major cereal crops, rice (Oryza sativa L.) provides the staple food for nearly half of population worldwide, especially in developing countries. Therefore, improving rice yield is always the priority for rice breeding. Macronutrients, especially nitrogen (N) and phosphorus (P), are two most important players for the grain yield of rice. However, with economic development and improved living standard, improving nutritional quality such as micronutrient contents in grains has become a new goal in order to solve the “hidden hunger.” Micronutrients, such as iron (Fe), zinc (Zn), and selenium (Se), are critical nutritional elements for human health. Therefore, breeding the rice varieties with improved nutrient use efficiency (NUE) is thought to be one of the most feasible ways to increase both grain yield and nutritional quality with limited fertilizer input. In this review, we summarized the progresses in molecular dissection of genes for NUE by reverse genetics on macronutrients (N and P) and micronutrients (Fe, Zn, and Se), exploring natural variations for improving NUE in rice; and also, the current genetic toolbox and future perspectives for improving rice NUE are discussed.