Grain filling is a complex agronomic trait that directly determines grain weight and quality in rice (Oryza sativa). Nevertheless, key factors affecting grain filling remain poorly understood. Here, we identified a grain filling gene, OsPK3, encoding a pyruvate kinase (PK). The loss of function of OsPK3 caused reduced PK activity and Suc translocation defects from source to sink in rice, which led to compromised grain filling. OsPK3 was constitutively expressed but had relatively higher expression levels in leaf and developing caryopsis and specific expression signals in tissues involved in Suc transport and unloading, supporting its biological function in regulation of grain filling by affecting Suc translocation. Subcellular localization analysis of OsPK3 revealed its association with mitochondria, and OsPK3 physically interacted and formed heterodimers in vivo with two other PK isozymes, OsPK1 and OsPK4. Both OsPK1 and OsPK4 localized to the mitochondria and cytosol and were recruited to the mitochondria by OsPK3. Despite their high sequence similarity, OsPK1 and OsPK4 had distinct expression patterns. As observed for ospk3, disruption of OsPK1 caused pleiotropic defects, while OsPK4 loss of function led to severely chalky grains without other obvious defects. Collectively, we revealed that two mitochondria-associated pyruvate kinase complexes, OsPK3-OsPK1/OsPK4, are involved in regulation of grain filling by stage-specific fine-tuning of Suc translocation.