We theoretically explore Josephson diode effect (JDE) in superconductor/quantum dot (QD)/superconductor junction in the presence of a magnetic field and Rashba spin-orbit interaction (RSOI). We calculate the Josephson current in our QD junction using Keldysh non-equilibrium Green's function technique. We show that JDE is induced in our chiral QD junction with large rectification coefficient (RC) in the presence of RSOI and external magnetic field simultaneously. Interestingly, the sign and magnitude of the RC are highly controllable by the magnetic field and RSOI. For realistic RSOI strength in the presence of magnetic field and chirality, the RC can be tuned to be as high as 70% by an external gate potential, indicating a giant JDE in our QD junction. Our proposed QD-based Josephson diode (JD) may serve as a potential superconducting device component.