Current theories of planetary evolution predict that infant giant planets have large radii and very low densities before they slowly contract to reach their final size after about several hundred million years(1,2). These theoretical expectations remain untested so far as the detection and characterization of very young planets is extremely challenging due to the intense stellar activity of their host stars(3,4). Only the recent discoveries of young planetary transiting systems allow initial constraints to be placed on evolutionary models(5-7). With an estimated age of 20 million years, V1298 Tau is one of the youngest solar-type stars known to host transiting planets; it harbours a system composed of four planets, two Neptune-sized, one Saturn-sized and one Jupiter-sized(3,9). Here we report a multi-instrument radial velocity campaign of V1298 Tau, which allowed us to determine the masses of two of the planets in the system. We find that the two outermost giant planets, V1298 Tau b and e (0.64 +/- 0.19 and 1.16 +/- 0.30 Jupiter masses, respectively), seem to contradict our knowledge of early-stages planetary evolution. According to models, they should reach their mass-radius combination only hundreds of millions of years after formation. This result suggests that giant planets can contract much more quickly than usually assumed.