RCW 38 is a deeply embedded young (similar to 1 Myr), massive star cluster located at a distance of 1.7 kpc. Twice as dense as the Orion nebula cluster, orders of magnitude denser than other nearby star-forming regions and rich in massive stars, RCW 38 is an ideal place to look for potential differences in brown dwarf formation efficiency as a function of environment. We present deep, high-resolution adaptive optics data of the central similar to 0.5 x 0.5 pc(2) obtained with NACO at the Very Large Telescope. Through comparison with evolutionary models, we determine masses and extinction for similar to 480 candidate members, and derive the first initial mass function (IMF) of the cluster extending into the substellar regime. Representing the IMF as a set of power laws in the form dN/dM alpha M-alpha, we derive the slope alpha = 1.60 +/- 0.13 for the mass range 0.5-20M circle dot, which is shallower than the Salpeter slope, but in agreement with results in several other young massive clusters. At the low-mass side, we find alpha = 0.71 +/- 0.11 formasses between 0.02 and 0.5M circle dot, or alpha = 0.81 +/- 0.08 for masses between 0.02 and 1M circle dot. Our result is in agreement with the values found in other young star-forming regions, revealing no evidence that a combination of high stellar densities and the presence of numerous massive stars affects the formation efficiency of brown dwarfs and very-low-mass stars. We estimate that the Milky Way galaxy contains between 25 and 100 billion brown dwarfs (with masses > 0.03M circle dot).