Inducing and controlling electric dipoles is hindered in the ultrathin limit by the finite screening length of surface charges at metal-insulator junctions(1-3), although this effect can be circumvented by specially designed interfaces(4). Heterostructures of insulating materials hold great promise, as confirmed by perovskite oxide superlattices with compositional substitution to artificially break the structural inversion symmetry(5-8). Bringing this concept to the ultrathin limit would substantially broaden the range of materials and functionalities that could be exploited in novel nanoscale device designs. Here, we report that non-zero electric polarization can be induced and reversed in a hysteretic manner in bilayers made of ultrathin insulators whose electric polarization cannot be switched individually. In particular, we explore the interface between ionic rock salt alkali halides such as NaCl or KBr and polar insulating Cu2N terminating bulk copper. The strong compositional asymmetry between the polar Cu2N and the vacuum gap breaks inversion symmetry in the alkali halide layer, inducing out-of-plane dipoles that are stabilized in one orientation (self-poling). The dipole orientation can be reversed by a critical electric field, producing sharp switching of the tunnel current passing through the junction.