Building materials from the atom up is the pinnacle of materials fabrication. Until recently the only platform that offered single-atom manipulation was scanning tunneling microscopy. Here controlled manipulation and assembly of a few atom structures are demonstrated by bringing together single atoms using a scanning transmission electron microscope. An atomically focused electron beam is used to introduce Si substitutional defects and defect clusters in graphene with spatial control of a few nanometers and enable controlled motion of Si atoms. The Si substitutional defects are then further manipulated to form dimers, trimers, and more complex structures. The dynamics of a beam-induced atomic-scale chemical process is captured in a time-series of images at atomic resolution. These studies suggest that control of the e-beam-induced local processes offers the next step toward atom-by-atom nanofabrication, providing an enabling tool for the study of atomic-scale chemistry in 2D materials and fabrication of predefined structures and defects with atomic specificity.