The speech signal carries both linguistic and non-linguistic information (e.g., a talker's voice qualities; referred to as indexical information). There is evidence that indexical information can affect some aspects of spoken word recognition, but we still do not know whether and how it can affect access to a word's meaning. A few studies support a dual-route model, in which inferences about the talker can guide access to meaning via a route external to the mental lexicon. It remains unclear whether indexical information is also encoded within the mental lexicon. The present study tests for indexical effects on spoken word recognition and referent selection within the mental lexicon. In two experiments, we manipulated voice-to-referent co-occurrence, while preventing participants from using indexical information in an explicit way. Participants learned novel words (e.g., bifa) and their meanings (e.g., kite), with each talker's voice linked (via systematic co-occurrence) to a specific referent (e.g., bifa spoken by speaker 1 referred to a specific picture of a kite). In testing, voice-to-referent mapping either matched that of training (congruent), or not (incongruent). Participants' looks to the target's referent were used as an index of lexical activation. Listeners looked faster at a target's referent on congruent than incongruent trials. The same pattern of results was observed in a third experiment, when testing was 24 hrs later. These results show that indexical information can be encoded in lexical representations and affect spoken word recognition and referent selection. Our findings are consistent with episodic and distributed views of the mental lexicon that assume multi-dimensional lexical representations.