We study the relation between the spectral gap above the ground state and the decay of the correlations in the ground state in quantum spin and fermion systems with short-range interactions on a wide class of lattices. We prove that, if two observables anticommute with each other at large distance, then the nonvanishing spectral gap implies exponential decay of the corresponding correlation. When two observables commute with each other at large distance, the connected correlation function decays exponentially under the gap assumption. If the observables behave as a vector under the U(1) rotation of a global symmetry of the system, we use previous results on the large distance decay of the correlation function to show the stronger statement that the correlation function itself, rather than just the connected correlation function, decays exponentially under the gap assumption on a lattice with a certain self-similarity in (fractal) dimensions D < 2. In particular, if the system is translationally invariant in one of the spatial directions, then this self-similarity condition is automatically satisfied. We also treat systems with long-range, power-law decaying interactions.