The human mind is endowed with innate primordial perceptions such as space, distance, motion, change, flow of time, matter. The field of cognitive science argues that the abstract concepts of mathematics are not Platonic, but are built in the brain from these primordial perceptions, using what are known as conceptual metaphors. Known cognitive mechanisms give rise to the extremely precise and logical language of mathematics. Thus all of the vastness of mathematics, with its beautiful theorems, is human mathematics. It resides in the mind, and is not ‘out there’. Physics is an experimental science in which results of experiments are described in terms of concrete concepts—these concepts are also built from our primordial perceptions. The goal of theoretical physics is to describe the experimentally observed regularity of the physical world in an unambiguous, precise and logical manner. To do so, the brain resorts to the well-defined abstract concepts which the mind has metaphored from our primordial perceptions. Since both the concrete and the abstract are derived from the primordial, the connection between physics and mathematics is not mysterious, but natural. This connection is established in the human brain, where a small subset of the vast human mathematics is cognitively fitted to describe the regularity of the universe. Theoretical physics should be thought of as a branch of mathematics, whose axioms are motivated by observations of the physical world. We use the example of quantum theory to demonstrate the all too human nature of the physics-mathematics connection: it is at times frail, and imperfect. Our resistance to take this imperfection sufficiently seriously (since no known experiment violates quantum theory) shows the fundamental importance of experiments in physics. This is unlike in mathematics, the goal there being to search for logical and elegant relations amongst abstract concepts which the mind creates.