The number of described cancer susceptibility syndromes continues to grow, as does our knowledge on how to manage these syndromes with the aim of early detection and cancer prevention. Oncologists now have greater responsibility to recognize patterns of cancer that warrant referral for a genetics consultation. While some patterns of common cancers are easy to recognize as related to hereditary cancer syndromes, there are a number of rare tumors that are highly associated with cancer syndromes yet are often overlooked given their infrequency. We present a review of ten rare tumors that are strongly associated with hereditary cancer predisposition syndromes: adrenocortical carcinoma, carcinoid tumors, diffuse gastric cancer, fallopian tube/primary peritoneal cancer, leiomyosarcoma, medullary thyroid cancer, paraganglioma/pheochromocytoma, renal cell carcinoma of chromophobe, hybrid oncocytoic, or oncocytoma histology, sebaceous carcinoma, and sex cord tumors with annular tubules. This review will serve as a guide for oncologists to assist in the recognition of rare tumors that warrant referral for a genetic consultation.