Neuronal activity within the central nervous system (CNS) strictly depends on homeostasis and therefore does not tolerate uncontrolled entry of blood components. It has been generally believed that under normal conditions, the endothelial blood-brain barrier (BBB) and the epithelial blood-cerebrospinal fluid barrier (BCSFB) prevent immune cell entry into the CNS. This view has recently changed when it was realized that activated T cells are able to breach the BBB and the BCSFB to perform immune surveillance of the CNS. Here we propose that the immune privilege of the CNS is established by the specific morphological architecture of its borders resembling that of a medieval castle. The BBB and the BCSFB serve as the outer walls of the castle, which can be breached by activated immune cells serving as messengers for outside dangers. Having crossed the BBB or the BCSFB they reach the castle moat, namely the cerebrospinal fluid (CSF)-drained leptomeningeal and perivascular spaces of the CNS. Next to the CNS parenchyma, the castle moat is bordered by a second wall, the glia limitans, composed of astrocytic foot processes and a parenchymal basement membrane. Inside the castle, that is the CNS parenchyma proper, the royal family of sensitive neurons resides with their servants, the glial cells. Within the CSF-drained castle moat, macrophages serve as guards collecting all the information from within the castle, which they can present to the immune-surveying T cells. If in their communication with the castle moat macrophages, T cells recognize their specific antigen and see that the royal family is in danger, they will become activated and by opening doors in the outer wall of the castle allow the entry of additional immune cells into the castle moat. From there, immune cells may breach the inner castle wall with the aim to defend the castle inhabitants by eliminating the invading enemy. If the immune response by unknown mechanisms turns against self, that is the castle inhabitants, this may allow for continuous entry of immune cells into the castle and lead to the death of the castle inhabitants, and finally members of the royal family, the neurons. This review will summarize the molecular traffic signals known to allow immune cells to breach the outer and inner walls of the CNS castle moat and will highlight the importance of the CSF-drained castle moat in maintaining immune surveillance and in mounting immune responses in the CNS.