The perception of places has been linked to a network of scene-selective regions in the human brain (OPA: occipital place area; PPA: parahippocampal place area; RSC: retrosplenial complex). However, the connectivity of these regions to each other and the rest of the brain remains poorly understood. Here, we measured the functional and structural connectivity of the scene network. Functional connectivity, measured at rest and during movie watching, revealed a bias between posterior and anterior scene regions that have been implicated in perceptual versus mnemonic aspects of scene perception. For example, OPA and posterior PPA showed greater connectivity with visual and dorsal attention networks, which may play a role in the visual representation of scenes. In contrast, anterior PPA and RSC showed preferential connectivity with default-mode and frontoparietal control networks and the hippocampus, which may reflect our memory for places. We also measured the structural connectivity of the scene network using diffusion tractography. This indicated both similarities and differences with the functional connectivity, highlighting distinctions between posterior and anterior but also ventral and dorsal scene regions. These findings provide a map of the connectivity of the scene network, informing possible roles for scene-selective regions in brain function and human behaviour.