Entorhinal grid cells, characterized by spatially periodic activity patterns, are thought to provide a universal spatial metric. However, grid cell firing-patterns are distorted in highly polarized environments such as trapezoids. Additionally, the functional role of grid cells in guiding behavior remains elusive. Here, we leverage immersive virtual reality using a novel motion platform to test the impact of environmental geometry on spatial memory in participants navigating a trapezoid arena. Object position memory in the trapezoid was degraded compared to a square control environment. Consistent with grid pattern distortions in rodents, this effect was more pronounced in the narrow than the broad part of the trapezoid. Remarkably, even outside of the encoding environment, these distortions persistently affected both navigated and judged distance estimates of never experienced paths between remembered positions and reconstructed memory maps. These distorted memory maps in turn explained behavior better than objective maps. Our findings demonstrate that environmental geometry interacts with human spatial memory similarly to how it affects rodent grid cells – thus strengthening the putative link between grid cells and behavior as well as cognitive functions beyond navigation.