Two-dimensional magnets have recently emerged as promising atomic-scale platforms for spintronic devices and information technologies. At the same time, significant progress has been made generating and manipulating highly-confined surface-plasmon polaritons (SPPs) in graphene-based van der Waals (vdW) heterostructures. Merging these two technologies in 2D materials offers a means of addressing local magnetic ordering using nano-light and presents novel avenues for magnetic-optoelectronic transduction. In our study, we use SPPs as a nanoscale probe of 2D anisotropy in the vdW magnet CrSBr using scattering-type scanning near-field optical microscopy (s-SNOM) to visualize SPPs in graphene/CrSBr heterostructures. We observe SPPs that possess significant temperature- and frequency-dependent anisotropic losses that mirror anisotropy in the underlying magnetic layer. This striking behavior results from a subtle interplay of magnetic ordering, dielectric properties, and interfacial charge transfer. Our results provide access to new frontiers in magnetically-coupled 2D polaritons, and highlight the versatility of confined light as a local probe of emergent 2D properties.