SAML Identity Broker - invalid_saml_response invalid_destination

I have a SAML identity provider configured. When I attempt to log with the provider from the keycloak login page I get the provider login page, log in, then an error in the keycloak logs after the redirect back to keycloak:

[] (default task-13) type=IDENTITY_PROVIDER_RESPONSE_ERROR, realmId=aerograph, clientId=null, userId=null, ipAddress=, error=invalid_saml_response, reason=invalid_destination

When I look at the SAML payload it has a Destination element that looks valid for an external address:

<Response xmlns="urn:oasis:names:tc:SAML:2.0:protocol"

I am running behind nginx, and set up the proxy as recommended. My proxy adds a path element /saml-sp, which I’ve configured in my standalone.xml as:

<property name="frontendUrl" value=""/>
<property name="forceBackendUrlToFrontendUrl" value="true"/>

I also followed the guidance in the docs here for setting up a proxy:

Please help, I’m not sure what I need to try next.