I ran into this issue when i had "do not connect if authentication fails" enabled, which from what i've read tries to connect through TLS, which if you dont have a compatible certificate installed on both ends, gets automatically rejected.
This can also be set in group policy settings, which i believe overrides the RDP setting.