Recently I was switching Office 365 tenant from ADFS to Azure AD Pass-through authentication (PTA). It all went smoothly with one exception. After removing one of the Azure AD Connect servers and all applications from its Azure AD Connect interface still is showing said agent, just inactive.
How do you remove it? You don't. After a few days, Microsoft will drop it automatically. While the solution may be short, it took me some time to find this information.