In a closed community in which a relying party only trusts the validation of credentials to be performed by a single identity provider, typically run by that same organization, then this is a non-issue. Other environments, such as in a federal government medium security infrastructure, a relying party may have an enumerated set of identity providers that it recognizes. In other environments, it is not clear how a relying party should choose the identity providers that it trusts.
I explored a similar set of options in a previous post, options that would support the full range of RP selectivity from promiscuous to prim.
Mark makes the point that the RP will probably make its decision to trust an IDP (or not) just once in order to establish the relationship, with that decision subsequently manifested in the IDP being placed in a white (or black) list or some other more easily assessed mechanism (when you hire a new employee, you examine their resume and references at the interview, not every time they show up for work).