This page describes how to run conformance tests and gather testing results for Relying Party libraries.
Determining which RP Conformance Profiles to Test
The first step in using the Relying Party library tests is to decide which RP conformance profiles you plan to test.
Most RPs will test the Basic RP profile and many will also test the Implicit RP and Hybrid RP profiles.
Most RPs will test the Config RP profile, which tests your ability to use OP configuration information from .well-known/openid-configuration endpoints.
Many RPs will also test the Dynamic RP profile, which tests your ability to dynamically register your RP with OPs. As a practical matter, given that the RP testing process uses a different OP path for each test, RP testing will be much easier if Dynamic Client Registration is used by the RP. Otherwise, manual registrations will have to be performed for each test.
The tests for each profile can be viewed at https://rp.certification.openid.net:8080/. The checkboxes on the page let you choose which Response Type or profile you want to see the test list for.
For example, when testing the Basic RP profile, after selecting the code response type, the test with test ID “rp-response_type-code” and description “Can make request using response_type ‘code'” is one of the tests included.
Choosing Your Relying Party ID(s)
The RP testing tool uses a relying party ID string of your choosing as an index for keeping track of server-visible results for you. You should choose an RP ID string that contains a meaningful short name for your RP, consisting of only letters, numbers, dashes, underscores, and periods. For instance, the Apache OIDC RP module chose to use the RP ID string mod_auth_openidc.
Unless you’re only testing with the code response type, it’s probably also useful to use a response type suffix on your RP ID, so that logs for different response_types are kept separate from one another. For instance, an example RP library named “myRP” might use these different relying party IDs during testing:
myRP.codemyRP.id_tokenmyRP.id_token-tokenmyRP.code-id_tokenmyRP.code-id_token-tokenmyRP.code-token
Test IDs, Issuers, and WebFinger
Each test has a unique test ID string. For instance, test IDs for the Basic profile include “rp-response_type-code” and “rp-id_token-bad-sig-rs256”.
An issuer value that identifies the test and your RP is used to run the test. It has the form:
https://rp.certification.openid.net:8080/RP_ID/TEST_ID
For instance, the issuer used when testing the “code” response_type for the Apache OpenID Connect RP is:
https://rp.certification.openid.net:8080/mod_auth_openidc/rp-response_type-code
If your RP supports WebFinger, you can look up the issuer for a test using either of these issuer identifier syntaxes:
acct:RP_ID.TEST_ID@rp.certification.openid.net:8080
https://rp.certification.openid.net:8080/RP_ID/TEST_ID
Actual issuer identifier examples are:
acct:mod_auth_openidc.rp-response_type-code@rp.certification.openid.net:8080
https://rp.certification.openid.net:8080/mod_auth_openidc/rp-response_type-code
Obtaining Configuration Information, Registering the RP, and Running the Test
Let ISSUER represent the issuer URL for a particular RP/test pair. The OP Configuration information for that pair is located at ISSUER/.well-known/openid-configuration, for example:
To run the test, first register the RP using the OP’s configuration information. The easiest way to do this is using OpenID Connect Dynamic Client Registration. Then perform an authentication, retrieve the UserInfo Endpoint contents, etc.
Note that for each test, it’s normal to (1) do a WebFinger lookup, (2) retrieve the OP configuration, (3) perform dynamic registration, (4) send an authentication request using the chosen response_type, and (5) process the authentication response. When preparing to submit a certification result, you’ll also (6) capture an RP log file and/or screen shot documenting how your RP handled the response.
Evaluating a Test Result
Each test description displayed from https://rp.certification.openid.net:8080/ includes an “Expected result” section that describes what must happen when the test is run. For instance, the expected result for the test rp-id_token-bad-sig-rs256 (Rejects ID Token with invalid asymmetric ‘RS256’ signature) is “Identify the invalid signature and reject the ID Token after doing ID Token validation.” In this case the test passes only if the RP rejected the invalid ID Token signature returned by the test OP.
For each test, it is the responsibility of the tester to read and understand the expected result and determine whether the RP passed the test as described or not. This is because the RP test server typically doesn’t have enough information to determine whether the RP did the right thing with the information provided or not. In the rp-id_token-bad-sig-rs256 test, for instance, the tester should look for an error message produced by the RP saying that the ID Token signature was invalid. (When submitting certification results, a screen shot of the error produced would need to be included.) If the RP were to silently ignore the bad signature and proceed as if it was OK, then that would be an example of a failed result for that test.
Each test run will create a log entry at a path of the form:
https://rp.certification.openid.net:8080/log/RP_ID/TEST_ID.txt
An actual example server log entry path is:
https://rp.certification.openid.net:8080/log/mod_auth_openidc/rp-response_type-code.txt
The tester should also look at the log entry for the test and verify that no errors were identified by the test server and logged. For instance, were the RP to mistakenly not include the “scope” value in an authorization request, an error would be logged.
All test results for an RP can be viewed at a path of the form:
https://rp.certification.openid.net:8080/log/RP_ID
An actual example of such a set of test logs is:
https://rp.certification.openid.net:8080/log/mod_auth_openidc
More on Running Tests and Next Steps
Some profiles require you to run tests with multiple response_type values. The Implicit profile requires you to test both id_token and id_token+token. The Hybrid profile requires you to test three combinations: code+id_token, code+id_token+token, and code+token. The RP will need to run these tests multiple times, using each of the required response_type values, to complete testing the profile. (Basic only requires you to test response_type=code.)
The Config and Dynamic profiles also are tested using response_type=code.
Certification of a profile requires that you have Success or Warning results for all tests in the profile. You cannot certify with any Failed results for the profile.
Once you have finished testing, proceed to Submission of RP Results. Any questions can be sent to certification@oidf.org.