Found Oracle Note
244527.1 which explains these wierd errors.
I am still trying to configure SSO to accept client-side certificiate authentication. I got the HTTP server to work with a wallet issued by the OCA and the Microsoft CA root added as a trusted certificate. SSO is still having issues with it though. Found these errors in the ssl_engine_log in the Apache logs:
[19/Apr/2007 10:45:23 04240] [error] SSL call to NZ function nzos_Handshake failed with error 28864 (server hw-05-0193.foo.bar.com:4443, client 10.11.13.3)[19/Apr/2007 10:45:23 04240] [error] SSL IO error [Hint: the client stop the connection unexpectedly]The 28864 error indicates a graceful exit with no error. But something is still causing the SSL IO error. . .
I think SSO is not recoginizing the client certificate's root.
Update: This error is unrelated to my problems (it's caused by something pinging the HTTP_Server and Oracle says it can be ignored). However, after following instructions found on Metalink to configure SSO to authenticate with client certificates I am getting a
"Certificate-based sign in failed. Please ensure that you have a valid certificate or contact the administrator." error.
It looks like SSO is still not recognizing the Microsoft CA trusted root I added to the wallet.
Looks like it's back to getting the Microsoft CA to issue a server certificate Oracle will play with.