Potential bugs identified in XHR LC Test Suite

[Sunava Dutta]

Thanks for writing these cases by LC exit. It really makes the process of providing feedback prior to CR a lot easier. I ran these (the tests below fail on Safari3 ,Firefox 3 and IE8) with my team and had a few questions. If these issues can be addressed we can give further feedback and recommendations on the results/implementations. (Let me know if I'm wrong on our test analysis!) The rest of the tests that fail (without issues in the test itself) are being investigated further by my team so expect more over the next few days as we dig in.


http://tc.labs.opera.com/apis/XMLHttpRequest/open/033.htm
This test seems to have a bug even though it passes. Line 24 uses top.opener.rr. The framework reports FAIL although the test passes.

http://tc.labs.opera.com/apis/XMLHttpRequest/getAllResponseHeaders/001.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/getAllResponseHeaders/004.php
http://tc.labs.opera.com/apis/XMLHttpRequest/getResponseHeader/001.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/getResponseHeader/007.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/013.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/014.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/015.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/016.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/001.htm
http://tc.labs.opera.com/apis/XMLHttpRequest/readyState/002.htm
These tests seem to be failing when the test autorun is used in IE. Running the tests individually causes them to pass on IE8. Looks like a bug in the test framework.


http://tc.labs.opera.com/apis/XMLHttpRequest/getAllResponseHeaders/005.php
This test fails because the network timeout is too short and passes sometimes (Unpredictable).

http://tc.labs.opera.com/apis/XMLHttpRequest/send/005.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/send/007.htm

http://tc.labs.opera.com/apis/XMLHttpRequest/send/008.htm

This seems to be a minor test bug. The file we are receiving does not contain the string "PASS", which is necessary for the test to pass.


http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/010.htm
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/020.htm
The exception object here is not directly supported by IE, causing us to fail here. Can the test be tweaked so we can test the XHR compliance here? Thanks!

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/021.htm
The PHP page doesn't seem to be producing valid content

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/034.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/035.php
Server-side PHP seems to be causing the test to pass or fail and we can't determine the PASS criteria. May we get a pointer to the source here? (I think this was given a long time before but I can't seem to dig it up!)

Meanwhile, I'd like to re-iterate a point I had raised up awhile back. Are the tests going to be 'complete' /comprehensive at CR in relation to the spec? MSFT obviously wants this test suite to be official ensuring that third parties do not write individual test cases undermining the credibility of the suite and demonstrating increased/decreased compliance post CR (when it's much harder to make changes).

Thanks!




--
Sunava Dutta
Program Manager (AJAX) - Developer Experience Team, Internet Explorer
One Microsoft Way, Redmond WA 98052
TEL# [phone]
FAX# [phone]

[Anne van Kesteren]

On Fri, 06 Jun 2008 04:47:31 +0200, Sunava Dutta

Post by Sunava Dutta
Meanwhile, I'd like to re-iterate a point I had raised up awhile back.
Are the tests going to be 'complete' /comprehensive at CR in relation to
the spec? MSFT obviously wants this test suite to be official ensuring
that third parties do not write individual test cases undermining the
credibility of the suite and demonstrating increased/decreased
compliance post CR (when it's much harder to make changes).

The test suite will be made complete (and "official") during the CR period
as per W3C policy. (And is required to be complete before we can exit CR.)

[Kartikaya Gupta]

To add to the list:

http://tc.labs.opera.com/apis/XMLHttpRequest/open/028.php
http://tc.labs.opera.com/apis/XMLHttpRequest/statusText/001.htm
- expects exceptions to be thrown when the spec has been updated to return null/""

http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/023.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/024.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/032.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/033.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/034.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/035.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/036.php
http://tc.labs.opera.com/apis/XMLHttpRequest/setRequestHeader/037.php
- test assumes bad value will be ignored instead of throwing

I also had a question about the complex/001.htm test case. This test case seems to imply that readystatechange listeners are ordered. However, this is not specified in the XHR spec. The DOM Events spec explicitly says that listeners may be triggered in any order. It seems that all major browsers do actually keep the listeners ordered, and I've run across at least one webpage that relies on this behavior, but I don't think it's good form for a W3C test to be relying on it.

Cheers,
kats

[Bjoern Hoehrmann]

Post by Kartikaya Gupta
I also had a question about the complex/001.htm test case. This test
case seems to imply that readystatechange listeners are ordered.
However, this is not specified in the XHR spec. The DOM Events spec
explicitly says that listeners may be triggered in any order. It seems
that all major browsers do actually keep the listeners ordered, and I've
run across at least one webpage that relies on this behavior, but I
don't think it's good form for a W3C test to be relying on it.

The triggering order is undefined in DOM Level 2 Events, but defined in
DOM Level 3 Events.