fedora updates need more systematic testing
There is a lot of discussion ongoing about how to change Fedora updates and several proposals for an update policy float around. But after I read parts of the FESCo log for the meeting last night, where an updates policy was discussed, I came to another conclusion. Fedora needs more systematic testing. There is a proposal to require at least three positive testing comments in Bodhi (aka 3 karma points) to allow testing updates to become stable. But afaics nobody prepared some statistics about what this would have meant for the previous updates, i.e. how many updates would have been pushed in the past from testint to stable with this policy enabled.
And this leads to a big problem, there is no real knowledge available how well Fedora updates are covered by testing. Looking at the Bodhi metrics, it seems that there is not that much testing going on for testing updates in F11, because the top testers seemed not to improve recently, but the top testers in F12 have already provided more feedback. But even with these comments, there is no way to properly detect how well the testing of a package is covered, e.g. there was an update that was clearly broken that still got positive karma by people who thought using it in a dependent package, but this was not true. I do not want to blame them, because the non-existent dependency was not obvious, but this shows why systematic testing is important. This does not meant that everything has to be tested perfectly, but it should be at least known how well updates are tested to know how to improve it.
My opinion on requiring karma for updates is, that before this is done, it should be made sure that there are enough people willing to test the updates or a automated package behaviour testing should be implemented. E.g. for every package for that a certain karma amount is required, at least one dedicated tester and several occasional testers should sign up. And the required number of the testers should reflect the number of karma points required for an update. If people want better updates in stable, they should imho contribute to testing them, even if they only spend one hour every month on it.
March 10th, 2010 at 16:13 +01:00
I’ve always supported the vision that users, testers, developers and packagers build dedicated teams, who collaborate as appropriate.
That won’t happen in the short term, though. It requires quite a lot of work in several areas, such as the Fedora Updates System and package tools. And I have doubts that “becoming a tester” is attractive enough, as it’s something where people need to accept responsibility.
March 10th, 2010 at 21:40 +01:00
“There is a proposal to require at least three positive testing comments in Bodhi (aka 3 karma points) to allow testing updates to become stable.”
This is almost certainly not going to happen. It was widely agreed at the meeting that it was too high a standard. The proposal that is being worked on this week is Bill Nottingham’s, which uses the criteria currently being applied to Fedora 13: only critpath updates require a certain karma level, and they only require +1 from a ‘proven tester’ and +1 from someone else (total +2).
“But afaics nobody prepared some statistics about what this would have meant for the previous updates, i.e. how many updates would have been pushed in the past from testint to stable with this policy enabled.”
That would be a fairly pointless metric, because obviously if you require a certain level of karma before an update gets pushed, it’s likely that users will be more inclined to give the feedback and maintainers will be more inclined to go out and find people to give feedback. Past performance would not be a useful indicator of the future in this case.
March 10th, 2010 at 23:44 +01:00
“That would be a fairly pointless metric, because obviously if you require a certain level of karma before an update gets pushed, it’s likely that users will be more inclined to give the feedback and maintainers will be more inclined to go out and find people to give feedback. Past performance would not be a useful indicator of the future in this case.”
Or more and more bugs will be closed with reason rawhide and Fedora itself becomes less and less usable, because it will contain lots of unfixed bugs. It is imho not that obvious in which direction things will envolve if you add this pressure. Afaics one big problem is, that there several people who do not want to use updates-testing, but demand that other people use updates-testing to ensure for them that the updates are good. But too much demand will imho only fail in FOSS communities. If the group is big enough, why didn’t they already contribute to more testing in updates-testing, e.g. using a virtual machine?