fedora updates need more systematic testing
There is a lot of discussion ongoing about how to change Fedora updates and several proposals for an update policy float around. But after I read parts of the FESCo log for the meeting last night, where an updates policy was discussed, I came to another conclusion. Fedora needs more systematic testing. There is a proposal to require at least three positive testing comments in Bodhi (aka 3 karma points) to allow testing updates to become stable. But afaics nobody prepared some statistics about what this would have meant for the previous updates, i.e. how many updates would have been pushed in the past from testint to stable with this policy enabled.
And this leads to a big problem, there is no real knowledge available how well Fedora updates are covered by testing. Looking at the Bodhi metrics, it seems that there is not that much testing going on for testing updates in F11, because the top testers seemed not to improve recently, but the top testers in F12 have already provided more feedback. But even with these comments, there is no way to properly detect how well the testing of a package is covered, e.g. there was an update that was clearly broken that still got positive karma by people who thought using it in a dependent package, but this was not true. I do not want to blame them, because the non-existent dependency was not obvious, but this shows why systematic testing is important. This does not meant that everything has to be tested perfectly, but it should be at least known how well updates are tested to know how to improve it.
My opinion on requiring karma for updates is, that before this is done, it should be made sure that there are enough people willing to test the updates or a automated package behaviour testing should be implemented. E.g. for every package for that a certain karma amount is required, at least one dedicated tester and several occasional testers should sign up. And the required number of the testers should reflect the number of karma points required for an update. If people want better updates in stable, they should imho contribute to testing them, even if they only spend one hour every month on it.