You have to apply the changes more than once to actually see them take effect. When you pick when the failure will happen, you have to press the confirm button there. Then you will return to the full list of failure categories, where you have to also use the "Apply changes" button to see the failure take effect.
If that doesn't work for whatever specific failure you're trying to cause, file a bug report here with the exact steps you take to see the issue.