After selecting "Set Default Values", the "Cancel" and "OK" buttons in Preferences work as usual, so selecting "Cancel" returns to the current defaults, while "OK" accepts the default values.
Note that "Set Default Values" followed by "OK" removes any hidden preferences you may have set, so it should be used with caution. However, the new TeXShop Manual in TeXShop Help, added in version 5.12, has a list of all hidden preferences at the very end. The list is long, but you may be able to find a preference there that you set and then forgot.
Now the items always display as integers and take integer values.
Moreover, setting an item causes an immediate change in any open source window, so users can experiment until they find satisfactory values. There is a small caveat. TeXShop only learns of changed values when the item is deactivated, so to see the result in open source windows, change a value and then click in a different box. It is tempting to instead push RETURN, but this accepts all changes and closes the Preferences window.
After making changes in the Style fields, the "Cancel" and "OK" buttons in Preferences work as usual to reject or accept all changes. If Style changes are rejected, source windows return to their previous style.