Inside "Admin Panel -> Members -> Permissions -> Modify Group" there is the permission titled, "Change other people's karma". This should be split into two checkbox options:
"Change other people's karma (applaud)"
"Change other people's karma (smite)"
This would allow you to limit who can give positive or negative karma, instead of both in one lump category.