The answer is yes, you can disable the control panel. But to be thorough you need to disable the .cpl applets as well.
Start, run, gpedit.msc
A more generic approach would be to restrict the ability to install or modify any device drivers under group policy, which can be done by selecting the appropriate group membership when they are added.