Yeah, it's one less medium between you and the game. That would also explain the popularity boom motion controls went through with the Wii, as barebones and gimmicky as they were. The same could be said about VR, and you are right about it still being in its early stages, but it does have to potential to evolve into something even more direct than touch controls.
Yeah, we should always strive for balance, of course, but when a new peripheral is out, everyone will try to turn it into a gimmick to impress players with minimal effort (even though developing content with the use of new technology is surely challenging). That's what Wii Sports did, that's what Fruit Ninja did, that's what the most popular VR game in the future will do. They're not masterpieces by any means, but they were good at what they were programmed for: showcase new functionalities and make something fun out of it.