Hey,
I'm still pretty new to modular too. I've been looking into this myself , and I guess the two routes I see are a more modular approach vs a more clean and digital approach.
To me it feels like one of the most alluring parts of modular is the experimental aspect of it. That, and the power of analog sound. It depends what you're trying to do. If you want accurate BPM with multiple sample options, you could get something like a Pamela's Workout plus a few Pico Drums and have that be enough. I've heard people say that even the idea of a drum module is anti-modular. Learning how to synthesize the sounds from scratch has been so far the most interesting, as well as the most valuable addition to extending my knowledge in VSTs.
Since you have the space, it might be worth it to slowly build and invest in individual oscillators and noise sources and clock generators so you can make interesting, original drum sounds and patterns. But if precision and efficiency are your goals then you'll probably want more digital modules.
Hope this helps.