I think a lot of people have been at a similar crossroads; to specialize in just non-rhythm synthesis or go jack of all trades groovebox. However every time I eyeball the mutant machine or jupiter storm I remember that I own an Octatrack and my rack has MIDI clock/trig in.....
Given that you have about 30-40 HP left, I guess what you have to decide is: can you fit enough rhythmic firepower in that space to be satisfied?
It depends on what you want. If you can be happy with a few Tiptop 808/909 modules and a trigger sequencer like this not yet out one or this then maybe go for it. Just know your rhythm section isn't going to be Mike Portnoy's Siamese Monster, and that is totally fine if that's what you're looking for.
If that is too much of a compromise, save up for some kind of standalone drum machine and build out your rack. You've got lots of good price points here from the Volca Beats to MFB's new stuff and seriously dangerous gear like Tempest and many of the Elektron boxes (my favorites).
These are just hands-on hardware options that seem good to me too, I'd urge you to do your own research. Remember you'd need some kind of MIDI clock/trig IN module if your box doesn't send CV/Clock signals out so your drum box can sync with your rack.
Also, I'm not sure why'd you want more envelopes either, but you're spot on with needing a clock divider for rhythm.