For the iPhone, you will need the camera connection kit or the Lightning-to-USB converter to get a USB port. I'm using an iPad Pro with a built-in USB-C, which gives the same result. Both have the disadvantage that you're working on battery and your port is blocked so you cannot charge. For the iPad, I can use a charging USB-C hub and charge while being connected to USB devices. I don't know if such a solution is possible for the iPhone. It's important, because as you integrate your iPhone more into your workflow, you'll notice your battery drops very quickly and it's no fun needing to charge half way through your session.
Second - the best way to go is an audio interface. I tried the IK Multimedia iRig Pro Duo but I didn't like it. It's promoted as a mobile unit, but it's pretty big and only offers 2 channels (so 1 stereo, or 2 mono). They tease you with a very extensive software bundle, but forget about that. It's the hardware you want to be useful. If all you want is to play samples into your modular, it may be sufficient.
I also have a Motu M4 on my desktop, which has 4 channels, so it's a bit more flexible. I like it a lot as a basic desktop unit. But for modular, it's still limited.
In my modular, I have an ES-9. I should see if I can do more with it, but I'm using 8 channels from my modular to my iPad with a return to the ES-9 for monitoring. The ES-9 has a stereo 6,3mm dual-jack for that as well as a TRS 6,3mm stereo headphone out. In principle, it's a 16 in 16 out interface. I'm recording all my stems like that as well as the mix, in AUM. Connected via a powered hub that's chargint the iPad, it's a dream solution if you want a flexible connection between your modular and your iPad.