Of course you can, there's really no need for magic if you are a swordsman or something similar. One of my characters is a swordsman who knows 0 spells, and I can vouch that there is no need for magic because I've already completed the game with him. All the stats that you would normally put in magic/intelligence can simply be used for strength/dexterity instead. One tip I would give you is to at least learn the miracle "Anti-magic Field" (cast it by equipping a Talisman of God, it only takes 10 MP to cast and last quite a while, even the smallest amount of MP should be sufficient), this miracle is really useful against Mind Flayers (in the Tower of Latria) and Black Phantoms who spam spells.
And as you are right about how there isn't that many MP recovering items, it's not hard to get your MP back by other means. Crescent weapons, the Phosphorescent pole and the Fragrant ring all have MP regain, and they stack, as well.