One more brewer here routinely making a lemon and ginger infused beer and no problems there. I typically have four large lemons sliced thin for 20 litres of wort. I infuse the lemons and ginger separately and add the infusion late in the boil. I always wash the lemons with hot water to get rid of anything they might spray them with.

Kefir might be different though (I don’t know anything about that process). But maybe making an infusion vs. chucking the peel in the fermentation might help.