You can!

Worth noting for anyone looking to play both N64 games is that OoT Gerudo stay stunned indefinitely (until the area is reloaded) but the Pirates in Majora’s Mask who are borrowed from the Gerudo guards only stay stunned for a short time before getting back up.

This incentizes use of the Stone Mask, which can be obtained from the invisible guard by giving him a red potion. If playing the N64 version, he is located outside of Ikana Graveyard, which is a place that you can get to at that point in the game but many might not have bothered exploring yet. In the 3DS version, they moved him directly into Pirates Fortress so he’s harder to miss, but it does require just a little bit of stealth to get to where he is first.