Do your access rules work in both directions? Do you have any strange routing going on? Do you need to configure a static route for the returning traffic?

Sorry, knowing very little about your setup means i can only suggest vague possibilities based on networks i work with.