Sure, removing your network from the equation is definitely a more secure option; just make sure the app isn’t using those granted permissions in the background when you’re done using it and log back into your network.