Orson Scott Card actually weighed in on a simple way to determine if something is fantasy or sci-fi since he writes both:

The difference between science fiction and fantasy…is simply this, science fiction has rivets, fantasy has trees.