I didn’t think that through; what I really meant: replace ALL numbers and letters with nothing. Even if you had years to memorize everything before that change, I don’t think you’d fare well. As you seem to have pointed out in your second-to-last paragraph.

BTW the metro in Mexico City uses (Aztec, Mayan?) symbols for their routes (in addition to numbers iirc) because many people can’t read. Same with many preschool games: instead of trying to teach numbers/letters/counting prematurely, they just use colors or animals instead.

And when you think that through, letters and numbers aren’t all that different from “take a left at that big oak”.