They aren’t dumb, peoples’ usage is just poorly informed and incorrect.

Famous/infamous are not synonyms, so you shouldn’t be using them interchangeably. Infamous specifically means “Famous for the wrong [read negative] reasons”. Like a serial killer. Or somebody who is famous for knocking over and breaking a priceless work of art.

If something is flammable, it can be set on fire. Like wood, or paper. If something is inflammable, that’s still true, but it has the additional property of being able to spontaneously combust, without being actively set alight. Like oils, or unstable chemicals, or some explosive material.

These are levels of nuance which are actually really useful, if handled correctly. The fundamental rule appears to be that in an “in…” word, the prefix gives specific detail about how the object holds the properties of the suffix.