1, Don’t target X11 specifically these days. Yes a lot of people still use it or at least support it in a backward-compatible manner, but Wayland is only increasing.

2, Don’t fear the use of libraries. SDL and GTK, being C-based, should both be feasible from assembly; at most you might want to build a C program that dumps constants (if -dM doesn’t suffice) and struct offsets (if you don’t want to hard-code them).