Phew, to run such a model - even just 27B quantized - at a reasonable speed requires some serious GPU Power.

Best thing I have is a 7800 XT 16gb (so ROCm it would be), don’t think that’s realistic and I’d rather invest the necessary amount of money for new GPUs into sth. more useful like a small electric vehicle than a slop cannon 🥴. Happy to hear actual useful things can be retained after the bubble finally crashes though. Oh please, let it crash tomorrow… together with the housing market…