I used to work as a PQI contractor at Nest, and we actually had test setups like this in the office that were just a circuit breadboard mounted on a plate behind the thermostat. The thermostat doesn't really "communicate" with the HVAC control system at all (all it does is just send ~3V along the circuits based on its current mode), so as long as you have the circuits routed properly on your board, the thermostat will think it's connected to a real system.

The stock firmware doesn't let you go above a certain temp (like 80F or something, I forget the exact limit). But if your custom firmware allows it, the only thing that would realistically happen is that it just runs a 3V circuit to what it thinks is the heater, infinitely, since nothing is actually causing the ambient temperature to raise at all. The ambient temp reading comes from a sensor on the thermostat, itself.