Bringing this back on topic. If anyone can see if they can get AP mode working using hostapd and the rest of the non-NetworkManager things, let me know. I am not sure if my setup is wrong, or if it is just not working.
@MoffKalast Glad you figured it out. You were probably missing allow-hotplug for the ethernet in your /etc/network/interfaces.
Rohan