Yes, if someone has physical access to your device, they could disassemble and/or pull keys from the flash.
There are not many good solutions to this yet. Secure elements (Securing LoRaWAN with Secure Elements) is one, but it’s far from standardized.