Due to recipient-based signature feature, your signature will refresh or reinsert itself every time you add a recipient address. If you do not intend to create rules for internal and external signatures or campaigns, you may disable this by modifying the XML or manifest file.
If you already deployed the add-in, you will need to select Update Add-in and upload the modified XML or manifest file.
Please see the below required modification.
Open the XML file with Notepad and Delete the below highlighted line. Save and Upload to Microsoft Integrated Apps.
<LaunchEvent Type="OnMessageRecipientsChanged" FunctionName="onMessageRecipientsChanged"/>
Note: There may be up to 72 hours replication time due to Microsoft Integrated Apps delay. Please test the function after this period to make sure that the update is already reflected.