Recipients of Outlook tasks may not be able to see Accept and Decline after enabling Server-side.
As a workaround, you may exclude Outlook tasks by adding an exception in your Office 365 Xink rule based on the subject line.
Go to your Office 365 mail flow, edit your Xink-Auto-ReRouting-Catcher rule and click add exception.
Make your keyword unique(Ex. "ExcludeFromReroute") so the rule will not filter out other emails by mistake.
Save your changes and ask your users to always send Outlook Tasks with that specific keyword in the subject line.