Hi,

At this moment, the loyalty program does not verify canceled transactions - it is not possible to set the integration to exclude canceled orders from being counted in the loyalty program.
You can, of course, do this through a rule/workflow that would subtract loyalty points under the desired conditions, but in such a situation you need to provide a fixed number of points that would be subtracted. 
Maybe it's worth considering this in terms of developing the functionality of the Loyalty Program?