Filter the target IP with low priority and see if it works or not.
many have misconception about QoS.it is not related to bandwidth limiting but bandwidth prioritizing.if you are using QoS preferred application will receive traffic first but it will not limit the amount of traffic to other applications only that they will get it after preferred application.in short total bandwidth usage over a fixed time will remain same.