One the one hand, agree, on the other, I think they’re trying to get at something beyond right-of-way with “shall not take advantage of” language for which perhaps other countries may get better. Something for which both aircraft should be considering:
I think you might be mixing apples and oranges. When flying, yes you always have to watch out for the other guy. But if there are not specific rules and Right of Ways, then you get people suddenly appearing where you (and no one else) expected them to come from. (And my Swedish friend agreed you may be pushing the watch out for the other guy and there are no Right of Ways concept.

)
It is pretty clear that the IFR and Straight In traffic needs to give way to anyone in the Pattern that they may affect. It's in the Regs, AIM, etc., but AC 90-66C puts it all in one place. (I also tossed in the one about not reporting a Fix on an Approach plate, but to give your distance and direction so VFR pilots know where you are.)
AC 90-66C Excerpts:
9.11.3 IFR Traffic. Pilots conducting instrument approaches in VMC should be particularly alert for other aircraft in the pattern so as to avoid interrupting the flow of traffic and should bear in mind they do not have priority over other VFR traffic. Pilots are reminded that circling approaches must comply with § 91.126(b) unless the approach procedure explicitly states otherwise. Remember, if the Chart Supplement lists right-hand traffic at a non-towered airport, pilots conducting practice instrument approaches in VMC should circle to the right to enter the traffic pattern.
9.11.5 Consider That Non-Instrument-Rated Pilots May Not Understand. A non-instrument-rated pilot might not understand radio calls referring to approach waypoints, depicted headings, or missed approach procedures. IFR pilots often indicate that they are on a particular approach, but that may not be enough information for a non-IFR-rated pilot to know your location. It is better to provide specific direction and distance from the airport, as well as the pilot’s intentions upon completion of the approach. For example, instead of saying, “PROCEDURE TURN INBOUND V-O-R APPROACH 36,” state, “6 MILES SOUTH INBOUND V-O-R APPROACH RUNWAY 36, PLANNING TURNING (right or left) TO AVOID FLYING OVER THE AIRPORT, or state, “6 MILES SOUTH INBOUND V-O-R APPROACH RUNWAY 36, LANDING FULL STOP.” You should be clear and concise. The other traffic needs to know what to expect in order to take appropriate actions that avoid a midair collision.
9.13 Pilot Responsibility to See and Avoid and Pilot Avoiding Careless and Reckless Operations (§ 91.13). In the airport traffic pattern, effective communication and a pilot’s responsibility to see and avoid are essential mitigations to avoid a possible midair collision. In addition, following established traffic pattern procedures eliminates excessive maneuvering at low altitudes which mitigate the risk of a loss-of-aircraft control. All aircraft in the pattern or those flying a straight-in approach for landing in order to mitigate the risk of a conflict should effectively communicate their type of aircraft, distance from the runway in use, and state that they will or will not land on the runway in use. It is important to remember that non-adherence to established procedures and operations in the airport traffic pattern or on the surface of the airport that could endanger the life or property of another and could expose the pilot to an investigation based upon their careless or reckless operation.