I believe this has been covered by an FAA written legal opinion.

There are numerous grey areas in the FAR's which are subject to interpretation. The FAR's at face value are kind of like Cliff's Notes...they cover the basic structure, but don't fill in all of the details. Since these are administrative regulations, not actual laws, the FAA can make their own interpretations which normally hold up in court. There are many, many such grey areas that are addressed in Advisory Circulars (such as the notorious "Holding Out" AC) or in FAA written legal opinions. Rule interpretation can also be solidly established by case law where someone appeals an FAA regulatory action, and the NTSB finds in favor of the FAA...that effectively makes the FAA interpretation solid and binding for anyone else in the future.

In this case I agree with the FAA in that dual-given is obviously intended for training purposes, not to allow time building. It really is not unreasonable to exclude activities which obviously have no training purpose from being logged as flight instruction. Especially since the FAA openly allows Safety Pilot time to be used for time building purposes.