Where does it say you have to log PIC time in order to get your SOE removed?

It says you must perform the duties of PIC. I simply logged total time and SIC time. I mad a comment after each flight saying I performed the duties of PIC as per 61.64 and had the captain sign it. The FAA did not have any problems with that.

But yes, you can log it as PIC if you are sole manipulator, but you do not have to in order to get the SOE removed.