If you are the designated PIC of a flight, then you can log PIC for the entire flight regardless of what seat you sit in or who is PF/PNF. This is Part 1 PIC time.
That would mean the other pilot, even though he is PIC qualified, is a designated SIC; he could only log PIC for that time when he is the PF (via sole manipulator clause). This is Part 61 PIC time.
On the legs where you have an IRO, you're still the designated PIC responsible for the airplane, right? As such, you log PIC for the entire flight.
In my one-jet operation, we have two PICs. My title is Chief Pilot, the other guy's title is Captain. Our arrangement is there is only one acting & actual PIC for any given flight - the guy occupying the left "command" seat. On any given flight, the guy in the left seat drives the bus and acts/logs PIC; the guy in the right seat acts as PNF and logs SIC. Swap seats, swap responsibilities, swap what you log. Works well for us...