I wouldn't worry about...

if you call the examiner and ask him he might think up a way where it couldn't be PIC... however I've done what you did (logging time) and several other people I know have once upon a time and nobody cared...

not to be sneaky... but I would just put PIC down... and not mention anything about being a safety pilot in the comment section... I've really only mentioned the safety pilot's name, etc if I was under the hood, not visa versa.

If you touched the controls on the ground at the destination airport... you could probably safely say that it was XC time in your situation.