Yes, if you filed your N400 prior to April 2004 (before the option to inform SSA automatically) you had to go in. I had to as well. Even after the form update, new USCs still were reporting they had to go in up until recently.
If you try ordering a replacement card now, it should let you go through the steps as they have you down as a USC.