Sorry for the delay in responding, it's been a whirlwind since the ceremony.
When I got to USCIS and handed my Oath Letter in at the desk leading to the ceremony hall, the supervisor asked me for green card. I replied "just the current green card or do you need the expired conditional green card and work permit cards too?", to which she replied "your green card and any other expired cards you have on you". I asked if I could keep them as souvenirs, and she said sorry, but no. So I had to hand all of them back. I guess I could have kept quiet about the conditional green card and work permit card, and could probably have kept hold of them. But as I brought them to her attention, I couldn't really hold them back and had to return them.
I did take photos of them before I went to the ceremony, so I still have those as a reminder of my journey from immigrant green card holder to citizen!