Oh. I’ve only just realized that the Activity entry appeared because of the birthday. I wouldn’t have thought that could happen if the Contacts and Calendar app aren’t installed.
Sorry, I don’t know how to remove a contact without the Contacts app.
Which brings me to the question: why don’t you install the Contacts app, delete the contact, and then uninstall the Contacts app again? Seems to be the easiest.
Regardless of that, it might make sense to create a bug report for it. The “add example contact on first login” feature was introduced 10 months ago. Today is the birthday of the example contact. So this is the first time it’s been noticed.