Damon's talking about the scene at the end of Rose (2x08), Where he confessed he loved her and then compelled her to forget it. If I recall correctly he said, "It's because I प्यार आप that I can't be selfish with you." So in 4x01 "I am that selfish. the first time we met's not all that आप remember." is a remark to that memory. Hope it helped.
Damon speaks about his first प्यार confession. But he at the same time indicates that he is not selfish, especially when it comes to Elena ("Because I am that selfish, but आप knew that already, the first night we met is not the only thing आप remember."). Check out the लेख "The Truth About Love: Why Elena's Logic in 4x01 Doesn't Make Sense", where it is explained why Damon's choice to save Elena is not selfish, because he is willing to भालू her hating him as long as she is सुरक्षित and alive.
Damon is referring to the scene in 2X08 ("Rose"), in which Damon tells Elena that he loves her and then compels her to forget. It is totally and 100% heart-breaking. The first episode of season 4 depicts Elena remembering that specific memory.