Would you let this happen even though its obviously not appropriate?

So im only 13 and this guy I like is 14. He was standing behind me and out of nowhere he started playing the "are you nervous?" game with me and I couldn't see him the entire time. He started out on my shoulder so I really thought of it as nothing at first. But he slowly caressed his hand to my back near the fastening of my bra, then went down to the side of my hip, and then he was rubbing my butt but I still said I wasn't nervous. Next thing I know, his hand is completely between my legs rubbing me down there (all clothes still on) then I start laughing and he just stops. Of course, since I like him I totally enjoyed it but i acted like it was just joking around. Do you think not stopping him was a bad idea?