I just accidently removed a context link instead of opening it because I clicked slightly too quickly. Definitely human error but since there is A) no confirmation to remove and B) no way that I know of to manually re-add a context link, the one I removed is now just gone...
I would move these two options away from each other, and/or add a confirmation when removing context links.