Keeping Yourself Small?

So many of us (me included) have spent years keeping ourselves small. Who are we afraid of and why do we care what others think?