For a computer to execute or respond to a command, it has to be translated into the only language a computer knows: the 0s and 1s of the binary number system. The 0s and 1s represent the on and off of the transistors.
What do you call one of these 0s or 1s? A bit. Which makes sense when you see how many of these bits it takes to represent a word, number, color, graphic or sound. They really are just a "bit" of something bigger.