Why Did Only 10% Of The American Population Fight In The Civil War?

During the civil war, roughly 3 million people fought in it. At the time America had roughly 30 Million. So all in all, about 10% This has really baffled me because, little boys (between 7 and 10 years old have been reported), women and also Black people fought in the war. I understand that if you paid $300, you could refuse to fight. Surely to God, 90% of people never had the money not to fight. Not including women and children of course who chose not to participate :)