@ghettokon Sounds like you have an 'audio curve' non-linear volume control pot!
If it is a linear pot, then you either use a voltage divider or add resistors to the top pin or bottom pin as required to get centre.
If 'volume control pot' then you will have to characterize the signal in your code.