You are correct that adultery is a term that restrictively only means "sex with some who is not your spouse." The term for adultery in Ex. 20.14 is [i]na'ap[/i], and refers specifically to married people having sex with someone else's spouse.

When we get to the New Testament, however, there are specific words for adultery (moicheuo) along with more generic words for illicit sex in a general situation. When the Bible brings those other terms and concepts into the picture, any sex outside of monogamous heterosexual marriage is a sin.

If you want to look some of them up, some of them are Acts 15.20, Romans 13.13, 1 Corinthians 6.18, Galatians 5.19, and others.

Doesn’t adultery only mean sex with someone who is married to someone else?

According to the 10 commandments, it says “though shalt not commit adultery”. The definition of adultery is sex with someone who is married to someone else. Doesn’t that mean that having sex before marriage is not a sin if you are both unmarried?