Matthew says it’s adultery if you even have lustful thoughts. Got that and agree.
What if you’re unmarried?
Also… what about the person (lets presume it’s a woman) whose being lusted over without knowing it?
It’s not fair on her! She has no choice and that in theory makes it rape-like.
What can you do about it?

Let me annoy you by starting with Matt 15:18 , which wasn’t the story you referred to (I’ll come to that in a minute).
“But the things that come out of the mouth come from the heart, and these make a man ‘unclean.’ 19 For out of the heart come evil thoughts, murder, adultery, sexual immorality, theft, false testimony, slander. 20 These are what make a man ‘unclean’; but eating with unwashed hands does not make him ‘unclean.’”

Jesus is trying to convince these outwardly observant Jews that all their outward observance is meaningless unless they have first got their heart right.

The modern day equivalent is when your friends say, “but there are lots of good people who are not Christians, surely they’ll go to heaven because they have been good?”

Oh no. That doesn’t get past Jesus. He says FIRST you have to get the HEART right, and submitted to God’s rule. Unless you do that, any good works you might do are done on your say so, for your sake, because you want to, because you think it would be a good idea.

But you can’t see the bigger picture, only God can – how do YOU know what is good? God wants shift you into a position where he can REALLY use you to do what is good. How can He do that if your heart is not even listening to Him, if all it hears is its own voice shouting, “THIS IS WHAT I WANT, THIS IS WHAT I WANT!”

So, let’s go back to your story. It’s from Matt 5 27   “You have heard that it was said, ‘Do not commit adultery.’ 28 But I tell you that anyone who looks at a woman lustfully has already committed adultery with her in his heart.

The point Jesus was making was the same point. If you are really careful to keep your hands off other people’s spouses, but spend all your time in lustful thoughts about them, that just shows that your heart is not being made pure by the action of God’s HOLY Spirit.

The fact you haven’t actually stolen other people’s spouses might be good in itself, but YOU are in the same mess inside yourself as if you HAD strayed.

So this is what Jesus is concentrating on – not the semi-legal discussion about what constitutes adultery (eg. what if the other person isn’t married but only living with someone, what if they are single, what if you are engaged to them, what if they are under the legal age for sex in your culture).

No, what Jesus is trying to tell these “holy” people is that they are too easily satisfied when they look at their lives. They have got all the observances, but have forgotten the God who it’s all for. They have the “form” of Godliness without the power of it. (2 Tim 3 5)

OK you say, but what DO I do in all those different circumstances? What is permissible if the other person isn’t married, or is already my fiance, etc.

The basic Bible answer is:
“ Go to God and get your heart right with Him and invite the Holy Spirit to mould your heart to match God’s heart. Then keep your heart close to Him every minute of the day. Then do whatever doesn’t break the relationship between you and Him. Then check out your actions every now and then with Scripture to make sure you are staying on the path and not straying off it.”

It’s different from following rules, but better.

I’m conscious that I haven’t answered your question in a way you might have expected. Please write in if you can still see areas which need clarifying.

Related Q & A