We know the struggle all too well. You feel like you’re talking to a wall and nothing is getting through, and any attempt to engage and inspire seems to fall flat. At some point, we all have to deal with problem employees: people who are just plain difficult. But it’s not the end of the world.

When you find yourself frustrated and angry with an employee, it can be easy to throw away hopes of improvement. But there might be more to the problem than meets the eye.

1. Before anything else, listen

The most important thing to remember is that your employee has a life outside of work. They have their own problems and hardships that you might be completely oblivious to. It’s easy to look at their behavior around the office and judge solely on that. You may think you have your mind made up about their character, but there are always more factors at play. Have a conversation, and seek to understand.

2. Give clear instructions on your expectations

You can’t expect your employee to read your mind. If you aren’t clear about what you want from them, they won’t make a change. The best way to make a difference is to simply sit down with them and talk. Learn about what’s going on with them and acknowledge their performance (without negative comments or accusations) while laying out your expectations. Be understanding and give constructive feedback, but keep in mind that this is a professional conversation. Putting your expectations in writing is also a good way to make them easily accessible and can clear up any confusion.

3. Be consistent

If you don’t like a certain behavior, don’t “let it slide” sometimes. Having a consistent set of expectations will make it so they know not to try to go against it in hopes of you being lenient “this time.” You serve as a role model for your employees. If you aren’t setting an example of meeting deadlines and bringing a good attitude, your employees won’t either. They will pay more attention to your actions than your words, so make sure they’re consistent with each other.

4. If you don’t see a change, set consequences

It’s important that you’re strong in what you expect. If they say they will make changes, but don’t, it’s time to take action. You shouldn’t make it sound like they’re hopeless. Say things like, “I still think you can do this.”

A good way to see a change in your employees is to set rules. Something like, “If this behavior doesn’t change by x date, there will be consequences.” You could say, “you’ll be let go,” or “you’ll be put on warning,” or “you won’t be eligible for a promotion.” Choose a consequence that’s fair yet significant.

5. Trust yourself

If all else fails, trust that you know what’s best. After giving them a fair chance to change, there’s a point where giving them more chances won’t help. It’s always good to be kind and understanding before being harsh and angry, but realize when you are being taken advantage of and put an end to it. If possible, don’t burn any bridges in the process. If you are clear and open in your position, this conversation shouldn’t come as a surprise to your employee. Trust that you know when it’s best for everyone for you to let them go.