When Avery was introduced to "Yellowstone" at the end of Season 1 and subsequently arrived in the bunkhouse in Season 2, Jimmy immediately had eyes for her. The reluctant cowboy showed it, too, in his own somewhat awkward way, and Avery even seemed to be a little impressed by him a couple of times. Still, it turns out Avery had a more traditional cowboy type in mind — a type Kayce Dutton fit perfectly.
She said as much in Season 4 when she met up with Kayce again, and even Kayce's wife Monica (Kelsey Asbille) noted the tension between them. While this character development was likely added in order to explain Avery's absence and return, it does help to make sense of why she may have quit her job at the ranch. From her perspective, someone she lived with clearly had feelings for her, while she herself had eyes for a married man. This means that if she went for Kayce, she could potentially ruin many lives, her own included. Thus, Avery's decision to remove herself from the whole situation makes a lot of sense.
While we haven't seen Avery yet in Season 5, it was teased in Season 4 that Kayce would have to make a big choice somewhere down the line. Though viewers aren't necessarily sure what that choice is, it's possible that it will be between Avery and Monica. After all, "Yellowstone" clearly enjoys teeing up a good love triangle.