The people in the house didn't keep watch constantly; you could see someone sitting in a room on the first floor now, and it was just one person, not the usual multiple shifts for 24-hour surveillance.
During the past few days, no one was watching Jesse here, which was a significant oversight. But since Hank and Gomez were both taken down, Jimmy's identity was not clear at that time, so it wasn't realistic for him to keep watch himself.
It wasn't until evening that Jimmy realized Jesse had returned from outside, meaning they had been watching in vain all day, originally planning to see him leave, but he hadn't returned all day.
Jimmy patted the DEA brother keeping watch out front, saying he would rest for a while since Jesse had just returned and probably wouldn't leave again soon.
However, he soon found himself unable to sleep. Near midnight, he received a phone call, "Mr. Yang, are you able to take the call now?"
