The religious leaders wanted to arrest Jesus because they realized he was telling the story against them--they were the wicked farmers. But they were afraid of the crowd, so they left him and went away.