WASHINGTON (AP) — When President Donald Trump made a congratulatory phone call to Russian leader Vladimir Putin, White House chief of staff John Kelly wasn’t on the line.

When Trump tapped John Bolton to be his next national security adviser, Kelly wasn’t in the room.

And when Trump spent a Mar-a-Lago weekend chafing over immigration and trade, Kelly wasn’t in sight.

Kelly, once empowered to bring order to a turbulent West Wing, has receded from view, his clout diminished, his word less trusted by staff and his guidance less tolerated by an increasingly go-it-alone-president.

Emboldened in his job, Trump has rebelled against Kelly’s restrictions and mused about doing away with the chief of staff post entirely. It’s all leading White House staffers and Trump allies to believe that Kelly is working on borrowed time.

In recent weeks, Trump has governed at breakneck pace, ousting aides and issuing surprise policy announcements on Twitter, recreating the helter-skelter feel of his first months in office. Kelly’s allies maintain that his retreat is strategic. They suggest that the belief that Kelly was Trump’s savior was an overstated idea all along and that the chief of staff is now content to loosen the reins and allow an increasingly comfortable president to govern from his gut.

But those close to the president say Trump has increasingly expressed fatigue at Kelly’s attempts to shackle him and say that while Trump is not ready to fire Kelly, he has begun gradually freezing out his top aide.

Trump recently told one confidant that he was “tired of being told no” by Kelly and has instead chosen to simply not tell Kelly things at all, according to a person who was not authorized to publicly discuss private conversations and spoke on condition of anonymity.

In Trump’s West Wing, once the rumors begin that an aide’s exit is forthcoming, the “stink” on that staffer never leaves, according to one of the nearly dozen of White House aides, former administration officials and outside advisers who spoke to The Associated Press under the same conditions.

As Kelly’s public profile and behind-the-scenes influence has faded, speculation has risen that chaos could return.

“It’s not tenable for Kelly to remain in this position so weakened,” said Chris Whipple, author of “Gatekeepers,” a history of modern White House chiefs of staff. “More than any of his predecessors, Donald Trump needs an empowered chief of staff to tell him what he does not want to hear. Trump wants to run the White House like the 26th floor of Trump Tower and it’s simply not going to work.”

Kelly was once a fixture at the president’s side, but Trump has now cut him out of a number of important decisions.

For months, Kelly made it a practice to listen in to many of the president’s calls, particularly with world leaders. While he is still on the line for some of those conversations, Kelly was not part of the call Trump made to Putin last month from the White House residence, during which Trump ignored advisers’ advice against congratulating the Russian president on his re-election.

Though Kelly had agitated for the removal of outgoing national security adviser H.R. McMaster, he counseled Trump against hiring Bolton, an incendiary neo-conservative media commentator. Trump did it anyway, offering the job to Bolton in a one-on-one meeting in the Oval Office and telling his chief of staff about it later.

As Trump spent the Easter weekend at his Florida resort and tweeted about his tariffs plan, Kelly was out of state, though the men did consult by phone. While Kelly has fumed about the ethics questions swirling around Environmental Protection Agency head Scott Pruitt, Trump was at least initially more supportive of Pruitt, telling him “we’ve got your back.”

The president also has cast aside the constraints the retired four-star Marine general tried to place on Trump about whom he could see and speak to. Those restrictions led shunned advisers to try to undermine the chief of staff in the press and with Trump. For months, former campaign manager Corey Lewandowski was barred from the White House, only to return when Kelly was overruled by Trump, according to four White House aides and outside advisers.

Many in the West Wing believed that Kelly’s attempts to curtail Trump’s interactions with Lewandowski, as well as Trump allies such as David Bossie and Anthony Scaramucci, were always destined to fail and alienate the president, who has privately contemplated about recreating the freewheeling nature of his campaign and Trump Tower office. Kelly also has clashed with Trump’s son-in-law and adviser Jared Kushner, who lost his security clearance after a policy change authored by the chief of staff.

Some White House aides contend Kelly has been intentionally giving Trump more leeway to be himself and that Kelly recognizes that’s what Trump wants. But allies acknowledged Kelly’s receding power and said he’s trying to keep his head down and focus on policy, such as the plan to mobilize the National Guard along the U.S. border with Mexico.

The White House declined to make Kelly available for an interview. In public, Trump praises his chief of staff, telling Marines in California last month that Kelly probably “likes what you do better than what he does, but he’s doing a great job.”

The speculation surrounding Kelly echoes the treatment of his predecessor, Reince Priebus, who was the subject of months of questions about how long he would last on the job. Priebus eventually resigned under pressure.

Kelly also no longer commands the same respect among some quarters of the staff. His role came under harsh scrutiny this year over his handling of the controversy surrounding ousted White House aide Rob Porter, who was accused of domestic abuse.

Kelly’s shifting version of events drew a loud rebuke from former communications director Hope Hicks, who had been dating Porter, and dismayed a number of West Wing staffers. That episode frustrated Trump, who also still remains agitated about an interview that Kelly gave to Fox News months ago in which he suggested that Trump had “evolved” in his thinking about the need for a wall on the Mexican border.

Kelly, who took the job last July, had previously told confidants he hoped to be on the job for a year. One person familiar with his thinking said the chief of state recently voiced doubt he would make it that far.

In this April 3, 2018 photo, President Donald Trump’s Chief of Staff John Kelly attends a news conference in the East Room of the White House in Washington. Kelly, once empowered to bring order to a turbulent West Wing, is receding from view, his clout diminished, his word less trusted by staff and his guidance less tolerated by an increasingly go-it-alone-president. Emboldened in his job, President Donald Trump has rebelled against Kelly’s restrictions and mused about doing away with the chief of staff post entirely, leaving White House staffers and Trump allies to believe that Kelly is working on borrowed time. (AP Photo/Andrew Harnik)