The Labour MP Jess Phillips has described her party as sexist for having no permanent female leaders in its history.

Asked at the Hay literary festival why Labour has never had a UK-wide female leader, Phillips responded that “like all institutions it’s a bit sexist, innit”. She said that “every institution that every single person in this room works for is led by the patriarchy”.

Phillips, the MP for Birmingham Yardley, resigned as minister for safeguarding and violence against women and girls on 12 May – one of four ministers to step down that day citing loss of confidence in the prime minister, Keir Starmer. Wes Streeting, the health secretary, followed on 14 May.

The Conservatives have had three female prime ministers, but Labour has never had a permanent female leader. Margaret Beckett and Harriet Harman served only as acting leaders.

Jess Phillips at the Hay festival on Monday. Photograph: Sam Hardwick/Hay Festival/PA

That record looks unlikely to change if a Labour leadership contest does take place this summer – with Andy Burnham as the likely frontrunner if he wins the Makerfield byelection on 18 June and Streeting his competition from the right of the party.

The UK party has never elected a female leader, but three women – Wendy Alexander, Johann Lamont and Kezia Dugdale – have led Scottish Labour. The Welsh Labour party was led until this month led by Eluned Morgan, who was the country’s first minister.

Angela Rayner being cleared by HMRC over her tax affairs earlier this month paved the way for a leadership bid, but she has not declared an intention to run and has said she will campaign for Burnham in next month’s byelection. Both belong to the so-called “soft left” of the party.

Phillips said she hoped for “wild cards” in a leadership contest, but refused to specify who they might be. “It’s whether Keir Starmer would stand in the contest, I suppose, is the wild card bit of it.” If Starmer does not step down, he would automatically appear on the ballot.

Phillips appeared at a panel event alongside the Green party’s Ellie Chowns and the Times journalist Patrick Maguire.

She called the position Labour had got itself into as “kind of bants”, later describing it as terribleand heart-wrenching.

Phillips said she was going to Makerfield to campaign for Burnham. Asked about the fact that Starmer would be campaigning for the man who was likely to replace him, she said “all that should matter to anyone in politics is that you care about the country and the right outcome”.

On the lack of female Labour leaders, Phillips added: “I have to say, the idea that a woman leads and it makes it better for women is not one that I’ve recognised in this country.

“My mortgage went up by a thousand pounds a month, cheers Liz [Truss]. I shouted at her, every time I saw her, the amount that she now owed me. She has not paid me back.”

Phillips also said that the fact Starmer was seen as a “good guy” had become “throwaway” but should not be because “we didn’t have that for a long time. We didn’t have decency.”

She said the “anti-politics” of the Greens and Reform UK sounded the same to her, and that while she might agree with the Greens more ideologically, she thought it was “very, very easy to just slag off the system and the people who are part of it without actually committing to the real solutions”.