Chancellor Philip Hammond has said he will prioritise spending on new homes and transport rather than following his predecessor George Osborne's aim to balance the books by 2020.
He told the Conservative conference the deficit was still too large and would need to be tackled "in due course".
But he said the Brexit vote may cause "turbulence" and business confidence would be on a "bit of a rollercoaster".
He said that it was "common sense" to invest to support growth and jobs.
The BBC's political editor Laura Kuenssberg said it was a "big change".
She said that while Mr Hammond had already dropped his predecessor's target of eliminating the deficit by 2019-2020, what was particularly striking was his refusal to set a new timetable to bring the budget into surplus.
In his conference speech, Mr Hammond said Mr Osborne's deficit reduction policies "were the right ones for that time" but that times had changed since the vote to leave the EU.
"When times change, we must change with them," he said. "So we will no longer target a surplus at the end of this Parliament. But make no mistake the task of fiscal consolidation must continue.
"The British people elected us on a promise to restore fiscal discipline. And that is exactly what we are going to do. But we will do it in a pragmatic way that reflects the new circumstances we face."
While the market had "calmed" since the Brexit vote, he said it had caused uncertainty for business and the government had a duty to act to support the economy.
Mr Hammond's speech came against a background of the pound falling to its lowest level against the dollar since early July, after the timetable for starting Brexit negotiations were set out on Sunday.
Addressing the party faithful he said that as part of a new "flexible and pragmatic" plan, the details of which will be fleshed out in next month's Autumn Statement, there would be greater scope for investment to boost the economy, including extra borrowing of £2bn to speed up the construction of new homes.
Mr Hammond said the government would use "all the tools at its disposal" to increase the amount of new housing stock "because making housing more affordable will be a vital part of building a country that works for everyone".
Asked whether the public investment plan marked the end of the "age of austerity", Mr Hammond told BBC Radio 4's Today the annual budget deficit was still "eye-wateringly large" and that a framework was still required to get the public finances into the black "in due course".
But while the UK economy appeared to be in robust shape, he said there was "anecdotal evidence" of greater caution since the Brexit vote - with some firms putting off investment decisions - and it was the government's duty to help prepare the economy for any potential softening and to boost long-term productivity.
"As we go into a period where inevitably there will be more uncertainty in the economy, we need the space to be able to support the economy through that period," he said. "If we don't do something, if we don't intervene to counteract that effect, in time it would have an impact on jobs and growth."
Out with the deficit hawks - in with the conservative pragmatists.
That might not, well, quite catch on. But there is certainly a sense of down with the old regime at the Tory conference today. Philip Hammond isn't just a totally different character to the former occupant of the job, George Osborne, but he is intent on junking a fair bit of Mr Osborne's approach too.
His targets for getting rid of the deficit are being ditched. Let's face it, George Osborne missed them time and again in any case, but it is politically a big shift for Mr Hammond and Mrs May to tear up the previous borrowing rules, when balancing the books was the central mission of the government they were all part of.
Instead, there will be a new "framework" but sources close to Mr Hammond believe that setting specific targets is a fool's errand. But given that Mr Osborne's target was extremely unlikely to be met, the political change is perhaps bigger than what it means in practice.
Mr Hammond rejected claims that he was now "more Balls than Osborne" - a reference to former shadow chancellor Ed Balls who called for more public investment before the 2015 election - saying that while Labour were making "ludicrous" promises on spending - his plan would be "careful and measured".
"I hope I am sounding like a Conservative pragmatist faced with the challenge of uncertainty in the economy," he said.
He added: "There is a distinction in my mind between investing in the things that will make Britain's economy more efficient in the future, make transport systems work better, communications systems work better and simply spending more on our day-to-day process of government."
stage on Monday with Mr Hammond, who was appointed by Mrs May to replace Mr Osborne when she became prime minister, promising £220m to help tech firms realise their ideas in the UK and guaranteeing firms which secure multi-year funding between now and the UK's exit that the UK will plug the gap after Brexit.
Earlier on Monday, Communities Secretary Sajid Javid set out new measures aimed at building a million new homes by 2020, saying existing house-building levels were totally inadequate.
The government will borrow £2bn to support the "Accelerated Construction" scheme, which aims to get houses built on publicly-owned "brownfield" land available for swift development. There will also be a £3bn Home Building Fund to provide loans to stimulate projects, which the government said would build more than 25,000 homes by 2020, with a long-term goal to build more than 200,000.
Mr Hammond's speech comes after Mrs May addressed the Tory conference on Sunday for the first time as prime minister, with the first day of the gathering focusing on the 23 June vote to leave the EU.
She announced she would formally trigger Brexit using Article 50 of the Lisbon Treaty by the end of March 2017, paving the way for the UK to be outside the EU by the summer of 2019. She also announced a "Great Repeal Bill" would be included in the next Queen's Speech to remove the European Communities Act 1972 from the statute book and that the UK would become an "independent, sovereign" country.
Asked if he was worried about the repercussions should the UK leave the EU's single market, Mr Hammond said he was focused on getting the best deal for British businesses and workers and although he was not one of those ministers focused specifically on Brexit - he said the Treasury would be "right at the heart" of discussions.