Zinedine Zidane's men have now won just one of their last five games as a Danilo own goal and Daniel Wass' finish led to elimination from the Copa.
A blistering free-kick effort from Cristiano Ronaldo was not enough to avoid another damaging defeat for Real Madrid, as Celta held on for a 2-2 draw to dump the Merengue out of the Copa del Rey.
Celta had stunned Madrid by winning 2-1 at the Santiago Bernabeu in last week's quarter-final first leg - a result that formed part of a three-game winless run for Zinedine Zidane's La Liga leaders.
Saturday's league victory over Malaga looked to have put Madrid back on track, despite the addition of Marcelo and Luka Modric to an injury list that already features Gareth Bale and Pepe.
Madrid remained the more fancied to go through despite the first-leg result, but fell further behind late in the first half when Danilo put through his own net - not that he knew much about it.
Ronaldo powered in a free-kick after 62 minutes as Madrid sought to claw back the deficit but Daniel Wass, having played a key role in Celta's opener, placed a shot beyond Kiko Casilla with five minutes left to re-establish the cushion.
Vazquez headed home in the 90th minute to give Madrid some hope but the visitors were unable to deny Celta a second successive appearance in the Copa semi-finals, leaving the men from the capital to focus on their LaLiga and Champions League campaigns.
Celta came into this match with something to defend, and defend they did in the early stages as Madrid set about wiping out the hosts' first-leg advantage.
The home side were happy to sit deep, soak up the pressure and wait for opportunities to hit Madrid on the break, although their tactics almost backfired when Ronaldo hit the woodwork after 26 minutes.
Isco's cross from the right was perfect for the Portugal captain, who would have headed Madrid in front on the night had it not been for the lightning-quick reactions of Sergio Alvarez, who palmed the powerful effort onto his crossbar.
That near miss appeared to spark Celta into life and they should have done better when a poor Casemiro clearance presented them with a glorious opportunity to stretch their aggregate lead.
The ball fell to Iago Aspas 18 yards out but he made the wrong decision in attempting, and failing, to square for John Guidetti and the move broke down.
Celta's new attacking approach did, however, pay off a minute from the break.
Finding yet more space in the Madrid defence, Wass succeeded where Aspas had earlier failed by finding Guidetti with a squared pass.
The Swedish forward's shot hit Casilla's leg before rebounding into the net off an unsuspecting Danilo.
Madrid were desperate for a goal but they struggled to carve out any clear-cut opportunities until Ronaldo ominously stood over a free-kick some 20 yards out.
The club's record goalscorer arrowed an effort down the left-hand side of the Celta wall and beyond Alvarez, who may question the positioning of his defensive line.
Madrid had their tails up and Sergio Ramos nodded just wide from a Toni Kroos free-kick, before another Ronaldo set-piece crept just outside the right-hand upright.
Wass looked to have put the result beyond doubt when an intricate passing move eventually saw the ball pop out to him on the edge of the penalty area, picking his spot and placing a shot beyond Casilla.
Substitute Vazquez made it 2-2 on the night with a 90th-minute header, but Madrid were beaten by the clock in their desperate bid for a crucial third goal.
Key Opta stats:
- Real Madrid have now lost more games in 2017 (three) than last year (two - all competitions).
- Cristiano Ronaldo is the only La Liga player to have scored a direct free-kick in three different competitions this season (only clubs).
- Celta have knocked out Real Madrid in Spanish Cup for the second time ever, first since 1941, 76 years ago.
- Only against Atletico Madrid (six) Cristiano Ronaldo has scored more Copa del Rey goals than against Celta (five).
- Eduardo Berizzo is the only manager to have won a competitive game against Luis Enrique, Zinedine Zidane and Diego Simeone.
