Spain, who have not lost a World Cup qualifying game in 28 years, were behind at half-time after Khvicha Kvaratskhelia fired Georgia ahead.
Manchester City’s Ferran Torres slid home Jordi Alba’s cross to equalise.
Alba was also credited with the assist for Olmo’s late long-range effort which gave the visitors a crucial win in front of 15,000 fans.
Georgia’s Levan Shengelia was then sent off moments before the final whistle.
Spain, who host Kosovo on Wednesday, drew their opening qualifier 1-1 at home to Greece.
Georgia visit Greece on Wednesday.