Built in the 17th century and reconstructed in the Neo-Classical style by Simone Cantoni(b. 1736-d. 1818), it was modified at the behest of Gian Giacomo Poldi Pezzoli (b. 1822-d. 1879) for his art collections in 1850-53 by Giuseppe Balzaretto (b. 1801-d. 1874), also responsible for the restyling of the public gardens on the other side of the medieval city gate just down the road. The museum opened in 1881 shortly after Poldi Pezzoli's death.
Gian Giacomo had a busy life. He participated in the anti-Austrian uprisings that led to the Cinque Giornate, whose success was betrayed by a Savoy at the nearby via Manzoni 6. Forced into exile for a year, he turned his attention and wealth to art collecting.
A bachelor, he surprised everyone in his will (the nearest cousins the most, I suspect) by donating it all to the private museum bearing his name. The incredible masterpieces he had gathered are continually augmented, as he expressly stated in his will. This, and the fact that the structure was extremely heavily bombed in WWII means that the collection is now a very different one, and the museum, itself, has changed completely. From the house museum it once was, it was rebuilt by the famous architect Luigi Caccia Dominioni in 1974 as a normal museum: except for a rare exception, the rooms no longer resembling the private house it once was. The museum was again retouched in 1993-1994, and the arms collection display completely turned on its head--scenographically, but museologically ineffectively in my opinion (that will earn me brownie points for sure...)--by the famous sculptor, Arnaldo Pomodoro. (Why ineffective you ask? Because it's dark, hard to see, and places too much emphasis on the whiz-bang of the sculptor's display than it does on the marvelous Renaissance pieces that seem to be there to serve the exhibit, and not visa versa, as it should be.)
Walk a few paces along the adjoining Poldi Pezzoli building (n. 14) towards the medieval gate, and you'll pass a break between this block of buildings and the next. Through the tempting and horrifically expensive restaurant "Don Lissander" (for Don "Alexander Manzoni," who also lived nearby) you'll be able to glimpse what used to be Poldi Pezzoli's back yard, a lovely 'small' park which he inherited from his Trivulzio mother, and which still belongs to the ennobled Trivulzio family and their family palazzo on the perpendicular via Bigli.
The garden isn't open to the public, but you can get a better and nosy glimpse of it if you go to the back of the commercial gallery at n. 16. It's quite unusual (for us hoi polloi, anyway) to have single family gardens in Milan...a smaller one--not too big, not too small, but juuuuuuust right (I'm not greedy)--on via Amedei that I yearned for for years also turns out to belong to their family...is this Fate?