He studied violin at Amsterdam Conservatory, and began his musical career as a violinist in the Netherlands Radio Philharmonic. He took conducting studies with Felix Hupka when he was at conservatory and then two Netherlands Radio Union conducting courses with Ferdinand Leitner. He was named second conductor with the Radio Union, leading four different orchestras or ensembles. He was promoted to principal conductor of the Netherlands Radio Philharmonic itself, at the age of 27.

After his good showing when called as a last-minute replacement at the Amsterdam Concertgebouw in Cherubini's Requiem in 1956, the Concertgebouw Orchestra frequently engaged him as a guest conductor.

After Eduard van Beinum died, two conductors were named principal conductor of the Concertgebouw: Haitink (at thirty-two the youngest principal conductor ever) and the veteran Eugen Jochum. Two years later, Jochum left, leaving Haitink in sole charge. He became the principal conductor and then artistic director of the LPO, retaining that position until 1979.

He recorded frequently with both orchestras, taking care not to favor one over the other out of concern for the musicians' feelings. An example of this occurred when he was planning a recording of Strauss's Ein Heldenleben, which has one of the most prominent of solo violin parts for a concertmaster. The LPO's concertmaster overheard these plans, and immediately started practicing the part so he'd be ready for the opportunity. Then the violinist learned to his disappointment that the sessions were planned for the Concertgebouw. When Haitink discovered that he had inadvertently hurt his violinist's feelings, he immediately scheduled the LPO to record Rimsky-Korsakov's Shcheherazade, the work with the other most prominent violin solo part.

He also frequently conducted opera productions in England as music director of the Glyndebourne Festival (1977-1988). He became music director of the Royal Opera in Covent Garden (London) in 1987, and has seen the company through the difficult period when its house was closed for radical renovations through the end of the 1990s.

In 1988 his association with the Concertgebouw ended when he resigned in protest over certain governmental economy measures affecting, in his view, the quality of the orchestra. He has increased his guest appearances in the years since, appearing with the Berlin Philharmonic, Vienna Philharmonic, London Philharmonic, London Symphony Orchestra, and the Boston Symphony Orchestra (of which he is the Principal Guest Conductor).

Reconciliation with the Concertgebouw occurred when, several years after his departure, the orchestra named him its Conductor Laureate, the first time the Concertgebouw has bestowed the title, in recognition of his great contribution to their history over a thirty-year period. In 1999 they invited him to lead them in a set of Concertgebouw concerts called the "Carte Blanche" Series. In it he led the Royal Opera House Orchestra, the Vienna Philharmonic, the Dresden Staatkapelle, Berlin Philharmonic, European Union Youth Orchestra, and Netherlands Radio Orchestra as well as the Concertgebouw Orchestra.

He has recorded a large quantity of records on the Philips, Decca (London) and EMI labels, with a wide range of repertory. These have included complete symphonies of Vaughan Williams, Mahler, Bruckner, Beethoven, Shostakovich, and Brahms, and many other works. He was won numerous awards, an honorary British knighthood, and the House Order of Orange-Nassau, given to him by the Queen of the Netherlands.