The Furnaces played their initial gigs at a small club called Enid's and branched out from there, going through several lineups of supporting musicians as they played gigs with the French Kicks, Sleater-Kinney, and Spoon. In 2002, they began working on their first album. By the time they signed to Rough Trade on the basis of their demo, their debut, Gallowsbird's Bark, was completed and the Fiery Furnaces were already at work on the follow-up. Gallowsbird's Bark arrived in fall 2003 and won critical acclaim for its charming kitchen-sink feel, but the band gained more momentum the following year, when praise for the debut album dovetailed with the release of the group's even more diverse and challenging sophomore album, Blueberry Boat, that summer.
The Fiery Furnaces spent much of 2004 touring with Ted Leo & the Pharmacists, Franz Ferdinand, and the Shins, but were already working on their third and fourth albums. Early in 2005, the Furnaces released the simply titled EP, a mini-album gathering most of their B-sides along with a few new songs. That fall, the band returned with Rehearsing My Choir, a challenging, stream-of-consciousness album featuring the Friedbergers' grandmother, Olga Sarantos. They moved to Fat Possum for Bitter Tea, a collection of poppier songs that arrived in spring 2006. ~ Heather Phares, All Music Guide