Isaac Wallis Baker was born on April 4, 1818 to Joseph Baker, of Beverly, Massachusetts. He was a sailor by profession; however, he was also an artist, daguerreotypist, and lecturer. After hearing about the discovery of gold in California, Baker joined a mining venture, the Beverly Joint Stock Company, sometime in 1849. During the long voyage to California on board the bark San Francisco, he amused himself by keeping a journal about the ship's activities. Once in California, Baker and two companions left the company and headed for the mines at Stoney Point on the south fork of the Feather River. Baker did stake a claim; however, when it ran out, he sold his mining equipment and returned to New York in November 1850.

He returned to California in May 1852 on board John Q. Adams. In March of 1853, he began to learn the daguerreotype trade. That fall Baker returned to the east coast. In 1854 he began to deliver lectures about his time in California throughout Massachusetts (Palmquist, 94).

Baker married Lydia Smith Larcom in April 1844. He died in September 1862, it is thought, in Sumatra.