James Duncan Phillips was born February 5, 1876, in San Francisco, California, to Stephen Henry (1823-1897) and Margaret (Duncan) Phillips (1847-1926). Phillips graduated from Salem High School and later Harvard College in 1897. His first job after graduation was in the Dwight Manufacturing company cotton mill in Chicopee, Massachusetts, but he left after one year. Shortly thereafter, Phillips began working in the editorial department of Houghton, Mifflin and Company. He later became general manager, and when the business was incorporated as Houghton Mifflin Company in 1907, he became director and a member of the executive committee. He established the publisher's educational offices in New York, San Francisco, Dallas, and Atlanta. Phillips also served as a director of the American Mutual Liability Insurance Company.

During World War I, Phillips was the Chairman of the Committee of Public Safety and was captain of the Machine Gun Company, 15th Regiment, infantry, of the Massachusetts State Guard. In his twenty-fifth anniversary report to his Harvard College class, Phillips noted that his company "at the time of the police strike in Boston, served under arms longer than any state troops ever had in Massachusetts." He also served as the food and coal administrator for his area during the war.

In March 1907 Phillips married Nannie Jenks Borden (1877-1963) at her family's ancestral home at Headcorn, Kent, England. The couple traveled extensively through Canada and to Hawaii, Fiji, New Zealand, Australia, Ceylon, the West Indies, Africa, and in England and Europe. When not traveling, they lived on their farm in Topsfield, Massachusetts. Due to a serious heart attack in 1936, Phillips' business activities became limited. He filled his time with historical research and writing. Numerous articles and books, most of which were about Salem's "Great Age of Sail," were published during the last twenty years of his life. His work includes: Salem in the Seventeenth Century, Salem and the Indies; the Story of the Great Commercial Era of the City, Salem in the Eighteenth Century, Pepper and Pirates; Adventures in the Sumatra Pepper Trade of Salem, The Life and Times of Richard Derby, Merchant of Salem, 1712-1783, When Salem Sailed the Seven Seas--in the 1790's, What Constitutes a Real Librarian, East India Voyages of Salem Vessels Before 1800, The Voyage of the Margaret in 1801, the First Salem Voyage to Japan, Salem in the Nineties, Town House Square in the Nineties, and Some of the People Who Crossed it, Famous Salem Entertainments of the Nineties, Salem in the Nineties and Some of the People Who Lived There, Chestnut Street 40 Years Ago: and the People Who Lived in and Around it, The Duncans of Londonderry, N.H.: the Children of George the Emigrant, The Stephen Phillipses, Folks in Haverhill in 1783, and Part of Salem in 1700.

Phillips was a member of the Union Club, the Harvard Clubs of Boston and New York, the University Club of Chicago, the Eastern Yacht Club, and was secretary and vice-president of the associated Harvard Clubs and president of the North Shore Harvard Club. He was a trustee of the Governor Dummer Academy and of the Essex Institute in Salem. He was also a member of the American Antiquarian Society, the Colonial Society of Massachusetts, and the New England Historic Genealogical Society, among others. Phillips died in October 1954.