James Boyce Montgomery was a railroad contractor who grew wealthy through construction projects in the eastern and western U.S. Born near Harrisburg, Penn., in 1832, he worked first as a journalist, eventually becoming editor and publisher of the Pittsburgh . He moved on to railroad construction in the late 1850s and built bridges and rail lines in Pennsylvania and vicinity in the following decade. He served on the boards of many railroad companies and purchased a portion of the Baltimore and Potomac Railroad. He moved to Portland, Oregon, in 1870 and constructed portions of the Northern Pacific Railroad, travelling to Europe to raise capital and obtain materials. In addition he organized other enterprises, such as the dredging of the Columbia River channel and construction of docks in the Portland area. He was also an influential figure in Republican party circles, although his only major public office was that of Multnomah County representative to the Oregon legislature, to which he was elected in 1890.

In 1861 Montgomery married Rachael Anthony and the couple had one son, Henry Moorhead Montgomery (1863-1932). After Rachael's death in 1863, James Montgomery married Mary S. Phelps in 1866. She came from a prominent Missouri family and was the daughter of John Smith Phelps, who served as the governor of the state from 1877 to 1886. Mary Phelps Montgomery (1846-1943) led a strenuous and long life filled with travels, social activities, and active club work. She served as hostess for her father during his gubernatorial term in Missouri; lived in Berlin in the late 1880s and at later periods; served on the Board of Lady Managers for the 1904 Louisiana Purchase Exposition in St. Louis; organized the Portland chapters of the Daughters of the American Revolution and Colonial Dames of America; travelled extensively; befriended famous people, such as President Ulysses S. Grant; and managed her husband's financial affairs after his death in 1900. Among her seven children were: Mary Montgomery Talbot, Antoinette Montgomery Frissel (b. 1871), Phelps Montgomery (b. 1872), Eliza Montgomery (b. 1874), Constance Montgomery Burrell (b. 1875), Russell Montgomery (1877-1900); and Margaret Montgomery Zogbaum (b. 1887).

Margaret Montgomery Zogbaum pursued a literary career and wrote essays, plays and novels. She lived much of her youth in Europe and had a brief love affair around 1912 with an Italian nobleman, Adolfo de Bosis. In 1914 she married a U.S. naval officer, Rufus Fairchild Zogbaum II, who was stationed in London and later became the commandant of the U.S. Naval Station in Pensacola, Florida. The two travelled throughout Europe and the Middle East and settled eventually near Tryon, North Carolina, in the late 1930s. Among their children were Wilfrid Zogbaum (1915-1965), an artist and musician, David Zogbaum, and Rufus Fairchild Zogbaum III (called "Fair").