Don't know if this might be helpful, as I've not been able to trace this family in Maine, but here's some information on these Colley's, two of whom were in Nevada County, CA from circa 1853. (The parents of Moses H., James F. & William Henry Colley were also natives of Maine):

In the early 1850s James Colley, who was a native of Maine, was in Nevada County, CA. Later his brother, William Henry Colley, after being discharged from the Civil War, came to this locale also. The two brothers were partners in Colley Bros. Union Meat Market in Nevada City.

James and his first wife, Susan (Weld) had two sons, James F. jr., and Fred. After Susan died, James F. Sr. married (Mrs.) Mary E. Cawley of New Hampshire, and after she died he married Amelia (Gillett) whom I believe had also been previously married.

William Henry Colley was apparently unmarried.

James F. Colley Sr. died in 1910, and his brother William Henry in 1912 both in Nevada County, CA. They also had a brother, Moses, who died in San Francisco County, CA in 1904. Moses had a wife named Elizabeth and a son named Fred A. who had a wife named Louisa (or Mary) and a daughter named Helen. Fred A. & family were in Redwood, Santa Clara Co., CA in 1910.

James F. Colley Jr. remained in Nevada Co., Ca and had a wife named Ida and a son named Ellsworth A. His brother Fred was in Napa, CA. in 1910.