Sorry about that. Here are the entries. By 1861, Charles is a butcher!

Town of Whitby

Scott, Charles, butcher, b. Lower Canada, W. Methodist, aged 45, married
Scott, Mary, b. Upper Canada, aged 44, married
Scott, Stanley, student, b. UC, aged 22
Scott, Jerome, student, b. UC, aged 20
Scott, Mary E., student, b. UC, aged 18
Scott, Charles Jr., student, b. UC, aged 16
Scott, Francis A. (female), student, b. UC, aged 13
Scott, Norman H., b. UC, aged 3

In the previous household, is an Emma Scott, a school teacher, b. Lower Canada, aged 18
(in household of John Hopper, also a school teacher).

1871 Census
Town of Whitby

Scott, Charles, aged 54, b. Quebec, W. Methodist, Irish, butcher, married
Scott, Mary, aged 53, b. Ontario, W. Methodist, U. States, married
Scott, Mary Ellen, aged 28, b. Ontario
Scott, Charles, aged 24, b. Ontario, butcher
Scott, Francis, aged 21, b. Ontario
Scott, Norman H., aged 12, b. Ontario

Also in Town of Whitby:

Scott, Jerome, aged 29, b. Ontario, W. Methodist, Irish, butcher, married (in May 1870)
Scott, Elizabeth, aged 24, b. England, W. Methodist, English, married

Unfortunately, no sign of John and family.