I don't know if this helps or not but in the 1859 City Directory for the City of Milwaukee, I found Amalius Keller (page 124) and George Keller (page 125). Here is how they are listed.
Keller, Amalius, carpenter, Market bet Knapp and Ogden
Keller, George, cabinet maker, Market bet Knapp and Ogden
If these Kellers are yours, then it appears that they lived together on Market Street in between Knapp and Ogden. Market street is a very short street and one side of the city hall is on Market Street. There aren't any homes in the area anymore though.
I also found a couple of more Kellers. They are 2 men named August Keller (a laborer and a clerk), Mrs. Josephine Keller (a washerwoman) and Martin Keller (a teamster). They all have different address'.
I hope this information is useful to you and I hope this helps.