Hi Julia, which test *exactly* did your brother take? Ancestry.com does indeed do Y-DNA and mtDNA tests but they are hard to find and not as common. The primary AncestryDNA test is autosomal DNA. If he took that one, then his Y-DNA would be unknown. The only tests that do autosomal and Y-DNA (and mtDNA) at the same time are Geno 2.0 and 23andme. For all others, you have to pay separately (though sometimes you can use the same sample).
If you are certain that your brother took a Y-DNA test and he did not get a haplogroup assignment, then there was an error and they should repeat the test for free (with a fresh sample if needed).