There are at least 3 possibilities here:
Possibility A: If it (he) were (or was) my brother, he would tell you off.
Possibility B: If it had been my brother, he would have told you off
Possibility C: If it (he) were (or was) my brother, he would have told you off.
Possibility A is a type 2 or “second” conditional sentence expressing a hypothetical or unreal condition. Since this “it” (in really) is not your brother, “you” did not get the rebuke or castigation. Possibility B is type 3 (third or “past”) conditional sentence expressing an unreal or imaginary past action. Possibility C is a type 4 or “mixed” (mixing type 2 & 3) conditional sentence. The main clause “he would have told you off” is a type 3 structure while the structure of the subordinate clause “If it (he) were (or was) my brother” is of type 2. The if-clause expresses a general fact (“He” is not my brother); the main clause describes a past consequence (“You” did not get the rebuke).