Islam has always kept aims and objectives in mind. Because the intent is to protect the chastity and honor of women dignity, checks have been imposed on every quarter from where there is a danger to the dignity and chastity of women even though it may be in the home itself. A brother-in-law who is the brother of husband may turn out to be dangerous to the chastity of the lady. That's why the Holy Prophet (Sallallahu alaihi wa sallam) had declared him as death for sister-in-law. This is because for a modest and chaste lady, any attack on her chastity is death itself.
Hence, there is a Hadith in Sahih Bukhari and Sahih Muslim:
Translation of Hadith: It has been narrated on the authority of Hadhrat ‘Uqba bin Aamir (May Allah be well pleased with him) that the Holy Prophet (Sallallahu alaihi wa sallam) said: Avoid visiting women. A Sahabi asked: O Prophet of Allah (Sallallahu alaihi wa sallam), what do you say about the husband’s brother? The Holy Prophet (Sallallahu alaihi wa sallam) said: The husband’s brother is death. (Sahih Bukhari, Hadith No. 4831)
The meaning of this is that the way everybody wants to save oneself from death and if there is a life-threatening situation, then everybody would prefer saving one’s life, in the same way in spite of there being a relationship, one should prefer practicing Hijab with the husband’s brother as well.