I am opening a child window using the below code:

<!-- language: lang-js -->
window.showModalDialog("FileUpload.aspx", "FileUpload", 
    "center:yes;resizeable=yes;dialogHeight:300px;dialogWidth:600px;");

I save the file uploaded in the child window(FileUpload.aspx)in its code behind FileUpload.vb page. Since server side code, its postbacking and opening a new browser .

After my functionality in the child window, when I close it using below code,

window.open('', '_self', '');
window.close();

it is closing the new browser opened because of postback but a copy of the same child window is still open when returning to the parent page.

I want to close all the instances of this child window.

The showModalDialog method will freeze JavaScript execution on the parent window until the dialog that it opens has closed, so we can rule out that as the reason why your second window opens. When FileUpload.aspx posts back from the server, it should work the same way as modeless aspx postbacks.

I think you should be able to get rid of the window.open() method and you should be fine.

Parent window:

window.showModalDialog("FileUpload.aspx", "FileUpload", 
    "center:yes;resizeable=yes;dialogHeight:300px;dialogWidth:600px;");

The Child window will call this when it's done.

window.close();