I think when we talk about the heritage of imperialism it's important to distinguish between "responsibility" and fault/blame. I am not to blame for the actions of my English ancestors/distant kin. I was naught but a twinkle in my great^N-grandfather's eye when the English conquered India.
But I am responsible, in the sense that I have to choose how I relate to that history and what I do to set right those wrongs.