What you have said above is correct, however it is not the "main" concern. There are numerous concerns. That is one key one. Since I have not seen others post other key concerns, I will also contribute an additional concern:

Sensitive data left lying around are/is very tempting to use for illegal economic and blackmail gains. The FBI in particular has a pattern of getting indicted for re-selling sensitive data, even before massive amounts of "seize now, convict later" was taking place.