It is software, of course you can (with access to source code and its tools). Car will record all of its sensor data, and we'll be able to analyze its decisions.

You can't pop the skull bone of a human driver and peek into their brain either.

The way to determine the guilt of a driver, human or electronic, is through their actions.

I mean, we were able to prove that the Uber car killed someone through reckless engineering because of its black box. It was obvious that the car saw the person and then deliberately ignored them, and the stock autonomous breaking system has been disabled.

Fault in driving accidents is well established, by law. No need to look at the algorithm at all - if the car was at fault under law then it's done.