Humans don't have single overriding goals that determine everything we do. We have some general goals - stay alive, be happy, be good, reproduce, etc. And there are a lot of other things we do that can be thought of us asu goals towards achieving top level goals.
But really it seems like people just do things. We are messy, sometimes wanting one thing and sometimes another. For lack of a better term humans are more "organic" than programmed.
I am newly diving into AI safety research, and it seems to me a lot of the problem with misaligned AI is at root an issue of the AI having a single overarching goal in the first place. If you have a super powerful intelligence that is set in acheiving one thing, it is like having a super powerful drill that only goes one direction. If the direction you give it is a little off, you have serious consequences because the tool is so powerful that it plows through anything in its way and ends up way of course.
But is there any reason an AI has to have a single goal? Does it seem likely possible to make intelligent AI that has goals more similarly to how people have goals?
And as a side note: I wonder if the organic, non-goal-directed manner in which humans are msde is somehow critical for consciousness. Or maybe even intelligence of a certain kind. It seems that we are not sure enough how consciousness arises that it might be the case that it only arises in "organic" beings.
Curious to hear anyone's thoughts. I'm sure there's existing thinking on this. If anyone can point me towards that I would greatly appreciate!
Thanks a lot! Very clear and helpful
Furthermore, a lion becomes more dangerous as it becomes more intelligent and capable, even if its terminal goal is not "maximize number of wildebeests eaten".