When pondering this question you might think it's a silly question, but do ponder further as this question is actually a good question believe it or not.
It's not like we try and remember. In a way, it just comes to us. When we think about it, though, it seems quite strange that our psychological “arrow of time” moves with the same force as that dictated by our entropic universe: which in a nutshell says things get more messed up over time. There are more ways a system can be disordered rather than ordered, hence why glass breaks into multiple pieces.
The laws of physics are all symmetrical in time. Classical mechanics, in which particles and objects obey Newton's laws, and General Relativity, which describes gravitation by describing how mass warps space-time, suggest that time is symmetric. According to these theories, the past and future are equally knowable: both can be calculated from the knowledge of the positions and velocities of particles at a given moment.
Here, we'll be summoning an old philosophy of time called Eternilsm but instead of philosophy nowadays it is known as four dimensionlism and the block universe theory. This uses General Relativity which has been proven time and time, so this radical view of time should be pondered seriously.
The arrow of time is usually explained in terms of the Universal Second Law of Thermodynamics, where entropy is a measure of the disorder or randomness in an isolated system, i.e., a physical system in isolation from external influences doesn’t have entropy because there are no interactions between elements. So if you want to figure out why you'd have to consider more than just broken glass, for example; it's about better understanding that entropy arises from energy not flowing into usable work in an isolated system. Making it harder for you to predict what will happen next - something that's needed for reasoning within the context of time. So, the psychological arrow of time aligns with the thermodynamic one.
But Leonard Mlodinow of the California Institute of Technology in Pasadena and Todd Brun of the University of Southern California say that this statement is not entirely accurate. In theory, you can remove any need for erasure or initialization since you could remember everything—which means that recording information in the memory would be reversible entirely in time. However, even in that case, the arrows of time must align because there is a broader principle at work.
The researchers argue that this extra ingredient is something they call generality. They illustrate the argument with a rotating turnstile that records the passage of gas molecules from one chamber to another. The system starts with most of the molecules in the left-hand chamber, and at any instant, the rotor reveals the net number that has passed from left to right since some earlier reference time.
However, since the laws of thermodynamics follow predictable and irreversible Newtonian principles, the readout could also be interpreted as showing the number of molecules that will pass between the time of the reading and some future reference time. One can show that this would be a correct guess since it allows for the possibility that that number can in principle be calculated. Brun tentatively asks “Why can’t we call that a memory of the future?” asks Brun.
The reason we cannot is that for the rotor to work as a memory of the past, the system’s state at an earlier reference time need not be exquisitely specified; any slight changes in the molecules’ positions at that time will not affect the readout at a later time. But equivalent slight changes in the state of thermal equilibrium elicited by the presence or action of an external influence leads to inconsistencies.
To see this, recall that the molecules started mostly in the left-hand chamber and are gradually equalizing their numbers on both sides of the rotor. Imagine “running the movie backwards” (according to Newtonian equations) from the future reference time to the readout time and seeing the molecules collectively move back toward the left-hand chamber. That extremely improbable event can only occur from one very specific arrangement of the molecules at the future time. If, before running time backward, you made any small changes, say, in the molecules’ positions, new collisions would occur during the time reversal that would rapidly set the system on a completely different course. The molecules would take the much more probable path of equalizing the populations and would not get close to the original state of the system at the readout time.
Mlodinow and Brun argue this cannot be considered memory, because it lacks generality. A memory only becomes a memory when it reflects the system's past state as well as future events whether or not they're prompted by environmental cues: It indicates a future state, but its information is specific depending on the circumstances of the system's behaviour.
“They have emphasized a very important problem in the meaning-of-time debate and provided an interesting solution,” says Lorenzo Maccone of the University of Pavia in Italy, who has previously considered the origin of the thermodynamic arrow of time in quantum physics"
However, to throw a spanner into our minds, the block universe theory describes 'now' as an arbitrary place in time, and states that the past, future, and present all exist simultaneously. Much in the same way that your current location doesn't exclude the existence of other locations, the block universe theory claims that being in the present doesn't mean the past and future aren't currently taking place. You can imagine this spacetime with a loaf of bread and each slice is our forward flow psychological perception of time but each slice already exists there like spacial dimensions. The past doesn't just go into thin air.
Of course, this is opposed to Presentism, which states that the past doesn't exist anymore and is constantly disappearing, thanks to that pesky notion of 'present' time. But this is nothing more than an illusion created in our brains. If you're watching a movie, we know that the ending already exists, spacetime is the same.
Could time travel be possible?
According to Dr Miller, hypothetically speaking, yes, it is possible. But there is one big caveat. We would have to figure out how to travel at a speed close to the speed of light, allowing us to use wormholes as a shortcut to travel into another "location" in spacetime. This would be possible due to a phenomenon known as time dilation. However, if we were to be able to create the technology to allow us to travel in time, we would not be able to affect our present by changing the past, Miller says. That's because the present exists at the same time as the past and is, therefore, inextricably linked to the past. No need to worry then that killing an insect in the past would lead to a snowballing chain of events that would set off another world war. "If I travel to the past, I am part of the past. Importantly, I was always part of the past," Miller says. In other words, going to the past would mean that we are simply fulfilling pre-ordained actions that are already written out in the block that is spacetime.
In opposition to the block universe, Big Think stated in a 2015 article that there's also criticism of the block universe theory, namely from physicist Lee Smolin. "The future is not now real and there can be no definite facts of the matter about the future," he said on numerous occasions. The article goes on to say that if everything is preordained the individual has no agency over their actions which can lead to fatalism or letting life just sort itself out without any input from us.
But, Space Ponder will say, just because freewill l may be an illusion, doesn't undermine our feeling of choice, it certainly does feel real. It's an illusion but it is a very real feeling illusion.
Keep pondering. :)