So, this week’s topic is a big one.  It touches on how our communications systems work, why statistical mechanics makes sense, and even the ultimate fate of the universe.  The topic is entropy.  When I use the word “entropy”,  those of you out there who know of the term will have very different ideas of what I am talking about depending on what your field is.  All of these meanings, however, directly relate to each other.  So, in order to understand the relationship between the three types of entropy and how they relate to the physical world, I must first explain a few background items.

Information, Counting, and Microstates


A 2 particle, 4 microstate system

A 2 particle, 4 microstate system


When we look at a physical system, it is often composed of many little parts, each able to move through their configuration spaces along any of their
degrees of freedom. The configuration space is simply the collection of ways to describe the outcome of an event (For example, flipping a coin has a configuration space of 3 points: heads, tails, and edge).   Each point in this configuration space is called a microstate, and the collection of all of the parts and their microstates is called the system’s macrostate.  An example would be a pair of point particles, each capable of existing in one of two positions along a one-dimensional space.  Each of the two possible positions for each particle is a microstate (giving us a total of 4 microstates for the system).  The whole configuration of the two particles is the macrostate of the system. Note that we obtain the total number of microstates by multiplying the set of states for each particle, not adding them. I guess I should have used more than 2 particles with 2 states in my example.😉

Boltzmann Entropy

So, if we want to take a look at “microstatiness” of a system, what we want is the entropy. In this physical case, we are looking at the “Boltzmann Entropy”.  This value tells us something about the number of microstates, and the probability of falling into any particular microstate.  If we take the number of positions in the system’s configuration space Ω, then the total entropy of a system S is (apologies for the math)

S = kBln(Ω)
(Where kB is Boltzmann’s constant)

This is cool, because it allows us to add entropies, rather than multiply them when we combine two systems, because ln(AB) = ln(A)+ln(B).  Thus, we can look at any system composed of parts able to exist in different microstates, and come up with a term to describe the freedom of the system itself to jump around from microstate to microstate.  The entropy is a kind of window into the black box of the system.

Von Neumann Entropy

If you extend the Boltzmann entropy to quantum systems, you get Von Neumann Entropy.  This encompasses the ability of quantum systems to become entangled, and thus reduce the ability for the system to flow from one microstate to another.  This type of entropy really isn’t much different, conceptually, from Boltzmann entropy, other than its inclusion of quantum features.

Shannon Entropy

The computer scientists out there will be thinking of Shannon entropy, named after the former Saturday Night Live comedian Molly Shannon. (Actually named after Claude Shannon, brilliant dude working for the brilliant-dude factory that was Bell Labs.  Also the inventor of this.)  Claude Shannon wrote a paper describing a formal meaning for “information” in his seminal paper, “A Mathematical Theory of Communication“.  I highly recommend that paper, it is pretty accessible even to laymen (I checked it out from the University Library in my first year).  Shannon entropy is the answer to questions like “How much information is in an English alphabet letter?” or “How much information can I cram on to a carrier signal?”.  To describe this, Shannon coined the term “bit”, the quantum of information.  The bit is the amount of information contained in 1 binary digit:  it is the amount of information contained in the answer to a True/False or Yes/No question.  And any other amount of information, whether a sonnet or a JPEG or a DVD-encoded movie, can be quantified as a number of bits.  This is also related back to the Boltzmann entropy: a system with more microstates can “encode” more information: the more microstates, the more distinct configurations, and thus the more information that can be stored. A coin, for example, if we remove the occasional “edge” result, encodes a single bit of information.  A coin with two heads encodes zero bits (as telling you this kind of coin is “heads up” is always true.  Tautologies are void of information.) You can also visualize entropy as uncertainty (as you have no uncertainty with a two-headed coin, you have zero entropy.  A normal coin has 2 possible results, so you have 1 bit of uncertainty as to which result will occur.)  These bits are the same as the ones you hear about in computer science (for example, a DVD stores 4.7GB, which translates to 37.6 billion bits.  This means all the information in a DVD could be stored in 37.6 billion coins, some tails-up, some heads-up.  Don’t pirate that sequence of coin flips!)

Ain’t that cool?

So, now you know about the nifty-cool interplay of the thermodynamic concept of entropy as disorganization, and the arrow of time (more universal entropy = THE FUTURE!), and information itself!  Thermodynamic entropy IS INFORMATION!  That is pretty amazing if you ask me.  This gives rise to a whole fascinating array of questions and fields to study, from the black hole information paradox (where do microstates go when they fall into a singularity?) to quantum computing (using wavefunctions as logic gates and other computer components) to data compression (removing extraneous elements from data to get its bit-length closer to the actual entropy).