[Note: don’t miss this afternoon’s webinar on the MIM, 2PM ET/1PM CT.] As our fall outlook series draws to a close, I thought I’d ask a data scientist who helped develop the MIM, the MrTopStep Imbalance Meter. His answers were fascinating. This is a revolutionary technology we believe will change how we see the markets.[pullquote]Over the last couple of years rapid advances in technology have facilitated an explosion in the capture and analysis of unstructured data.[/pullquote]
First I asked him to explain what the MIM is. We’ve said many times, it’s not an indicator, not a green for buy and red for sell “system.” His answer was to the point: “The MIM aggregates imbalance data on NYSE listed stocks prior to the 3:45 PM EST dissemination of market-on-close imbalances.”
Well, that is exactly what it does. The MOC goes public at 3:45 PM. But the people making those orders often know what they’re going to do long before then. And in various ways, they let others know. If you were on the floor and knew the right people you might hear what people were thinking about doing in the last hour of trading, you might know some of that inside “data.”
That’s the sort of information our sources on the CME floor are privy to, and every day we hear about it person-to-person. But how do you find out what an entire market is planning? The MIM does that kind of aggregation, which is a fancy word for a whole lot of intelligent listening.
What does the MIM listen to? In my previous conversations with the engineers, they used the phrase “unstructured data.” I asked what unstructured data is and whether it exists in all markets. The answer:
Over the last couple of years rapid advances in technology have facilitated an explosion in the capture and analysis of unstructured data. That is, unstructured data are [not arranged] in any organized or efficient manner.
Our MIM feed structures unorganized imbalance data using a voice recognition technology that pushes data to the cloud in an efficient, accurate manner. After structuring, cleaning and storing over a year’s worth of pre 15:45pm imbalance data, deep statistical analysis yielded a significant correlation between the imbalance data and the market direction into the close of trading.
Imagine being in the middle of a large room full of hundreds of people and listening to all their conversations and in a short time being able to predict how many of them would take a vacation in the next three months, or how many were millionaires, or how many would order the fish at dinner. Unstructured data means just that, data coming in a large stream without any organization into categories or ranks of importance. The engineers had to come up with new algorithms just to make sense of the data they were gathering, just to turn it from a noise into a signal.
And once they did that, they found that the signal gave clear, often eerily accurate predictions of a major market move in the last minutes of trading. That is what we’ve been bringing to the subscribers to the MrTopStep Trading Room (14-day free trial). It’s available to not just big hedge funds and institutions, but individuals.
I then asked if the MIM could give a longer-term view of, say, the rest of 2013, or if it only worked on the closing imbalance of one day. His answer:
The data pushed out by MIM is correlated to market direction post 15:45. There may be many different insights that can be drawn from different analyzes but MIM is not intended to make macro predictions about the future direction of markets. A full historical data set is available and researchers are welcome to conduct additional research to extract unique insights from the data over different time frames.
In other words, we never asked that question, but the data and the algorithms are there for anyone who wants to. How cool is that? So why hasn’t anyone asked the question yet? His last two answers address that in a pithy and humorous way.
I asked, “If you were in the market right now, would you be bullish, bearish, or neither?”
His answer showed the kind of focus it takes to be a good engineer—or a good trader, for that matter: “I don’t think about such questions.”
I ended with this question. His answer hints at a larger problem in our financial system as a whole: “What has been the most surprising discovery in your process of developing the MIM?”
His answer went right to what I think may be the biggest insight of all: “That no one had efficiently analyzed the pre-15:45 imbalance data before.”
A lot of times in trading and in life, we spend enormous effort looking for the answer to a question like, “What indicator would help me identify tops and bottoms?” When the question we really should be asking is, “Is another indicator what I really need?”
We get some people asking if the MIM is a good indicator or what percentage of “winners” it gives you. That’s what all those ads have drilled into us, that the right question is about finding the “best” indicator or “system,” the one with 80% winners. You know, the one the Wall Street insiders don’t want you to know about (exclamation point).
I’ll tell you something I’m sure is right 80% of the time: if you are stuck on the idea that an indicator is what you need, what you really need is to meditate for 10 minutes a day, get some exercise, drink more water, and keep a trading journal.
Understand what the MIM really is: not an indicator, but a whole new way of seeing data. Even data that seems like noise until you find a better way of listening to it. It can tell you with uncanny accuracy about a major move in the last half hour of trading. It may not be the only move. The market may end that move and return to the previous trend before it’s over. So just blindly placing a buy order because the MIM shows green and not planning your exit is like walking around all day with your umbrella open because the forecast says rain. Timing is everything.
But you can time with much greater precision if you have data like the MIM to guide your decisions. The MIM won’t tell you when or whether to buy or sell.
[pullquote]Let me say that again: The MIM will not tell you when or whether to buy or sell. [/pullquote]But it will tell you what orders the majority of traders are planning to place. You can choose to join them, stand aside, or exit just as they enter. All are good trading decisions, but they are your decisions. No machine can decide for you, not even one as sophisticated and revolutionary as the MIM.
What did you learn from one of the creators of this technology? For me, the biggest lesson comes from the question he refused to answer, when he replied, “I don’t think about such questions.” To be good or even great at something, you have to know what to focus on and also what not to be distracted by. You have to know what questions you won’t think about.
Having said that, we welcome all your questions about the MIM at this afternoon’s webinar at 2PM ET/1PM CT. Hosted by Danny Riley and TheStreet.com’s Jill Malandrino, it is the first time we are revealing the details of the MIM publicly, during the trading session. After the webinar we’ll continue to show the MIM so you can watch it in action through the close of the trading day. You won’t want to miss it.