December 13, 2022
Once upon a time scientists would dream of the day when they could have enough information to make decisions based on data. Young readers may have to go to history books to see computer science majors taking stacks of punch cards to a computer room so they can get an answer in the morning.
Fast forward to 2022, we have so much data we don’t know how to handle it. The overview is simple – gather up a reasonable number of data sets and pour it through an algorithm and then out pops the answer. What could possibly go wrong?
For example, back in 2017, it was reported that the DoD collected 22 terabytes of data a day. You would have to add many zeros to that number to see what they are collecting today.
As a result, people with a doctorate in mathematics, like Dr. Elsa Schaefer from LinQuest, work alongside developers to wrestle with questions about what data to gather to make valid decisions.
"And the idea of MLOps is to walk an engineer from importing your data through deploying your final model, and helping that engineer detect any little blips"
During the interview, she used terms like Data Wrangling, Machine Language Operations (MLOps), and data brittleness. It appears that there is as much an art as it is a science to competently gather data for decisions to be made. The term “brittle” is intriguing.
Let’s say you have an application with a large data set that is working well. It is quite possible that a developer could pour that data into a data set, and it may cause problems. Because it may cause a system to break, it is called “brittle.”
LinQuest is developing a platform to help federal leaders gain a better understanding of using machine data to eliminate expensive data management errors. If you would like to pursue this topic further, you may want to download an overview of this platform, curiously called HAL, or Harness for Adaptive Learning.
If you enjoyed this article, you may want to listen to Episode #19 More Devices than People: Managing the Madness
John Gilroy
Has been in front of a microphone since 1991. He can help you structure, launch, and promote your company podcast. johngilroy@theoakmontgroupllc.com
Leave a Reply