In addition, he stated that whereas massive organizations resembling Uber and Walmart have put in and are utilizing lakehouse choices, mainstream enterprises, to a big extent, haven’t but moved to them, as a result of “today it requires building a via a do-it-yourself approach where you build your own, you cobble together a bunch of open source tools. If you have a deep engineering bench, you can do that. If you don’t have that deep engineering bench, that becomes very difficult.”
Kyle Weller, VP product at Onehouse, added that organizations presently face two challenges: “[They have] chosen a Databricks or Snowflake, and that dictates the rest of their architectural choices, or they are in a situation where they’re looking to open source, but that complexity of self managing is preventative from exploring multiple engines.”
Each engine has a novel specialty, he stated, noting, “Flink was not invented for no reason. Flink was invented to address real time stream processing. Ray wasn’t invented just to be another item on the shelf. Ray was invented special purpose for AI use cases, ML use cases, data science.”