Open Table Formats (Iceberg, Delta Lake, Hudi, etc) are the approach we've taken. That let's us offer a query engine but also let's the tenant bring their preferred engine (Snowflake, Spark, DuckDB, etc). It also addresses dirty reads and some other state problems that come from trying to use the file system. It scales as much as the bucket does, so we haven't found a use case we couldn't scale to yet.
We (https://prequel.co) recently started offering this as a white labeled capability so anyone can offer it without building it yourself. Its a newer capability to our export product where instead of sending the data to the tenant's data warehouse, we enable you to provision an S3/GCS/ABS/etc bucket with the data formatted. Credential management, analytics, etc is all batteries included so you don't have to do that either. The initial interest from our customers was around BI integrations but agent use is starting to pick up which is kinda interesting to see.
what are the properties of the table? is the whole table 2B rows or is that the amount of row updates/inserts that happen per period (day, week, month)? We do handle a lot of large table volumes on both our products already, but there will be cost tradeoffs the larger the data volume and lower the latency you try to target.
Aha! That makes so much sense. Thank you for this.
Edit: Ah, right, the downside is that this is not going to have good olap query performance when interacting directly with the sqlite tables. So still necessary to copy out to duckdb tables (probably in batches) if this matters. Still seems very useful to me though.
Analytics is done in "batches" (daily, weekly) anyways, right?
We know you can't get both, row and column orders at the same time, and that continuously maintaining both means duplication and ensuring you get the worst case from both worlds.
Local, row-wise writing is the way to go for write performance. Column-oriented reads are the way to do analytics at scale. It seems alright to have a sync process that does the order re-arrangement (maybe with extra precomputed statistics, and sharding to allow many workers if necessary) to let queries of now historical data run fast.
It's not just about row versus column. OLAPs are potentially denormalised as well, and sometimes pre-aggregation, such as rolling up by day, by customer.
If you really need to get performance you'll be building a star schema.
Not all olap-like queries are for daily reporting.
I agree that the basic architecture should be row order -> delay -> column order, but the question (in my mind) is balancing the length of that delay with the usefulness of column order queries for a given workload. I seem to keep running into workloads that do inserts very quickly and then batch reads on a slower cadence (either in lockstep with the writes, or concurrently) but not on the extremely slow cadence seen in the typical olap reporting type flow. Essentially, building up state and then querying the results.
I'm not so sure about "continuously maintaining both means duplication and ensuring you get the worst case from both worlds". Maybe you're right, I'm just not so sure. I agree that it's duplicating storage requirements, but is that such a big deal? And I think if fast writes and lookups and fast batch reads are both possible at the cost of storage duplication, that would actually be the best case from both worlds?
I mean, this isn't that different conceptually from the architecture of log-structured merge trees, which have this same kind of "duplication" but for good purpose. (Indeed, rocksdb has been the closest thing to what I want for this workload that I've found; I just think it would be neat if I could use sqlite+duckdb instead, accepting some tradeoffs.)
> the question (in my mind) is balancing the length of that delay with the usefulness of column order queries for a given workload. I seem to keep running into workloads that do inserts very quickly and then batch reads on a slower cadence (either in lockstep with the writes, or concurrently) but not on the extremely slow cadence seen in the typical olap reporting type flow. Essentially, building up state and then querying the results.
I see. Can you come up with row/table watermarks? Say your column store is up-to-date with certain watermark, so any query that requires freshness beyond that will need to snoop into the rows that haven't made it into the columnar store to check for data up to the required query timestamp.
In the past I've dealt with a system that had read-optimised columnar data that was overlaid with fresh write-optimised data and used timestamps to agree on the data that should be visible to the queries. It continuously consolidated data into the read-optimised store instead of having the silly daily job that you might have in the extremely slow cadence reporting job you mention.
You can write such a system, but in reality I've found it hard to justify building a system for continuous updates when a 15min delay isn't the end of the world, but it's doable if you want it.
> I'm not so sure about "continuously maintaining both means duplication and ensuring you get the worst case from both worlds". Maybe you're right, I'm just not so sure. I agree that it's duplicating storage requirements, but is that such a big deal? And I think if fast writes and lookups and fast batch reads are both possible at the cost of storage duplication, that would actually be the best case from both worlds?
I mean that if you want both views in a consistent world, then writes will bring things to a crawl as both, row and column ordered data needs to be updated before the writing lock is released.
Yes! We're definitely talking about the same thing here! Definitely not thinking of consistent writes to both views.
Now that you said this about watermarks, I realize that this is definitely the same idea as streaming systems like flink (which is where I'm familiar with watermarks from), but my use cases are smaller data and I'm looking for lower latency than distributed systems like that. I'm interested in delays that are on the order of double to triple digit milliseconds, rather than 15 minutes. (But also not microseconds.)
I definitely agree that it's difficult to justify building this, which is why I keep looking for a system that already exists :)
Yeah exactly. Wouldn't it be easier to make it self reported but use a mechanism to incentivize accurate market prices. Like enforce that the owner can't sell for more than x% the reported amount and rents are capped based on the reported amount. Use of the property as collateral has to match the reported value. You can only be insured for value up to that reported number. Lots of things the state can do to create that incentive.
I agree with your take here. I have a partner that anyone in this thread would label as a #1 but my two siblings are essentially on the same plane of intimacy. If my partner left my life for whatever reason, I'd still have my siblings and while losing someone is hard it wouldn't destroy me because of that support. Its a fairly normal dynamic in my family. I had two uncles who lived together their whole lives, even through relationships that didn't end up working out even though they had children. My grandmother lived in the same apartment building as her brother for 30 years and her other two siblings lived in the same neighborhood. Her husband (my grandfather) died a few years into their marriage and her siblings (who never got married) always fulfilled the role that a partner would play.
I agree with your premise but ultimately its the responsibility of governments to manage how society distributes what is produced. In the US at least, it seems most of the tension here comes from the fact that we've decided that if you are not doing "work", then you don't "deserve" food, shelter, or healthcare. Trying to stop automation always seems like a fools errand because there is simply too much incentive to automate.
Though I can imagine a counter argument from a technologist/futurist positing that the incentive to automate in part comes from that merciless system of survival.
The modern Luddite, I think, doesn't necessarily "hate" looms, or their inventors. They just don't have any faith that those necessary compensations you describe will ever happen, or at least not quickly enough to save them. Perhaps the government is convinced by exactly that argument above. Or at very least too apathetic (or financially tangled) to fight it.
We all act along the axes where we can affect something, effect something. Smashing looms is a fool's errand, but sometimes that's all the power you feel you have.
> The modern Luddite, I think, doesn't necessarily "hate" looms, or their inventors. They just don't have any faith that those necessary compensations you describe will ever happen, or at least not quickly enough to save them. Perhaps the government is convinced by exactly that argument above. Or at very least too apathetic (or financially tangled) to fight it.
Agree
> We all act along the axes where we can affect something, effect something. Smashing looms is a fool's errand, but sometimes that's all the power you feel you have.
However I think what's fascinating about this is it requires believing you can't change your skill set to something else. Sometimes I don't think its as simple as feeling empowered, but also fear and anxiety about change.
Can we get any of Iceberg/Delta/Hudi that isn't terribly complex to setup? Like configurable completely from Standard SQL via the CREATE EXTERNAL TABLE syntax.
