By that logic we should also consider the case of cutting the number of layers in half because that would also reduce hidden state between token generation.
In your generalized example I think the concern is when the additional evaluation effectively becomes a replacement for CoT, where something like the coconut research could replace it completely.
However, I don’t think we’re anywhere close to that with Astra.
No. It’s not at all by definition hidden reasoning.
Looping transformers uses additional calculations (repeating layers) to generate a token.
Reasoning (in this context) is test time generation of multiple tokens that allow a model to have a scratch pad to refine its thoughts, chain of thought reasoning in other words.
Doing the former in no way means that you have to hide the latter.
Raschka is right in this post, The Information article was wrong. The Astra system card does concede reasoning traces are sometimes smaller, but this could be for a lot of reasons, including simple efficiency. And it absolutely doesn’t mean they are going away or completely obscured.
The Last Week in AI podcast from Sept 8 seems to have gotten this wrong as well. Jeremie Harris rages that OpenAI implemented latent reasoning, ala the coconut paper, which could potentially actually obscure reasoning traces. But for the life of me, I do not know how he arrived at this conclusion and see no evidence that this has happened in Astra.
It can lead to hidden reasoning, if the looping allows it to stuff enough information outside visible CoT. Open AI demostrates such an ability by asking it to solve problems while thinking about something else entirely. All the other models are unable to do this except Astra. It doesn't have to be a substitute for CoT to cause monitorability issues.
There's latent space thinking inside the model and then there's the thinking chain of thought words you see the model output. Of course the former is still happening even when you say 'don't think about it' but the latter can be controlled a great deal better with Astra.
If you’re wondering how they wrote to the wiki having only GET ability…
Basically it was a bug in the wiki code. They transferred the POST form parameters to GET URL parameters, and wiki internally doesn’t distinguish between the two.
The 1st edition CouchDB book from 2010 explained it like this:
> We write software to improve our lives and the lives of others. Usually this involves taking some mundane information—such as contacts, invoices, or receipts—and manipulating it using a computer application. CouchDB is a great fit for common applications like this because it embraces the natural idea of evolving, self-contained documents as the very core of its data model.
> Self-Contained Data
> An invoice contains all the pertinent information about a single transaction—the seller, the buyer, the date, and a list of the items or services sold. As shown in Figure 1, “Self-contained documents”, there’s no abstract reference on this piece of paper that points to some other piece of paper with the seller’s name and address. Accountants appreciate the simplicity of having everything in one place. And given the choice, programmers appreciate that, too.
> Yet using references is exactly how we model our data in a relational database! Each invoice is stored in a table as a row that refers to other rows in other tables—one row for seller information, one for the buyer, one row for each item billed, and more rows still to describe the item details, manufacturer details, and so on and so forth.
Iow, a document database stands in contrast to a relational db in that these JSON things we store in them are more stand-alone “documents” compared to storing data in rows and columns in a relational db like PostgreSQL or SQLite.
The thing that people always miss about this takeaway is that while it is a truism, most data is actually inherently relational. Even in your example given, the individual components that are made to assemble that document are better represented as relational datastores
The "relational" in relational databases is not about department number in employees referencing departments.
What the model calls relations are sets of n-tuples where each attribute value has a domain.
SQL databases call their version of relations tables, and, in that view, a database with a single table is still relational.
Now I don't think SQL databases are relational but the analogy still holds (I'd just say that a relational database can have just a single relation)
If you meant it this way and I misunderstood you, apologies, though I'm not sure I'd say most data is actually inherently relational (even though I do think the relational model is the best one we have so far for databases).
However, if you meant that most data has relationships (as the ones we enforce with foreign keys in sql databases) then I agree with you, and I think using database management systems that don't have good support for representing this type of relationships between data entities will only work in niche cases and will eventually cause more trouble than benefits in general-purpose use cases.
Here is a realisation from my recent work: it is not if the data is relational - because if you need you can always push the data into relations - just like you can push it into hierarchies if all you have is files and directories. It is about how much churn there is in the relations. You can model relations with just links or something and if there is not much churn you can keep it in git and it will work OK - the problem starts when your relations start churning (like when you have reviews that need to be updated on both review instruction change and document change) this is when you see yourself start building a relational db (badly).
In the real world of business, generally you want to store these pieces of information together to establish a historical record, not have references or links to other tables, etc. Links and relations are fragile, as anyone who's clicked through to a 404 can attest. Standalone documents last as long as the media that stores them.
Nothing here says the data isn't relational. It strongly disagrees, with reasons, why it's not better represented as relational.
Personally I prefer the relational stance, and there are a lot of people who don't get it who say things like "this data isn't relational", but that's not the argument GP made.
I would argue most data is inherently(naively?) hierarchical(the document), relational structured data is a clever but unintuitive mechanism to introduce powerful analytic opportunities to a set of data.
Basically a document is a report, a large disjoint volume of information on a subject, I consider this the natural form of data because this is how it is collected and how most people think about it. relational is sort of like storing that data as vertical slices through your stack of reports. Not natural at all but much nicer for analysis across the data set.
It's a MongoDBism. The MongoDB community used document to mean the nonrelational equivalent of a row in a relational database. But over time there was definitional shift, and now it means a JSON blob, even if that blob is in a relational database.
The "document" is sort of the native data type, Put everything in a big hierarchical structure, It is very flexible but analytics across the set can suffer. "relational" is another way to store data, break your big hierarchy into sets of related rows and store the rows as a table, If I were to describe it in geometric terms where the document is a report on a paper, the relation is a vertical slice through a stack of those reports. This is slightly non-intuitive but provides for interesting analysis opportunities.
But nothing prevents you from treating your relational database as a document database, set it up as a key-value store where each key is is the document title and each value is a large blob of document data. If your documents are fairly consistent it is also easy enough to build indexes and query features to regain some of the analytical ability of relational data.
1) The internal or wire representation of data from document DBs isn't necessarily JSON, though it's normally converted to such.
2) "Document" has undergone a bit of semantic drift thanks to HTML and XML. In an informational context it means "structured, hierarchical unit of data containing mostly text". The data from forms, invoices, and the like needs to be collected and stored, even if it isn't properly normalized and relationalized (or is en route to being such) so a "document database" is thought to be suited to this task
I dunno, whatever, I'm in the "just fucking use postgres until you can justify why you shouldn't" camp.
I think it originates with the early web. JSON being a replacement for XML, and "document" being the general description for the response to a web server.
So, this is not supported by the data when people are asked.
Sexual effects alone hit about 50–70% people. Then you could face nausea, insomnia, profuse sweating when you’re still, emotional blunting, and weight gain.
I don’t want to discourage anyone, they can save lives. It is a net benefit for some people, but even then most people will experience side effects and wish they didn’t have to take them.
> The most common side effects reported by patients in the study by Hu et al1 of 401 outpatients taking SSRIs were drowsiness (38%), dry mouth (34%), and sexual dysfunction (34%)
> which was conducted in 1999-2000, questioned patients about side effects via phone interviews within 75 to 105 days of starting antidepressant therapy. The side effects deemed most bothersome were sexual dysfunction (17%)
Just because one study has the number 34% in it, does not prove the numbers I gave were wrong and does not mean they are anecdotal.
This is something that’s been studied from many angles and there’s more than one number that’s defensible based on the context and dataset.
For example, even the study you cited says:
“In a study of 344 patients by Montejo-Gonzalez et al, 58% of patients reported sexual dysfunction when physicians directly inquired”, and “14% with spontaneous reporting”.
Do you think the one 25 year old short-term study that you found by yourself is the only study that has been done, and that anything that deviates from it is a distortion?
That’s only true regarding the one sentence about the singularity not being a point.
People like to reduce papers to a simple hot take, but the paper is more than that, offers viewpoints that are non-standard and speculation about new possibilities.
Why would you bring up fraud in the midst of science research in the US being burned to the ground?
Fraud is not the reason it is happening. Even the people who are making the cuts in this case have clarified the purported reason and it has nothing to do with fraud.
There are too may bad scientists, and I don't want to fund them. When the sciences became status-oriented, group-think and profit-seeking became the norm.
This administration is trash, but I'm still tired of paying taxes to fund jokers.
If your 'science' is so important, then any other country in the world will be happy to scoop you up. gtfo
Just one example out of many: Today, I looked up research on EMDR treatments. Major problems in replication for things outside of PTSD. Yet, therapists are told by their boards to push the treatment for all sorts of things.
Have many of you actually looked at a lot of the garbage funded by NIH?
I mean, I otherwise agree with the sentiment of your post but I feel like most people are rating him pretty well.
reply