Search posts, tags, users, and pages
Roman very nice writeup!
Understood if you're tired of learning yet-another-ORM at this point :-), but if you want to check out Joist (joist-orm.io), it's an ORM we've written and would appreciate your input/thoughts on it.
To your chagrin, we'd also don't have a robust query builder yet :-) (we defer complex query use cases to Knex for now), but we do consider ActiveRecord to be a big source of inspiration (a lot of our domain objects, validation rules, and lifecycle hooks are inspired by ActiveRecord).
I think in a few cases we've surpassed ActiveRecord in ergonomics (specifically since we build on dataloader, we get N+1 avoidance basically for free; and our reactive rules & derived fields is pretty unique I think), but overall it's still a high bar to meet!
Thanks for reading!
These are my thoughts on Joist after briefly reading through the docs, sorry that not examining it more deeply:
continual, verbatim mapping of the database schema to your object model
Am I getting it right, first user need to create table in any way they want, and then they run your codegen and this lib generates model files?
Some other ORMs are doing the opposite: you're writing models and then they generate migrations.
I simply can't understand why - why not to follow simple and reliable RoR approach, I guess the same as in other languages? Write migration, write model, there are scaffolds for lazy people.
"reactive" validation rules - that's interesting, I remember was using it all the time in Rails, if specific column or columns was changed it will run specific validation.
I suggest to make change docs/readme so it's more obvious for the new comer what is special about Joist. "Schema-driven code generation" - new comer doesn't know what is it, you could write it like "Automatically generates code for models out of your database". As a new comer, I don't really see how it's better than others, I see it's similar to MikroORM and has additional features, something graphql specific, but how is it better for everyday use?
"Guaranteed N+1 safe" well, in the page of docs "Avoiding N+1" the first example you show is NOT N+1 safe. And to see how to do it right reader has to scroll to the very bottom of this page.
Fast tests: let me brag about my approach, I patched node-pg so in each tests gets wrapped in a transaction, and it simply rolled back after each test, no need to clean db at all. And this approach worked for 3 of 6 ORM's in this article. Some time later I'm going to write about this and to publish a lib.
Syntax is similar to MikroORM (which I don't like):
const em = newEntityManager();
const a = await em.load(Author, "a:1");
If you're saying ActiveRecord is an inspiration, why not:
const a = Author.find("a:1")
// or
const a = Author.findBy({ id: 'a:1' })
// or
const a = Author.where({ id: 'a:1' }).take()
book.reviews.get;
Looks a bit weird, conventionally we use verbs for functions, here we have property get.
Would be better to do it without property, I believe this should be possible:
book.reviews
Thanks for the response Roman!
I like your suggestion about making the docs more directly call out Joist's strengths... I'll work on that.
create table first ... codegen generates model files
Yep! Just like RoR auto-adds first_name getters/setters, Joist does the same for firstName/etc. columns (as well as has many/has ones); the difference is that Joist adds it to a base class, so that TypeScript can see the fields & do type-checking.
But net-net is that you get the same "just an empty model" "class Author {}" when starting out (...okay with an "extends AuthorCodegen" snuck in there :-)) as you do with Rails. I just really loved that low-boilerplate aspect of Rails.
reactive ... in Rails
Yep, it's similar, although AFAIU Rails the rules are only reactive on fields within the current model; in Joist, you can have rules that run when fields on related entities change, i.e. re-run an Author rule whenever it gets a new Book, or one of its Books' titles changes.
First example is not N+1 safe
Ah yeah, I think I focused too much on explaining what N+1 was for new comers. That's fair, that page could do a tldr first.
Running tests in transactions ... then roll them back
Ah yeah! That is also what Rails does, afaiu (by default? I think it's configurable iirc), but that means you can't test any code that commits transactions (or maybe you could use subtransactions?). Which for us is GraphQL mutation resolvers. Afaict the flush database approach is ballpark-same-speed and you don't have that same limitation. Another upshot of Joist's approach is that if your test fails, any test data you've setup will still be in your local db to look at/diagnosis if you need to. But I think both are good; as long as the tests are fast! :-)
book.reviews
Yep! That makes sense, but the wrinkle is: is book.reviews a BookReview[] or Promise<BookReview[]>?
Other ORMs like TypeORM afaiu make you choose one or the other up-front, when writing the model. If you choose Promise<BookReview[]>, it is the safest, but annoying to always .load() it to access. If you choose BookReview[], it is really easy to access, but then you risk accidentally calling it w/o it being loaded yet and getting a runtime error.
I agree Joist's approach looks odd, but it's both: a) really safe; if book.reviews has not be explicitly preloaded with a populate hint, you must call reviews.load(); but it's also b) really ergonomic b/c if you have preloaded book.reviews, then some TypeScript magic automatically makes .get visible to you, and you get really pleasant synchronous access.
In a way, this is similar to Prisma, being able to use "kinda like GQL" snippets of a load hint ({ book: { author: { publisher } }) to preload a bunch of data in 1 await call, but with Joist the hints are based on the combination of your db schema + your custom relations & fields (kinda similar to ActiveRecord scopes) instead of strictly being database tables like they are in Prisma.
Author.find / instead of em.find
Yeah, honestly 80% of this is that we started with MikroORM and its EntityManager API, before writing Joist & migrating our codebase to it. But the other 20% is that, unlike Rails (but the same as Mikro and older ORMs like Hibernate), Joist uses a unit of work pattern to delay/batch SQL statements until a "flush" method is called. So we need some sort of "EntityManager" / "Unit Of Work" "thing" to pass around, so :shrug: I think it's net/net not a big difference. You're right that it is not strictly the Active Record pattern, but I think Joist follows the spirit of the Active Record pattern, which is that models extremely closely/strictly match the database tables, vs. the Data Mapper pattern, which is where the models can drift from the schema in ways that seem nice at first but quickly get complex imo.
Stephen Haberman I'm working on own ORM too and just want to share my ideas/vision on those points:
But net-net is that you get the same "just an empty model" "class Author {}" when starting out
Migration is responsible for changing db schema, not only for creating tables, but also changing, renaming, sometimes we do data updates in migration.
And in the model we define current columns, it doesn't take much time, developer can easily jump to User and see the fields. But I decided to go further, let it be possible to define a validation schema right in the model! If you know Zod library, I mean to define columns in similar way like they do.
Something like:
const User = model({
schema: (t) => ({
id: t.serial().primaryKey(),
email: t.text().email(),
})
})
And now it's not only db types, but also validation types.
Later in the controller we can reuse this schema to validate payload:
const createUserSchema = User.schema.pick({
email: true,
name: true,
// ...
})
router.post('/user', (req, res) => {
const data = createUserSchema.parse(req.body)
data // TS knows exactly what the type is
})
but that means you can't test any code that commits transactions (or maybe you could use subtransactions?)
Yes :)
Just published package for this: npmjs.com/package/pg-transactional-tests
There is explanation how it handles nested transactions, previously it was a script I copied from project to project and it helped me a lot even on a production project.
Yep! That makes sense, but the wrinkle is: is book.reviews a BookReview[] or Promise<BookReview[]>?
I'm going to implement nested loads with json_agg, i.e postgres can aggregate data to json arrays by it's own.
As for the interface, .include method should be fine:
const result = await Post.include('comments')
result // is an array of posts
const post = result[0]
post.comments // array of comments
Also it should accept sub queries, so we can load filtered comments:
const result = await Post.include({
comments: Comment.where({ ...conditions }).order({ createdAt: 'DESC' })
})
By the way, is it possible in Joist to load Post with their Comments ordered by comment field or apply some condition?
Regarding whether book.reviews is a BookReview[] or Promise<BookReview[]> - well, I suspect it's tricky to customize result of em.load, but in theory it's possible to return BookReview[] type only if this resource was preloaded, and Promise<BookReview[]> otherwise.
Regarding Unit of Work, I simply don't understand it's benefits, I like the concept of simple structures with fewer magic, when you can see clearly in the code what's going on. Maybe you could suggest an example where it shines?
const user = await User.find(1)
const data: Partial<UserType> = {}
// ...some logic to change user
if (someParam) {
data.foo = 1
data.bar = '...'
}
// and in the end:
await User.update(user, data)
Here is quite typical update flow, without Unit of Work so no need for creating some session, track any states, etc. Even without "ActiveRecord" pattern, User.update is a static method, while user is a plain lightweight JS object.
So in my opinion, Unit of Work adds a bit of boilerplate, overhead, complicates things, or am I wrong?
I don't mean to criticize Joist, just have different preferences and beliefs. So maybe node.js has no decent ORM to use with pleasure (yet?), but we have so huge variety of libs which are mixing completely different ideas, and continuing building new ones, there is something good in it (I guess).
Late reply, but Roman :
it doesn't take much time
We have ~250 tables at this point, and granted we could/would have incrementally typed out each entity by hand, but it's also nice to run codegen and know that the output is 100% what the schema is.
developer can easily jump to User and see the fields
Yeah, I do like that, we have the same albeit capability in the our UserCodegen files; which I think is basically the same thing.
suspect it's tricky to customize result of em.load, but in theory it's possible to return BookReview[] type only if this resource was preloaded, and Promise<BookReview[]> otherwise.
Yes, this type-system trickiness is exactly what Joist implements (if you go find our Loaded type), but the "trick" is "only sometimes adding .get".
If we changed book.reviews to sometimes-array/sometimes-promise-array, then you can't have helper methods like doSomethingWithBook(b1) that accept both books w/previews loaded & books w/o previews loaded, because the reviews field types are too different.
And that pretty quickly gets annoying (we tried something similar to it). So, yeah it "looks weird", but I assert the combination of reviews.load+reviews.get is net/net the most ergonomic solution.
Unit of Work adds a bit of boilerplate
I tried to lay out the benefits in the docs (just updated):
joist-orm.io/docs/features/unit-of-work
email: t.text().email(),
I don't mind that per se; in Joist it would be:
authorConfig.addRule(mustBeEmail("email"))
Which isn't a DSL-pleasant for what you've built-in (like .email()), but I think generalizes to complex/user-driven rules better. And also supports the new field-level validation rules really well (...docs wip...).
[possible with] Joist to load Post with their Comments ordered by comment field or apply some condition
Not via the post1.comments.load(...) method, no; because we cache the loaded-ness of collections + also automatically keep both-side up to date (if you do c1.post.set(p1), then p1.comments.get will automatically has c1 in it), we need them to be the "vanilla" / whole collection.
That said, you could use a dedicated query like:
await em.find(Comment, { post: p1, ...other conditions... }, { orderBy: { title: "ASC" } });
So, it's still doable, just not "via the object graph" (i.e. post.comments.load/get) since we're specifically asking for something that is not "100% the object graph" but instead of custom view of it.
...granted, we could probably add something like post1.comments.load({ ...condition..., ...order... }) that was just syntax sugar for the em.find version, but we don't have that yet.
Also Roman, you had a great call out here:
in the page of docs "Avoiding N+1" the first example you show is NOT N+1 safe
The code snippet on that page was right, but you're right the copy was pretty confusing; that example actually is N+1 safe in Joist.
I updated the docs today to hopefully be a little more clear:
joist-orm.io/docs/goals/avoiding-n-plus-1s
Would love to hear if that's better/not better, and any other thoughts you had.
Hi Stephen Haberman! I see you are continuing to develop Joist ORM (the last commit is yesterday), great to see such insistence.
I developed my ORM too. And updated this article to include a brief overview. Surely it's completely different from yours, I hope you could find some time to check it out, it may have some interesting ideas. I'd be happy to get feedback from the developer of another ORM (even harsh)!
Hey Roman! I took a few looks at Orchid. I like the docs! In terms of feedback, I think my biggest take away is what you also highlight in your docs: personally I wouldn't really call Orchid an ORM, and instead would call it a query builder, ala Knex and Kysely. Granted, "query builder" doesn't really roll off the tongue like "ORM" :-). "Query DSL"? Hm, not sure. Terminology pedanticism aside :-), congrats on shipping!
Stephen Haberman thank you for checking it out!
I understand your point, so I added an explanation on the very first page of the docs that is not a "traditional" ORM. But it is an ORM like Prisma.
None of the query builders is helping with relations, literally none of them, so if I call it a query builder it's like ignoring the most fun part of it: working with relations. Here is a code example in benchmarks section.
Calling it a DSL or a database SDK would tell nothing to potential users. Everyone knows Prisma, and I guess if they can call it ORM, then mine can be called too.
Anyway, there are no strict clear bounds of what ORM is. Different people have different expectations.
So for me, query builder is abstracting only tables, while ORM is also abstracting relations. And how exactly it is implemented - instantiating class or not - is just an implementation detail.
Roman hey!
But it is an ORM like Prisma.
Ah yeah, that's a good point; I suppose I have the same qualm with Prisma as well :-D
query builders is helping with relations
Ah interesting, yeah that is a good point. Thanks for linking to the nested load example. That is a nice way to handle the boilerplate of joins. Looks nice!
With Joist, I've shied away from doing any query building that is not "return me exactly this entity", with the rationale that we'd probably be best off using custom SQL/knex queries for that anyway, but yeah I can see how those "custom"/"not-an-entity" result queries are much nicer in Orchid than doing joins in a query builder. Nice job!
Bookmarking for later. :-)