Is Middleman a good fit for my use case - 1M+ hierarchically arranged pages?

#1

I’ve been using Rails to put laws online, making them easier to read and link to. Although data-driven, the site is 99% static. Some states laws only need to update every two years. Others, once per day. I have the data in well formed JSON, ready to import.

I’ve always felt like Rails isn’t quite a good fit; the requests don’t need to be dynamically served. I’m thinking I might be able to use a site generator like Middleman plus a small dynamic app for the search engine.

Thanks for any input!

#2

You can give it a go. The only challenge I am seeing is the generation time. But once you generate it and upload it, you don’t need to do anything.

Perhaps there is a way with Middleman to “lock” some posts so this way, if you want to update something, you don’t need to regenerate everything. Actually, I am wondering the same thing so that would be awesome if someone can shed more light on this process.

#3

For my site (about 330 pages), middleman generates the site in around 30 seconds, so scaling that up I’m guessing around 24 hours to generate a million pages. Sounds a bit painful.

#4

Yeah, so there could be a way to exclude some pages from the process and only generate the new one or the one you wish. Perhaps a key and value in data like generate : true. Just an idea, this is not in documentation :slight_smile:

#5

My largest Middleman site has about 5000 pages, plus large numbers of images, and takes a noticeable amount of time – maybe 10-15 minutes – to rebuild on an 8GB Linode VM. I would think that a million page site could be pretty slow to rebuild; hours if not days. During the build process, Middleman creates a complex data structure called the sitemap, which loads at least some information about every page on the site. A sitemap with a million entries would probably have significant memory requirements (understatement!)

You might want to take a look at Hugo, which is a static site generator written in Go. I’ve played around a little bit with Hugo, and while I haven’t used it yet for any major projects, I can confirm that it is extremely fast.