Such gains by limiting included headers is surprising to me, as it’s the first thing anyone would suggest doing. Clang-tidy hints in QtCreator show warnings for includes that are not used. For me this works pretty well to keep build times due to headers under control. I wonder, if reducing the amount of included headers already yields such significant gains, what other gains can be had, and what LOC we’re talking about. I’ve seen dramatic improvements by using pch for instance. Or isolating boost usage.
Such gains by limiting included headers is surprising to me, as it’s the first thing anyone would suggest doing.
Yes indeed. I think this is a testament of the loss of know-how we’re seeing in software engineering in general, and how overambitious but underworking developers try to stake claims in technical expertise when they even failed to onboard onto the very basics of a tech stack.
I’m sure it’s a matter of time before there’s a new post in Figma’s blog showing off their latest advanced technique to drive down build times: onboarding ccache. Followed by another blog post on how Figma is researching cutting edge distributed computing techniques to optimize build times by replacing ccache with sccache.
I’ve had mixed results with ccache myself, ending up not using it. Compilation times are much less of a problem for me than they were before, because of the increases in processor power and number of threads. This together with pchs and judicously forward declaring and including only what you use.
I’ve had mixed results with ccache myself, ending up not using it.
Which problems did you experienced?
Compilation times are much less of a problem for me than they were before, because of the increases in processor power and number of threads.
To each its own, but with C++ projects the only way to not stumble upon lengthy build times is by only working with trivial projects. Incremental builds help blunt the pain but that only goes so far.
This together with pchs (…)
This might be the reason ccache only went so far in your projects. Precompiled headers either prevent ccache from working, or require additional tweaks to get around them.
Also noteworthy, msvc doesn’t play well with ccache. Details are fuzzy, but I think msvc supports building multiple source files with a single invocation, which prevents ccache to map an input to an output object file.
ccache folder size started becoming huge. And it just didn’t speed up the project builds, I don’t remember the details of why.
This might be the reason ccache only went so far in your projects. Precompiled headers either prevent ccache from working, or require additional tweaks to get around them.
Right, that might have been the reason.
To each its own, but with C++ projects the only way to not stumble upon lengthy build times is by only working with trivial projects. Incremental builds help blunt the pain but that only goes so far.
When I tried it I was working on a 100+ devs C++ project, 3/4M LOC, about as big as they come. Compilation of everything from scratch was an hour at the end. Switching to lld was a huge win, as well as going from 12 to compilation 24 threads. The code-base in a way you don’t need to build everything to work on a specific part, using dynamically loaded libraries to inject functionality in the main app.
I was a linux dev there, the pch’s worked, not as well as for MSVC where they made a HUGE difference. Otoh lld blows the microsoft linker out of the water, clean builds were faster on msvc, incremental faster on linux.
ccache folder size started becoming huge. And it just didn’t speed up the project builds, I don’t remember the details of why.
That’s highly unusual, and suggests you misconfigured your project to actually not cache your builds, and instead it just gathered precompiled binaries that it could not reuse due to being misconfigured.
When I tried it I was working on a 100+ devs C++ project, 3/4M LOC, about as big as they come.
That’s not necessarily a problem. I worked on C++ projects which were the similar size and ccache just worked. It has more to do with how you’re project is set, and misconfigurations.
Compilation of everything from scratch was an hour at the end.
That fits my usecase as well. End-to-end builds took slightly longer than 1h, but after onboarding ccache the same end-to-end builds would take less than 2 minutes. Incremental builds were virtually instant.
Switching to lld was a huge win, as well as going from 12 to compilation 24 threads.
That’s perfectly fine. Ccache acts before linking, and naturally being able to run more parallel tasks can indeed help, regardless of ccache being in place.
Surprisingly, ccache works even better in this scenario. With ccache, the bottleneck of any build task switches from the CPU/Memory to IO. This had the nice trait that it was now possible to overcommit the number of jobs as the processor was no longer being maxed out. In my case it was possible to run around 40% more build jobs than physical threads to get a CPU utilization rate above 80%.
I was a linux dev there, the pch’s worked, (…)
I dare say ccache was not caching what it could due to precompiled headers. If you really want those, you need to configure ccache to tolerate them. Nevertheless it’s a tad pointless to have pch in a project for performance reasons when you can have a proper compiler cache.
Such gains by limiting included headers is surprising to me, as it’s the first thing anyone would suggest doing. Clang-tidy hints in QtCreator show warnings for includes that are not used. For me this works pretty well to keep build times due to headers under control. I wonder, if reducing the amount of included headers already yields such significant gains, what other gains can be had, and what LOC we’re talking about. I’ve seen dramatic improvements by using pch for instance. Or isolating boost usage.
Yes indeed. I think this is a testament of the loss of know-how we’re seeing in software engineering in general, and how overambitious but underworking developers try to stake claims in technical expertise when they even failed to onboard onto the very basics of a tech stack.
I’m sure it’s a matter of time before there’s a new post in Figma’s blog showing off their latest advanced technique to drive down build times: onboarding ccache. Followed by another blog post on how Figma is researching cutting edge distributed computing techniques to optimize build times by replacing ccache with sccache.
TIL about sccache, thanks
I’ve had mixed results with ccache myself, ending up not using it. Compilation times are much less of a problem for me than they were before, because of the increases in processor power and number of threads. This together with pchs and judicously forward declaring and including only what you use.
Which problems did you experienced?
To each its own, but with C++ projects the only way to not stumble upon lengthy build times is by only working with trivial projects. Incremental builds help blunt the pain but that only goes so far.
This might be the reason ccache only went so far in your projects. Precompiled headers either prevent ccache from working, or require additional tweaks to get around them.
https://ccache.dev/manual/4.9.1.html#_precompiled_headers
Also noteworthy, msvc doesn’t play well with ccache. Details are fuzzy, but I think msvc supports building multiple source files with a single invocation, which prevents ccache to map an input to an output object file.
ccache folder size started becoming huge. And it just didn’t speed up the project builds, I don’t remember the details of why.
Right, that might have been the reason.
When I tried it I was working on a 100+ devs C++ project, 3/4M LOC, about as big as they come. Compilation of everything from scratch was an hour at the end. Switching to lld was a huge win, as well as going from 12 to compilation 24 threads. The code-base in a way you don’t need to build everything to work on a specific part, using dynamically loaded libraries to inject functionality in the main app.
I was a linux dev there, the pch’s worked, not as well as for MSVC where they made a HUGE difference. Otoh lld blows the microsoft linker out of the water, clean builds were faster on msvc, incremental faster on linux.
That’s highly unusual, and suggests you misconfigured your project to actually not cache your builds, and instead it just gathered precompiled binaries that it could not reuse due to being misconfigured.
That’s not necessarily a problem. I worked on C++ projects which were the similar size and ccache just worked. It has more to do with how you’re project is set, and misconfigurations.
That fits my usecase as well. End-to-end builds took slightly longer than 1h, but after onboarding ccache the same end-to-end builds would take less than 2 minutes. Incremental builds were virtually instant.
That’s perfectly fine. Ccache acts before linking, and naturally being able to run more parallel tasks can indeed help, regardless of ccache being in place.
Surprisingly, ccache works even better in this scenario. With ccache, the bottleneck of any build task switches from the CPU/Memory to IO. This had the nice trait that it was now possible to overcommit the number of jobs as the processor was no longer being maxed out. In my case it was possible to run around 40% more build jobs than physical threads to get a CPU utilization rate above 80%.
I dare say ccache was not caching what it could due to precompiled headers. If you really want those, you need to configure ccache to tolerate them. Nevertheless it’s a tad pointless to have pch in a project for performance reasons when you can have a proper compiler cache.