Compiling large number of DSC MOF files is slow
We use logic to dynamically compose a ConfigurationData structure and pass it to the configuration. This logic executes quickly. Calling the configuration generates more than 5,000 MOF files for unique nodes. The process takes 1.5 hours on modern server-class hardware with 16GB RAM. Also, the MOF files are all created at the end of the process, rather than one-at-a-time throughout the process. This causes high memory usage. Please optimize the PSDesiredStateConfiguration module to generate large quantities of MOF files more quickly. This issue adds significant delay to the DSC pipeline.
Appears Im late to this conversation but reading comments about 70 resources and 500 resources makes me wonder how you are getting around the issue described here and that Im hitting?
Ryan Bartram commented
We too have this issue but Arie makes a point about fundamentally changing the way the MOFs are created. If you have the same configuration applied to lots of different machines or at least partially. Consider using a partial configuration, would mean you would only need to update 1 MOF for shared changes
Mariusz Serwin commented
We are having the same issue it takes 35 minutes to generate around 30 Mofs
Ashley McGlone commented
WMF 5.0 on Windows Server 2012 R2.
Arie H. commented
@Zuldan: maybe you're doing this wrong then. You need to find bugs at an early stage in a "Dev \ Integration" tests. Not in Prod.
I'm not saying there's a 100% fault free in Prod that you didnt find in Dev \ Int. However it will happen less and less if you have a decent Unit tests \ Integration Tests \ OVF implemented.
It shouldn't be a feature of the product (DSC) just because your tests aren't implemented. else its just covering the real issues :)
If there's something that is deemed "small change to servers", consider maybe implementing it via DSC wasn't the best way. Maybe GPO \ GPP would have been better. I know the line between DSC and Server GPO \ GPP gets thinner and thinner but one of the criteria's deciding when to use each is recoverability and maintainability.
If you do find yourself having to recreate 1900 mof files every time for a small change, perhaps the answer isn't in increasing the speed of MOF creation (has its limits in terms of resources) but rather what I wrote in my first comment about Diffing DSC scripts and creating a Diff Mof that updates an existing one.
Arie H. commented
Agree on the part where the MOF file needs to be created immediately after its finished processing that specific node and not at the end of all the X amount of nodes.
I dont see however a reason to do all 5000 at one go. Its not like you're going to deploy all 5000 at once, this is a no-no in terms of network and potential management nightmare when things go wrong on large scale. Even, if say you have a setting you want to change at the base of each config that affects all of your nodes, you will still deploy this in stages, thus creating the MOF at once is not really matter how much time it takes, if you plan it correctly to match your stage nodes.
(this might also goes into somewhat "grey" area of what should be in DSC and what in Server group policy).
I would like to see something like :
I have a DSC script that created a MOF.
I have a change to the script.
I would like to compare the first version to the second version and the result will be just the changes to the MOF I need (think about a DSC script that is 500 resources long...)
I then apply the "Update" MOF to the node.
This means lesser payload in terms of network, lesser resources on the node applying the change.
and better pipeline all along.
Ashley thank you for creating this uservoice. We have the same issue with MOF generation on a mass scale (takes a very long time). 1,900 MOF files is taking roughly 45 minutes for us to generate. Not great when you want to make quick small changes to your servers. Also not great when you find a bug and you need to roll back a servers configuration. That's an additional 45 minutes downtime for the business.
Resources per MOF: 83
Powershell version: 5.0
Ashley McGlone commented
The configurations contain approximately 70 simple resources (registry, service, etc.).