Responded to @kasparsd:
Importantly, it doesn’t matter if just one sitemap is slow to respond — it will still encourage Google to throttle crawling.
Most of the popular SEO plugin sitemap implementations are known to have both issues on sites with 10k posts or more — there is a delay in regeneration and the dynamic response times can be 30 seconds and more even on dedicated enterprise hosting.
Make sure your sitemaps are quick to update and fast to render at ALL times!