External pipeline and sitemap manipulators

Hello,

I’m working on a rather large Middleman site and am in the process of using Webpack for managing our JS. I have it all running fine but I am having an issue that it is slowing down the local server and the build. When I instrument the server I can see sitemap.manipulator getting repeated on the same names which is the reason for the slow down. Without the external pipeline this does not happen.

We were hoping that removing sprockets would give us a bit of a speed boost in the built but the sitemap.manipulator repetition is actually making it worse.

I have been looking at this for quite some time and am at a loss on how to prevent it. I’m hoping someone out there might be able to help.

Many thanks,
Jeff

== The Middleman is loading
== Executing: yarn start
== External: yarn run v1.22.4
== External: $ ./node_modules/webpack/bin/webpack.js --mode development --watch --progress --color
10% building 0/0 modules 0 active
webpack is watching the files…

== External: Hash: ac9ebcf8ce96a6516d53
== External: Version: webpack 4.41.5
== External: Time: 1277ms
== External: Built at: 2020-05-13 20:10:01
== External: Asset Size Chunks Chunk Names
== External: components/javascript/accountants_forms.js 10.8 KiB accountants_forms [emitted] accountants_forms
== External: components/javascript/anchor_offset.js 5.81 KiB anchor_offset [emitted] anchor_offset
== External: components/javascript/animated_illustrations.js 4.5 KiB animated_illustrations [emitted] animated_illustrations
== External: components/javascript/appear.js 6.55 KiB appear [emitted] appear
== External: components/javascript/careers_tracking.js 6.72 KiB careers_tracking [emitted] careers_tracking
== External: components/javascript/chart.js 213 KiB chart [emitted] chart
== External: components/javascript/clipboard.js 37.4 KiB clipboard [emitted] clipboard
== External: components/javascript/coronavirus_furlough_calculator.js 240 KiB coronavirus_furlough_calculator [emitted] coronavirus_furlough_calculator
== External: components/javascript/countdown.js 5.57 KiB countdown [emitted] countdown
== External: components/javascript/depot.js 19.3 KiB depot [emitted] depot
== External: components/javascript/directory_form.js 47.8 KiB directory_form [emitted] directory_form
== External: components/javascript/disqus_count.js 5.6 KiB disqus_count [emitted] disqus_count
== External: components/javascript/dom_helpers.js 5.07 KiB dom_helpers [emitted] dom_helpers
== External: components/javascript/download_select.js 4.54 KiB download_select [emitted] download_select
== External: components/javascript/email_capture_forms.js 7.64 KiB email_capture_forms [emitted] email_capture_forms
== External: components/javascript/enablement_form_submission.js 49.7 KiB enablement_form_submission [emitted] enablement_form_submission
== External: components/javascript/fieldguide.js 9.98 KiB fieldguide [emitted] fieldguide
== External: components/javascript/flat_rate_scheme.js 27.6 KiB flat_rate_scheme [emitted] flat_rate_scheme
== External: components/javascript/homepage.js 5.81 KiB homepage [emitted] homepage
== External: components/javascript/load_css.js 7.07 KiB load_css [emitted] load_css
== External: components/javascript/location.js 21 KiB location [emitted] location
== External: components/javascript/main.js 68.5 KiB main [emitted] main
== External: components/javascript/main_head_critical.js 107 KiB main_head_critical [emitted] main_head_critical
== External: components/javascript/match_height.js 6.96 KiB match_height [emitted] match_height
== External: components/javascript/mileage_calculator.js 27.2 KiB mileage_calculator [emitted] mileage_calculator
== External: components/javascript/multitouch.js 18.7 KiB multitouch [emitted] multitouch
== External: components/javascript/one_trust_cookie_checker.js 9.94 KiB one_trust_cookie_checker [emitted] one_trust_cookie_checker
== External: components/javascript/one_trust_functional_cookie_checks.js 5.59 KiB one_trust_functional_cookie_checks [emitted] one_trust_functional_cookie_checks
== External: components/javascript/one_trust_remove_duplicate_cookie.js 10.7 KiB one_trust_remove_duplicate_cookie [emitted] one_trust_remove_duplicate_cookie
== External: components/javascript/one_trust_save_refresh.js 4.62 KiB one_trust_save_refresh [emitted] one_trust_save_refresh
== External: components/javascript/portal_form.js 5.51 KiB portal_form [emitted] portal_form
== External: components/javascript/prism.js 33.4 KiB prism [emitted] prism
== External: components/javascript/real_user_metrics.js 25.1 KiB real_user_metrics [emitted] real_user_metrics
== External: components/javascript/referral_code.js 10.2 KiB referral_code [emitted] referral_code
== External: components/javascript/siema.js 18.1 KiB siema [emitted] siema
== External: components/javascript/tabs.js 6.6 KiB tabs [emitted] tabs
== External: components/javascript/timeline.js 6.67 KiB timeline [emitted] timeline
== External: components/javascript/trustpilot_ga_tracking.js 6.81 KiB trustpilot_ga_tracking [emitted] trustpilot_ga_tracking
== External: components/javascript/vat_calculator.js 26.4 KiB vat_calculator [emitted] vat_calculator
== External: components/javascript/wistia.js 46.8 KiB wistia [emitted] wistia
== External: components/javascript/zendesk.js 20.7 KiB zendesk [emitted] zendesk
== External: js-manifest.json 2.63 KiB [emitted]
== External: Entrypoint accountants_forms = components/javascript/accountants_forms.js
== External: Entrypoint anchor_offset = components/javascript/anchor_offset.js
== External: Entrypoint animated_illustrations = components/javascript/animated_illustrations.js
== External: Entrypoint careers_tracking = components/javascript/careers_tracking.js
== External: Entrypoint chart = components/javascript/chart.js
== External: Entrypoint clipboard = components/javascript/clipboard.js
== External: Entrypoint countdown = components/javascript/countdown.js
== External: Entrypoint depot = components/javascript/depot.js
== External: Entrypoint dom_helpers = components/javascript/dom_helpers.js
== External: Entrypoint download_select = components/javascript/download_select.js
== External: Entrypoint email_capture_forms = components/javascript/email_capture_forms.js
== External: Entrypoint enablement_form_submission = components/javascript/enablement_form_submission.js
== External: Entrypoint fieldguide = components/javascript/fieldguide.js
== External: Entrypoint homepage = components/javascript/homepage.js
== External: Entrypoint location = components/javascript/location.js
== External: Entrypoint main = components/javascript/main.js
== External: Entrypoint match_height = components/javascript/match_height.js
== External: Entrypoint multitouch = components/javascript/multitouch.js
== External: Entrypoint one_trust_cookie_checker = components/javascript/one_trust_cookie_checker.js
== External: Entrypoint one_trust_functional_cookie_checks = components/javascript/one_trust_functional_cookie_checks.js
== External: Entrypoint one_trust_remove_duplicate_cookie = components/javascript/one_trust_remove_duplicate_cookie.js
== External: Entrypoint one_trust_save_refresh = components/javascript/one_trust_save_refresh.js
== External: Entrypoint portal_form = components/javascript/portal_form.js
== External: Entrypoint prism = components/javascript/prism.js
== External: Entrypoint referral_code = components/javascript/referral_code.js
== External: Entrypoint siema = components/javascript/siema.js
== External: Entrypoint tabs = components/javascript/tabs.js
== External: Entrypoint timeline = components/javascript/timeline.js
== External: Entrypoint trustpilot_ga_tracking = components/javascript/trustpilot_ga_tracking.js
== External: Entrypoint wistia = components/javascript/wistia.js
== External: Entrypoint zendesk = components/javascript/zendesk.js
== External: Entrypoint coronavirus_furlough_calculator = components/javascript/coronavirus_furlough_calculator.js
== External: Entrypoint flat_rate_scheme = components/javascript/flat_rate_scheme.js
== External: Entrypoint mileage_calculator = components/javascript/mileage_calculator.js
== External: Entrypoint vat_calculator = components/javascript/vat_calculator.js
== External: Entrypoint directory_form = components/javascript/directory_form.js
== External: Entrypoint main_head_critical = components/javascript/main_head_critical.js
== External: Entrypoint real_user_metrics = components/javascript/real_user_metrics.js
== External: Entrypoint appear = components/javascript/appear.js
== External: Entrypoint disqus_count = components/javascript/disqus_count.js
== External: Entrypoint load_css = components/javascript/load_css.js
== External: [./node_modules/webpack/buildin/global.js] (webpack)/buildin/global.js 472 bytes {enablement_form_submission} {prism} {wistia} {directory_form} {load_css} [built]
== External: [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {coronavirus_furlough_calculator} [built]
== External: + 87 hidden modules
== Instrument (sitemap.manipulator): 752.18ms
{:name=>:sitemap_ondisk}
== Instrument (sitemap.manipulator): 105.145ms
{:name=>:sitemap_endpoint}
== Instrument (sitemap.manipulator): 104.56899999999999ms
{:name=>:sitemap_proxies}
== Instrument (sitemap.manipulator): 78.227ms
{:name=>:sitemap_redirects}
== Instrument (sitemap.manipulator): 266.385ms
{:name=>:sitemap_ignore}
== Instrument (sitemap.manipulator): 98.564ms
{:name=>:sitemap_import}
== Instrument (sitemap.manipulator): 191.842ms
{:name=>:routing}
== Instrument (sitemap.manipulator): 3283.6949999999997ms
{:name=>:front_matter}
== Instrument (sitemap.manipulator): 96.163ms
{:name=>:sitemap_move_files}
== Instrument (sitemap.manipulator): 88.424ms
{:name=>:collections}
== Instrument (sitemap.manipulator): 200.383ms
{:name=>:routing}
== Instrument (sitemap.update): 5270.879ms
{:reasons=>[:touched_file, :registered_new_manipulator_front_matter, :registered_new_manipulator_sitemap_ondisk, :registered_new_manipulator_sitemap_import, :registered_new_manipulator_sitemap_endpoint, :registered_new_manipulator_sitemap_proxies, :registered_new_manipulator_sitemap_redirects, :registered_new_manipulator_sitemap_move_files, :registered_new_manipulator_sitemap_ignore, :registered_new_manipulator_routing, :registered_new_manipulator_collections]}
== Locales: en-GB, en-US, en (Default en-GB)
== Blog Sources: blog/en-GB/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: company/press-room/en-GB/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: features/customer-stories/en-GB/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: blog/en-US/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: company/press-room/en-US/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: features/customer-stories/en-US/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: blog/en/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: company/press-room/en/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: features/customer-stories/en/{lang}/{year}-{title}.html (:prefix + :sources)
== Blog Sources: /practice-portal/blog/{year}-{title}.html (:prefix + :sources)
== Blog Sources: /practice-portal/videos/{year}-{title}.html (:prefix + :sources)
== Blog Sources: /accountants/partner-stories/{year}-{title}.html (:prefix + :sources)
== Instrument (sitemap.manipulator): 678.307ms
{:name=>:sitemap_ondisk}
== Instrument (sitemap.manipulator): 95.915ms
{:name=>:sitemap_endpoint}
== Instrument (sitemap.manipulator): 90.188ms
{:name=>:sitemap_proxies}
== Instrument (sitemap.manipulator): 88.82600000000001ms
{:name=>:sitemap_redirects}
== Instrument (sitemap.manipulator): 749.726ms
{:name=>:sitemap_ignore}
== Instrument (sitemap.manipulator): 104.239ms
{:name=>:sitemap_import}
== Instrument (sitemap.manipulator): 230.378ms
{:name=>:routing}
== Instrument (sitemap.manipulator): 598.321ms
{:name=>:front_matter}
== Instrument (sitemap.manipulator): 445.548ms
{:name=>:i18n}
== Instrument (sitemap.manipulator): 1049.051ms
{:name=>:fapd_page_builder}
== Instrument (sitemap.manipulator): 1929.5310000000002ms
{:name=>:guides_page_builder}
== Instrument (sitemap.manipulator): 3943.145ms
{:name=>:blog_blog_articles}
== Instrument (sitemap.manipulator): 284.385ms
{:name=>:blog_blog_tags}
== Instrument (sitemap.manipulator): 427.851ms
{:name=>:blog_blog_paginate}
== Instrument (sitemap.manipulator): 1531.569ms
{:name=>:blog_press_room_articles}
== Instrument (sitemap.manipulator): 434.18600000000004ms
{:name=>:blog_press_room_paginate}
== Instrument (sitemap.manipulator): 1539.962ms
{:name=>:blog_customer_stories_articles}
== Instrument (sitemap.manipulator): 386.399ms
{:name=>:blog_customer_stories_paginate}
== Instrument (sitemap.manipulator): 1473.606ms
{:name=>:“blog_blog_en-US_articles”}
== Instrument (sitemap.manipulator): 317.49100000000004ms
{:name=>:“blog_blog_en-US_tags”}
== Instrument (sitemap.manipulator): 480.725ms
{:name=>:“blog_blog_en-US_paginate”}
== Instrument (sitemap.manipulator): 1860.046ms
{:name=>:“blog_press_room_en-US_articles”}
== Instrument (sitemap.manipulator): 452.28000000000003ms
{:name=>:“blog_press_room_en-US_paginate”}
== Instrument (sitemap.manipulator): 1512.9289999999999ms
{:name=>:“blog_customer_stories_en-US_articles”}
== Instrument (sitemap.manipulator): 394.20300000000003ms
{:name=>:“blog_customer_stories_en-US_paginate”}
== Instrument (sitemap.manipulator): 1811.975ms
{:name=>:blog_blog_en_articles}
== Instrument (sitemap.manipulator): 335.54200000000003ms
{:name=>:blog_blog_en_tags}
== Instrument (sitemap.manipulator): 429.142ms
{:name=>:blog_blog_en_paginate}
== Instrument (sitemap.manipulator): 1453.7230000000002ms
{:name=>:blog_press_room_en_articles}
== Instrument (sitemap.manipulator): 393.93600000000004ms
{:name=>:blog_press_room_en_paginate}
== Instrument (sitemap.manipulator): 1773.9270000000001ms
{:name=>:blog_customer_stories_en_articles}
== Instrument (sitemap.manipulator): 448.64300000000003ms
{:name=>:blog_customer_stories_en_paginate}
== Instrument (sitemap.manipulator): 1451.515ms
{:name=>:blog_practice_portal_blog_articles}
== Instrument (sitemap.manipulator): 342.674ms
{:name=>:blog_practice_portal_blog_tags}
== Instrument (sitemap.manipulator): 432.05100000000004ms
{:name=>:blog_practice_portal_blog_paginate}
== Instrument (sitemap.manipulator): 1285.942ms
{:name=>:blog_practice_portal_videos_articles}
== Instrument (sitemap.manipulator): 236.46200000000002ms
{:name=>:blog_practice_portal_videos_tags}
== Instrument (sitemap.manipulator): 421.043ms
{:name=>:blog_practice_portal_videos_paginate}
== Instrument (sitemap.manipulator): 1404.932ms
{:name=>:blog_accountant_partner_stories_articles}
== Instrument (sitemap.manipulator): 424.899ms
{:name=>:blog_accountant_partner_stories_paginate}
== Instrument (sitemap.manipulator): 366.428ms
{:name=>:directory_indexes}
== Instrument (sitemap.manipulator): 276.005ms
{:name=>:sitemap_move_files}
== Instrument (sitemap.manipulator): 274.245ms
{:name=>:collections}
== Instrument (sitemap.manipulator): 461.549ms
{:name=>:routing}
== Instrument (sitemap.update): 35129.545ms
{:reasons=>[:touched_file, :registered_new_manipulator_i18n, :registered_new_manipulator_fapd_page_builder, :registered_new_manipulator_guides_page_builder, :first_run_change_ignore, :registered_new_manipulator_blog_blog_articles, :registered_new_manipulator_blog_blog_tags, :registered_new_manipulator_blog_blog_paginate, :registered_new_manipulator_blog_press_room_articles, :registered_new_manipulator_blog_press_room_paginate, :registered_new_manipulator_blog_customer_stories_articles, :registered_new_manipulator_blog_customer_stories_paginate, :“registered_new_manipulator_blog_blog_en-US_articles”, :“registered_new_manipulator_blog_blog_en-US_tags”, :“registered_new_manipulator_blog_blog_en-US_paginate”, :“registered_new_manipulator_blog_press_room_en-US_articles”, :“registered_new_manipulator_blog_press_room_en-US_paginate”, :“registered_new_manipulator_blog_customer_stories_en-US_articles”, :“registered_new_manipulator_blog_customer_stories_en-US_paginate”, :registered_new_manipulator_blog_blog_en_articles, :registered_new_manipulator_blog_blog_en_tags, :registered_new_manipulator_blog_blog_en_paginate, :registered_new_manipulator_blog_press_room_en_articles, :registered_new_manipulator_blog_press_room_en_paginate, :registered_new_manipulator_blog_customer_stories_en_articles, :registered_new_manipulator_blog_customer_stories_en_paginate, :registered_new_manipulator_blog_practice_portal_blog_articles, :registered_new_manipulator_blog_practice_portal_blog_tags, :registered_new_manipulator_blog_practice_portal_blog_paginate, :registered_new_manipulator_blog_practice_portal_videos_articles, :registered_new_manipulator_blog_practice_portal_videos_tags, :registered_new_manipulator_blog_practice_portal_videos_paginate, :registered_new_manipulator_blog_accountant_partner_stories_articles, :registered_new_manipulator_blog_accountant_partner_stories_paginate, :registered_new_manipulator_directory_indexes]}
== Instrument (sitemap.manipulator): 690.0889999999999ms
{:name=>:sitemap_ondisk}
== Instrument (sitemap.manipulator): 171.6ms
{:name=>:sitemap_endpoint}
== Instrument (sitemap.manipulator): 148.479ms
{:name=>:sitemap_proxies}
== Instrument (sitemap.manipulator): 89.785ms
{:name=>:sitemap_redirects}
== Instrument (sitemap.manipulator): 1011.403ms
{:name=>:sitemap_ignore}
== Instrument (sitemap.manipulator): 87.774ms
{:name=>:sitemap_import}
== Instrument (sitemap.manipulator): 203.63400000000001ms
{:name=>:routing}
== Instrument (sitemap.manipulator): 632.393ms
{:name=>:front_matter}
== Instrument (sitemap.manipulator): 523.952ms
{:name=>:i18n}
== Instrument (sitemap.manipulator): 971.2460000000001ms
{:name=>:fapd_page_builder}
== Instrument (sitemap.manipulator): 286.617ms
{:name=>:guides_page_builder}
== Instrument (sitemap.manipulator): 1926.987ms
{:name=>:blog_blog_articles}
== Instrument (sitemap.manipulator): 294.965ms
{:name=>:blog_blog_tags}
== Instrument (sitemap.manipulator): 413.75ms
{:name=>:blog_blog_paginate}
== Instrument (sitemap.manipulator): 1641.712ms
{:name=>:blog_press_room_articles}
== Instrument (sitemap.manipulator): 409.784ms
{:name=>:blog_press_room_paginate}
== Instrument (sitemap.manipulator): 1514.908ms
{:name=>:blog_customer_stories_articles}
== Instrument (sitemap.manipulator): 556.323ms
{:name=>:blog_customer_stories_paginate}
== Instrument (sitemap.manipulator): 1711.327ms
{:name=>:“blog_blog_en-US_articles”}
== Instrument (sitemap.manipulator): 278.155ms
{:name=>:“blog_blog_en-US_tags”}
== Instrument (sitemap.manipulator): 705.4780000000001ms
{:name=>:“blog_blog_en-US_paginate”}
== Instrument (sitemap.manipulator): 1645.469ms
{:name=>:“blog_press_room_en-US_articles”}
== Instrument (sitemap.manipulator): 449.954ms
{:name=>:“blog_press_room_en-US_paginate”}
== Instrument (sitemap.manipulator): 1767.049ms
{:name=>:“blog_customer_stories_en-US_articles”}
== Instrument (sitemap.manipulator): 441.49699999999996ms
{:name=>:“blog_customer_stories_en-US_paginate”}
== Instrument (sitemap.manipulator): 1495.536ms
{:name=>:blog_blog_en_articles}
== Instrument (sitemap.manipulator): 236.131ms
{:name=>:blog_blog_en_tags}
== Instrument (sitemap.manipulator): 433.012ms
{:name=>:blog_blog_en_paginate}
== Instrument (sitemap.manipulator): 1471.435ms
{:name=>:blog_press_room_en_articles}
== Instrument (sitemap.manipulator): 413.129ms
{:name=>:blog_press_room_en_paginate}
== Instrument (sitemap.manipulator): 1443.223ms
{:name=>:blog_customer_stories_en_articles}
== Instrument (sitemap.manipulator): 539.698ms
{:name=>:blog_customer_stories_en_paginate}
== Instrument (sitemap.manipulator): 1423.2450000000001ms
{:name=>:blog_practice_portal_blog_articles}
== Instrument (sitemap.manipulator): 244.79500000000002ms
{:name=>:blog_practice_portal_blog_tags}
== Instrument (sitemap.manipulator): 410.713ms
{:name=>:blog_practice_portal_blog_paginate}
== Instrument (sitemap.manipulator): 1335.864ms
{:name=>:blog_practice_portal_videos_articles}
== Instrument (sitemap.manipulator): 303.055ms
{:name=>:blog_practice_portal_videos_tags}
== Instrument (sitemap.manipulator): 401.385ms
{:name=>:blog_practice_portal_videos_paginate}
== Instrument (sitemap.manipulator): 1276.597ms
{:name=>:blog_accountant_partner_stories_articles}
== Instrument (sitemap.manipulator): 389.42199999999997ms
{:name=>:blog_accountant_partner_stories_paginate}
== Instrument (sitemap.manipulator): 360.556ms
{:name=>:directory_indexes}
== Instrument (sitemap.manipulator): 303.38599999999997ms
{:name=>:sitemap_move_files}
== Instrument (sitemap.manipulator): 360.74699999999996ms
{:name=>:collections}
== Instrument (sitemap.manipulator): 463.02700000000004ms
{:name=>:routing}
== Instrument (sitemap.update): 31892.124ms
{:reasons=>[:touched_file]}
== View your site at “http://jeff-knox-FA503.local:3333”, “http://192.168.0.27:3333
== Inspect your site configuration at “http://jeff-knox-FA503.local:3333/__middleman”, “http://192.168.0.27:3333/__middleman

After the == Instrument (sitemap.update): 35129.545ms and the long list of reasons, the same thing happens all over again but this time with just 1 reason {:reasons=>[:touched_file]}

I’ve also tested clearing out everything else in config.rb to make sure it’s not related to other extensions but the same behaviour of repetition still exists.

You might want to file an issue for this.