haikuwebkit/Websites/perf.webkit.org/tools/sync-buildbot.js

59 lines
2.0 KiB
JavaScript
Raw Permalink Normal View History

New buildbot syncing scripts that supports multiple builders and slaves https://bugs.webkit.org/show_bug.cgi?id=156269 Reviewed by Chris Dumez. Add sync-buildbot.js that supports scheduling A/B testing jobs on multiple builders and slaves. The old python script (sync-with-buildbot.py) could only support a single builder and slave for each platform, test pair. The main logic is implemented in BuildbotTriggerable.syncOnce. Various helper methods are added throughout the codebase and tests have been refactored. BuildbotSyncer has been updated to support multiple platform, test pairs. It's now responsible for syncing everything on each builder (on a buildbot). Added more unit tests for BuildbotSyncer and server tests for BuildbotTriggerable, and refactored test helpers and mocks as needed. * public/v3/models/build-request.js: (BuildRequest.prototype.status): Added. (BuildRequest.prototype.isScheduled): Added. * public/v3/models/metric.js: (Metric.prototype.fullName): Added. * public/v3/models/platform.js: (Platform): Added the map based on platform name. (Platform.findByName): Added. * public/v3/models/test.js: (Test.topLevelTests): (Test.findByPath): Added. Finds a test based on an array of test names; e.g. ['A', 'B'] would find the test whose name is "B" which has a parent test named "A". (Test.prototype.fullName): Added. * server-tests/api-build-requests-tests.js: (addMockData): Moved to resources/mock-data.js. (addAnotherMockTestGroup): Ditto. * server-tests/resources/mock-data.js: Added. (MockData.resetV3Models): Added. (MockData.addMockData): Moved from api-build-requests-tests.js. (MockData.addAnotherMockTestGroup): Ditto. (MockData.mockTestSyncConfigWithSingleBuilder): Added. (MockData.mockTestSyncConfigWithTwoBuilders): Added. (MockData.pendingBuild): Added. (MockData.runningBuild): Added. (MockData.finishedBuild): Added. * server-tests/resources/test-server.js: (TestServer): (TestServer.prototype.remoteAPI): (TestServer.prototype._ensureTestDatabase): Don't fail even if the test database doesn't exit. (TestServer.prototype._startApache): Create a RemoteAPI instance to access the test sever. (TestServer.prototype._waitForPid): Increase the timeout. (TestServer.prototype.inject): Replace global.RemoteAPI during the test and restore it afterwards. * server-tests/tools-buildbot-triggerable-tests.js: Added. Tests BuildbotTriggerable.syncOnce. (MockLogger): Added. (MockLogger.prototype.log): Added. (MockLogger.prototype.error): Added. * tools/detect-changes.js: (parseArgument): Moved to js/parse-arguments.js. * tools/js/buildbot-syncer.js: (BuildbotBuildEntry): (BuildbotBuildEntry.prototype.syncer): Added. (BuildbotBuildEntry.prototype.buildRequestStatusIfUpdateIsNeeded): Added. Returns a new status for a build request (of the matching build request ID) if it needs to be updated in the server. (BuildbotSyncer): This class (BuildbotSyncer.prototype.addTestConfiguration): Added. (BuildbotSyncer.prototype.testConfigurations): Returns the list of test configurations. (BuildbotSyncer.prototype.matchesConfiguration): Returns true iff the request can be scheduled on this builder. (BuildbotSyncer.prototype.scheduleRequest): Added. Schedules a new job on buildbot for a request. (BuildbotSyncer.prototype.scheduleFirstRequestInGroupIfAvailable): Added. Schedules a new job for the specified build request on the first slave that's available. (BuildbotSyncer.prototype.pullBuildbot): Return a list of BuildbotBuildEntry instead of an object. Also store it on an instance variable so that scheduleFirstRequestInGroupIfAvailable could use it. (BuildbotSyncer.prototype._pullRecentBuilds): (BuildbotSyncer.prototype.pathForPendingBuildsJSON): Renamed from urlForPendingBuildsJSON and now only returns the path instead of the full URL since RemoteAPI takes a path, not full URL. (BuildbotSyncer.prototype.pathForBuildJSON): Ditto from pathForBuildJSON. (BuildbotSyncer.prototype.pathForForceBuild): Added. (BuildbotSyncer.prototype.url): Use RemoteAPI's url method instead of manually constructing URL. (BuildbotSyncer.prototype.urlForBuildNumber): Ditto. (BuildbotSyncer.prototype._propertiesForBuildRequest): Now that each syncer can have multiple test configurations associated with it, find the one matching for this request. (BuildbotSyncer._loadConfig): Create a syncer per builder and add all test configurations to it. (BuildbotSyncer._validateAndMergeConfig): Added the support for 'SlaveList', which is a list of slave names present on this builder. * tools/js/buildbot-triggerable.js: Added. (BuildbotTriggerable): Added. (BuildbotTriggerable.prototype.name): Added. (BuildbotTriggerable.prototype.syncOnce): Added. The main logic for the syncing script. It pulls existing build requests from the perf dashboard, pulls buildbot for pending and running/completed builds on each builder (represented by each syncer), schedules build requests on buildbot if there is any builder/slave available, and updates the status of build requests in the database. (BuildbotTriggerable.prototype._validateRequests): Added. (BuildbotTriggerable.prototype._pullBuildbotOnAllSyncers): Added. (BuildbotTriggerable.prototype._scheduleNextRequestInGroupIfSlaveIsAvailable): Added. (BuildbotTriggerable._testGroupMapForBuildRequests): Added. * tools/js/database.js: * tools/js/parse-arguments.js: Added. Extracted out of tools/detect-changes.js. (parseArguments): * tools/js/remote.js: (RemoteAPI): Now optionally takes the server configuration. (RemoteAPI.prototype.url): Added. (RemoteAPI.prototype.getJSON): Removed the code for specifying request content. (RemoteAPI.prototype.getJSONWithStatus): Ditto. (RemoteAPI.prototype.postJSON): Added. (RemoteAPI.prototype.postFormUrlencodedData): Added. (RemoteAPI.prototype.sendHttpRequest): Fixed the code to specify auth. * tools/js/v3-models.js: Don't include RemoteAPI here as they require a configuration for each host. * tools/sync-buildbot.js: Added. (main): Added. Parse the arguments and start the loop. (syncLoop): Added. * unit-tests/buildbot-syncer-tests.js: Added tests for pullBuildbot, scheduleRequest, as well as scheduleFirstRequestInGroupIfAvailable. Refactored helper functions as needed. (sampleiOSConfig): (smallConfiguration): Added. (smallPendingBuild): Added. (smallInProgressBuild): Added. (smallFinishedBuild): Added. (createSampleBuildRequest): Create a unique build request for each platform. (samplePendingBuild): Optionally specify build time and slave name. (sampleInProgressBuild): Optionally specify slave name. (sampleFinishedBuild): Ditto. * unit-tests/resources/mock-remote-api.js: (assert.notReached.assert.notReached): (MockRemoteAPI.url): Added. (MockRemoteAPI.postFormUrlencodedData): Added. (MockRemoteAPI._addRequest): Extracted from getJSONWithStatus. (MockRemoteAPI.waitForRequest): Extracted from inject. For tools-buildbot-triggerable-tests.js, we need to instantiate a RemoteAPI for buildbot without replacing global.RemoteAPI. (MockRemoteAPI.inject): (MockRemoteAPI.reset): Added. Canonical link: https://commits.webkit.org/174395@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@199123 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-04-06 23:19:29 +00:00
#!/usr/local/bin/node
'use strict';
let BuildbotTriggerable = require('./js/buildbot-triggerable.js').BuildbotTriggerable;
let RemoteAPI = require('./js/remote.js').RemoteAPI;
let fs = require('fs');
let parseArguments = require('./js/parse-arguments.js').parseArguments;
function main(argv)
{
let options = parseArguments(argv, [
{name: '--server-config-json', required: true},
{name: '--buildbot-config-json', required: true},
{name: '--seconds-to-sleep', type: parseFloat, default: 120},
[perf dashboard] Add sequential mode for perf dashboard A/B testing. https://bugs.webkit.org/show_bug.cgi?id=223886 Reviewed by Ryosuke Niwa. Add 'testgroup_repetition_type' to 'analysis_test_groups' table to store repetition type which defaults to 'alternating'. Add support to schedule 'sequential' A/B testing which schedules all iterations for the first configuration (A) before the iterations for the second configuration (B). Before this patch, all A/B testing alternated between two different configurations for each iteration. Add retry logic for 'sequential' A/B testing. Update syncing script to not proceed with next configuration the current configuration successfully completed more iterations than the initially requested or retries exceeded the list. Fix a potential race in syncing script that 'Promise.all' may cause test group not scheduled in order. * browser-tests/test-group-form-tests.js: Updated and added unit tests. * init-database.sql: Added 'testgroup_repetition_type' field to 'analysis_test_groups' table. * migrate-database.sql: Updated migration script to conditionally add 'testgroup_repetition_type' field. * public/api/test-groups.php: Added 'repetitionType' field in api return value. * public/include/build-requests-fetcher.php: Fetching build requests under a triggerable should include test group with all test groups finished but still need retry. This will allow syncing script to resolve the build request by ID and block other request to be scheduled on the builder. * public/include/commit-sets-helpers.php: Added code to create build requests based on test mode. * public/privileged-api/add-build-requests.php: Added support to only add build requests to one commit set. * public/privileged-api/create-analysis-task.php: Added 'repetitionType' support. * public/privileged-api/create-test-group.php: Added 'repetitionType' support. * public/privileged-api/update-test-group.php: * public/shared/common-component-base.js: * public/v3/components/custom-configuration-test-group-form.js: Added 'repetitionType' support. (CustomConfigurationTestGroupForm.prototype.startTesting): * public/v3/components/customizable-test-group-form.js: (CustomizableTestGroupForm.prototype.startTesting): * public/v3/components/test-group-form.js: (TestGroupForm.prototype.startTesting): * public/v3/models/analysis-task.js: (AnalysisTask.async create): (AnalysisTask): * public/v3/models/build-request.js: (BuildRequest): (BuildRequest.prototype.updateSingleton): Allow build request order to be updated. * public/v3/models/test-group.js: (TestGroup): (TestGroup.prototype.updateSingleton): (TestGroup.prototype.repetitionType): (TestGroup.prototype.repetitionCountForCommitSet): Refactored `repetitionCount` since repetition may be different between configurations. (TestGroup.prototype.hasRetries): A helper function to tell if a test group has retry runs. (TestGroup.prototype.additionalRepetitionNeededToReachInitialRepetitionCount): (TestGroup.prototype.successfulTestCount): (TestGroup.prototype.isFirstTestRequest): (TestGroup.prototype.precedingBuildRequest): (TestGroup.prototype.retryCountForCommitSet): (TestGroup.prototype.retryCountsAreSameForAllCommitSets): (TestGroup.prototype.async _createAlternatingRetriesForTestGroup): Moved from 'retry-failed-build-requests' module. (TestGroup.async _createSequentialRetriesForTestGroup): Moved from 'retry-failed-build-requests' module. (TestGroup.prototype.async cancelPendingRequests): (TestGroup.prototype.async didSendNotification): (TestGroup.prototype.async addMoreBuildRequests): (TestGroup.async createWithTask): (TestGroup.async createWithCustomConfiguration): (TestGroup.async createAndRefetchTestGroups): (TestGroup.prototype.async scheduleMoreRequestsOrClearFlag): Moved from 'retry-failed-build-requests' module. (TestGroup.async fetchById): Added support to fetch test group by ID. * public/v3/pages/analysis-task-page.js: (AnalysisTaskChartPane.prototype.didConstructShadowTree): (AnalysisTaskResultsPane.prototype.didConstructShadowTree): (AnalysisTaskConfiguratorPane.prototype.setTestGroups): (AnalysisTaskConfiguratorPane): (AnalysisTaskTestGroupPane.prototype.didConstructShadowTree): (AnalysisTaskTestGroupPane.prototype._renderCurrentTestGroup): (AnalysisTaskTestGroupPane.prototype._retrySummary): (AnalysisTaskPage.prototype.didConstructShadowTree): (AnalysisTaskPage.prototype.async _retryCurrentTestGroup): (AnalysisTaskPage.prototype.async _bisectCurrentTestGroup): (AnalysisTaskPage.set const): (AnalysisTaskPage.prototype.async _createTestGroupAfterVerifyingCommitSetList): (AnalysisTaskPage.prototype.async _createCustomTestGroup): * public/v3/pages/chart-pane.js: (ChartPane.prototype.didConstructShadowTree): (ChartPane.prototype.async _analyzeRange): * public/v3/pages/create-analysis-task-page.js: (CreateAnalysisTaskPage.prototype.async _createAnalysisTaskWithGroup): * server-tests/api-build-requests-tests.js: Added a unit test. * server-tests/api-upload-root-tests.js: Updated unit tests. * server-tests/privileged-api-add-build-requests-tests.js: Updated and added unit tests. * server-tests/privileged-api-create-analysis-task-tests.js: Added unit tests. * server-tests/privileged-api-create-test-group-tests.js: Updated and added unit tests. * server-tests/privileged-api-update-test-group-tests.js: Updated and added unit tests. * server-tests/resources/common-operations.js: (async assertThrows): * server-tests/resources/mock-data.js: (MockData.addMockData): (MockData.set addMockBuildRequestsWithRoots): (MockData.set addAnotherMockTestGroup): (MockData.set mockTestSyncConfigWithSingleBuilder): (MockData.sampleBuildData): * server-tests/tools-buildbot-triggerable-tests.js: Updated and added unit tests. * server-tests/tools-sync-buildbot-integration-tests.js: Updated unit tests. (async createTestGroupWithPatch): (createTestGroupWihOwnedCommit): * tools/js/buildbot-syncer.js: (BuildbotSyncer.prototype.scheduleRequestInGroupIfAvailable): (BuildbotSyncer.prototype.pullBuildbot): * tools/js/buildbot-triggerable.js: (BuildbotTriggerable): (BuildbotTriggerable.prototype.async syncOnce): (BuildbotTriggerable.prototype.async _scheduleRequest): (BuildbotTriggerable.prototype._shouldDeferSequentialTestingRequestWithNewCommitSet): * tools/js/measurement-set-analyzer.js: (MeasurementSetAnalyzer.prototype.async _analyzeMeasurementSet): (MeasurementSetAnalyzer): * tools/js/retry-failed-build-requests.js: Removed. * tools/run-analysis.js: Use updated API to retry. (async analysisLoop): (async processTestGroupMayNeedMoreRequests): * tools/sync-buildbot.js: Added command line option '--max-retry-factor'. * unit-tests/analysis-task-tests.js: Updated unit tests. * unit-tests/buildbot-syncer-tests.js: Updated unit tests. (createSampleBuildRequest): * unit-tests/measurement-set-analyzer-tests.js: * unit-tests/retry-failed-build-requests-tests.js: Moved unit tests to test-group-tests.js. * unit-tests/test-groups-tests.js: Added unit tests from 'retry-failed-build-requests-tests.js'. Canonical link: https://commits.webkit.org/238151@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@278072 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2021-05-26 01:23:22 +00:00
{name: '--max-retry-factor', type: parseFloat, default: 3},
New buildbot syncing scripts that supports multiple builders and slaves https://bugs.webkit.org/show_bug.cgi?id=156269 Reviewed by Chris Dumez. Add sync-buildbot.js that supports scheduling A/B testing jobs on multiple builders and slaves. The old python script (sync-with-buildbot.py) could only support a single builder and slave for each platform, test pair. The main logic is implemented in BuildbotTriggerable.syncOnce. Various helper methods are added throughout the codebase and tests have been refactored. BuildbotSyncer has been updated to support multiple platform, test pairs. It's now responsible for syncing everything on each builder (on a buildbot). Added more unit tests for BuildbotSyncer and server tests for BuildbotTriggerable, and refactored test helpers and mocks as needed. * public/v3/models/build-request.js: (BuildRequest.prototype.status): Added. (BuildRequest.prototype.isScheduled): Added. * public/v3/models/metric.js: (Metric.prototype.fullName): Added. * public/v3/models/platform.js: (Platform): Added the map based on platform name. (Platform.findByName): Added. * public/v3/models/test.js: (Test.topLevelTests): (Test.findByPath): Added. Finds a test based on an array of test names; e.g. ['A', 'B'] would find the test whose name is "B" which has a parent test named "A". (Test.prototype.fullName): Added. * server-tests/api-build-requests-tests.js: (addMockData): Moved to resources/mock-data.js. (addAnotherMockTestGroup): Ditto. * server-tests/resources/mock-data.js: Added. (MockData.resetV3Models): Added. (MockData.addMockData): Moved from api-build-requests-tests.js. (MockData.addAnotherMockTestGroup): Ditto. (MockData.mockTestSyncConfigWithSingleBuilder): Added. (MockData.mockTestSyncConfigWithTwoBuilders): Added. (MockData.pendingBuild): Added. (MockData.runningBuild): Added. (MockData.finishedBuild): Added. * server-tests/resources/test-server.js: (TestServer): (TestServer.prototype.remoteAPI): (TestServer.prototype._ensureTestDatabase): Don't fail even if the test database doesn't exit. (TestServer.prototype._startApache): Create a RemoteAPI instance to access the test sever. (TestServer.prototype._waitForPid): Increase the timeout. (TestServer.prototype.inject): Replace global.RemoteAPI during the test and restore it afterwards. * server-tests/tools-buildbot-triggerable-tests.js: Added. Tests BuildbotTriggerable.syncOnce. (MockLogger): Added. (MockLogger.prototype.log): Added. (MockLogger.prototype.error): Added. * tools/detect-changes.js: (parseArgument): Moved to js/parse-arguments.js. * tools/js/buildbot-syncer.js: (BuildbotBuildEntry): (BuildbotBuildEntry.prototype.syncer): Added. (BuildbotBuildEntry.prototype.buildRequestStatusIfUpdateIsNeeded): Added. Returns a new status for a build request (of the matching build request ID) if it needs to be updated in the server. (BuildbotSyncer): This class (BuildbotSyncer.prototype.addTestConfiguration): Added. (BuildbotSyncer.prototype.testConfigurations): Returns the list of test configurations. (BuildbotSyncer.prototype.matchesConfiguration): Returns true iff the request can be scheduled on this builder. (BuildbotSyncer.prototype.scheduleRequest): Added. Schedules a new job on buildbot for a request. (BuildbotSyncer.prototype.scheduleFirstRequestInGroupIfAvailable): Added. Schedules a new job for the specified build request on the first slave that's available. (BuildbotSyncer.prototype.pullBuildbot): Return a list of BuildbotBuildEntry instead of an object. Also store it on an instance variable so that scheduleFirstRequestInGroupIfAvailable could use it. (BuildbotSyncer.prototype._pullRecentBuilds): (BuildbotSyncer.prototype.pathForPendingBuildsJSON): Renamed from urlForPendingBuildsJSON and now only returns the path instead of the full URL since RemoteAPI takes a path, not full URL. (BuildbotSyncer.prototype.pathForBuildJSON): Ditto from pathForBuildJSON. (BuildbotSyncer.prototype.pathForForceBuild): Added. (BuildbotSyncer.prototype.url): Use RemoteAPI's url method instead of manually constructing URL. (BuildbotSyncer.prototype.urlForBuildNumber): Ditto. (BuildbotSyncer.prototype._propertiesForBuildRequest): Now that each syncer can have multiple test configurations associated with it, find the one matching for this request. (BuildbotSyncer._loadConfig): Create a syncer per builder and add all test configurations to it. (BuildbotSyncer._validateAndMergeConfig): Added the support for 'SlaveList', which is a list of slave names present on this builder. * tools/js/buildbot-triggerable.js: Added. (BuildbotTriggerable): Added. (BuildbotTriggerable.prototype.name): Added. (BuildbotTriggerable.prototype.syncOnce): Added. The main logic for the syncing script. It pulls existing build requests from the perf dashboard, pulls buildbot for pending and running/completed builds on each builder (represented by each syncer), schedules build requests on buildbot if there is any builder/slave available, and updates the status of build requests in the database. (BuildbotTriggerable.prototype._validateRequests): Added. (BuildbotTriggerable.prototype._pullBuildbotOnAllSyncers): Added. (BuildbotTriggerable.prototype._scheduleNextRequestInGroupIfSlaveIsAvailable): Added. (BuildbotTriggerable._testGroupMapForBuildRequests): Added. * tools/js/database.js: * tools/js/parse-arguments.js: Added. Extracted out of tools/detect-changes.js. (parseArguments): * tools/js/remote.js: (RemoteAPI): Now optionally takes the server configuration. (RemoteAPI.prototype.url): Added. (RemoteAPI.prototype.getJSON): Removed the code for specifying request content. (RemoteAPI.prototype.getJSONWithStatus): Ditto. (RemoteAPI.prototype.postJSON): Added. (RemoteAPI.prototype.postFormUrlencodedData): Added. (RemoteAPI.prototype.sendHttpRequest): Fixed the code to specify auth. * tools/js/v3-models.js: Don't include RemoteAPI here as they require a configuration for each host. * tools/sync-buildbot.js: Added. (main): Added. Parse the arguments and start the loop. (syncLoop): Added. * unit-tests/buildbot-syncer-tests.js: Added tests for pullBuildbot, scheduleRequest, as well as scheduleFirstRequestInGroupIfAvailable. Refactored helper functions as needed. (sampleiOSConfig): (smallConfiguration): Added. (smallPendingBuild): Added. (smallInProgressBuild): Added. (smallFinishedBuild): Added. (createSampleBuildRequest): Create a unique build request for each platform. (samplePendingBuild): Optionally specify build time and slave name. (sampleInProgressBuild): Optionally specify slave name. (sampleFinishedBuild): Ditto. * unit-tests/resources/mock-remote-api.js: (assert.notReached.assert.notReached): (MockRemoteAPI.url): Added. (MockRemoteAPI.postFormUrlencodedData): Added. (MockRemoteAPI._addRequest): Extracted from getJSONWithStatus. (MockRemoteAPI.waitForRequest): Extracted from inject. For tools-buildbot-triggerable-tests.js, we need to instantiate a RemoteAPI for buildbot without replacing global.RemoteAPI. (MockRemoteAPI.inject): (MockRemoteAPI.reset): Added. Canonical link: https://commits.webkit.org/174395@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@199123 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-04-06 23:19:29 +00:00
]);
if (!options)
return;
syncLoop(options);
}
function syncLoop(options)
{
let serverConfig = JSON.parse(fs.readFileSync(options['--server-config-json'], 'utf8'));
let buildbotConfig = JSON.parse(fs.readFileSync(options['--buildbot-config-json'], 'utf8'));
let buildbotRemote = new RemoteAPI(buildbotConfig.server);
[perf dashboard] Add sequential mode for perf dashboard A/B testing. https://bugs.webkit.org/show_bug.cgi?id=223886 Reviewed by Ryosuke Niwa. Add 'testgroup_repetition_type' to 'analysis_test_groups' table to store repetition type which defaults to 'alternating'. Add support to schedule 'sequential' A/B testing which schedules all iterations for the first configuration (A) before the iterations for the second configuration (B). Before this patch, all A/B testing alternated between two different configurations for each iteration. Add retry logic for 'sequential' A/B testing. Update syncing script to not proceed with next configuration the current configuration successfully completed more iterations than the initially requested or retries exceeded the list. Fix a potential race in syncing script that 'Promise.all' may cause test group not scheduled in order. * browser-tests/test-group-form-tests.js: Updated and added unit tests. * init-database.sql: Added 'testgroup_repetition_type' field to 'analysis_test_groups' table. * migrate-database.sql: Updated migration script to conditionally add 'testgroup_repetition_type' field. * public/api/test-groups.php: Added 'repetitionType' field in api return value. * public/include/build-requests-fetcher.php: Fetching build requests under a triggerable should include test group with all test groups finished but still need retry. This will allow syncing script to resolve the build request by ID and block other request to be scheduled on the builder. * public/include/commit-sets-helpers.php: Added code to create build requests based on test mode. * public/privileged-api/add-build-requests.php: Added support to only add build requests to one commit set. * public/privileged-api/create-analysis-task.php: Added 'repetitionType' support. * public/privileged-api/create-test-group.php: Added 'repetitionType' support. * public/privileged-api/update-test-group.php: * public/shared/common-component-base.js: * public/v3/components/custom-configuration-test-group-form.js: Added 'repetitionType' support. (CustomConfigurationTestGroupForm.prototype.startTesting): * public/v3/components/customizable-test-group-form.js: (CustomizableTestGroupForm.prototype.startTesting): * public/v3/components/test-group-form.js: (TestGroupForm.prototype.startTesting): * public/v3/models/analysis-task.js: (AnalysisTask.async create): (AnalysisTask): * public/v3/models/build-request.js: (BuildRequest): (BuildRequest.prototype.updateSingleton): Allow build request order to be updated. * public/v3/models/test-group.js: (TestGroup): (TestGroup.prototype.updateSingleton): (TestGroup.prototype.repetitionType): (TestGroup.prototype.repetitionCountForCommitSet): Refactored `repetitionCount` since repetition may be different between configurations. (TestGroup.prototype.hasRetries): A helper function to tell if a test group has retry runs. (TestGroup.prototype.additionalRepetitionNeededToReachInitialRepetitionCount): (TestGroup.prototype.successfulTestCount): (TestGroup.prototype.isFirstTestRequest): (TestGroup.prototype.precedingBuildRequest): (TestGroup.prototype.retryCountForCommitSet): (TestGroup.prototype.retryCountsAreSameForAllCommitSets): (TestGroup.prototype.async _createAlternatingRetriesForTestGroup): Moved from 'retry-failed-build-requests' module. (TestGroup.async _createSequentialRetriesForTestGroup): Moved from 'retry-failed-build-requests' module. (TestGroup.prototype.async cancelPendingRequests): (TestGroup.prototype.async didSendNotification): (TestGroup.prototype.async addMoreBuildRequests): (TestGroup.async createWithTask): (TestGroup.async createWithCustomConfiguration): (TestGroup.async createAndRefetchTestGroups): (TestGroup.prototype.async scheduleMoreRequestsOrClearFlag): Moved from 'retry-failed-build-requests' module. (TestGroup.async fetchById): Added support to fetch test group by ID. * public/v3/pages/analysis-task-page.js: (AnalysisTaskChartPane.prototype.didConstructShadowTree): (AnalysisTaskResultsPane.prototype.didConstructShadowTree): (AnalysisTaskConfiguratorPane.prototype.setTestGroups): (AnalysisTaskConfiguratorPane): (AnalysisTaskTestGroupPane.prototype.didConstructShadowTree): (AnalysisTaskTestGroupPane.prototype._renderCurrentTestGroup): (AnalysisTaskTestGroupPane.prototype._retrySummary): (AnalysisTaskPage.prototype.didConstructShadowTree): (AnalysisTaskPage.prototype.async _retryCurrentTestGroup): (AnalysisTaskPage.prototype.async _bisectCurrentTestGroup): (AnalysisTaskPage.set const): (AnalysisTaskPage.prototype.async _createTestGroupAfterVerifyingCommitSetList): (AnalysisTaskPage.prototype.async _createCustomTestGroup): * public/v3/pages/chart-pane.js: (ChartPane.prototype.didConstructShadowTree): (ChartPane.prototype.async _analyzeRange): * public/v3/pages/create-analysis-task-page.js: (CreateAnalysisTaskPage.prototype.async _createAnalysisTaskWithGroup): * server-tests/api-build-requests-tests.js: Added a unit test. * server-tests/api-upload-root-tests.js: Updated unit tests. * server-tests/privileged-api-add-build-requests-tests.js: Updated and added unit tests. * server-tests/privileged-api-create-analysis-task-tests.js: Added unit tests. * server-tests/privileged-api-create-test-group-tests.js: Updated and added unit tests. * server-tests/privileged-api-update-test-group-tests.js: Updated and added unit tests. * server-tests/resources/common-operations.js: (async assertThrows): * server-tests/resources/mock-data.js: (MockData.addMockData): (MockData.set addMockBuildRequestsWithRoots): (MockData.set addAnotherMockTestGroup): (MockData.set mockTestSyncConfigWithSingleBuilder): (MockData.sampleBuildData): * server-tests/tools-buildbot-triggerable-tests.js: Updated and added unit tests. * server-tests/tools-sync-buildbot-integration-tests.js: Updated unit tests. (async createTestGroupWithPatch): (createTestGroupWihOwnedCommit): * tools/js/buildbot-syncer.js: (BuildbotSyncer.prototype.scheduleRequestInGroupIfAvailable): (BuildbotSyncer.prototype.pullBuildbot): * tools/js/buildbot-triggerable.js: (BuildbotTriggerable): (BuildbotTriggerable.prototype.async syncOnce): (BuildbotTriggerable.prototype.async _scheduleRequest): (BuildbotTriggerable.prototype._shouldDeferSequentialTestingRequestWithNewCommitSet): * tools/js/measurement-set-analyzer.js: (MeasurementSetAnalyzer.prototype.async _analyzeMeasurementSet): (MeasurementSetAnalyzer): * tools/js/retry-failed-build-requests.js: Removed. * tools/run-analysis.js: Use updated API to retry. (async analysisLoop): (async processTestGroupMayNeedMoreRequests): * tools/sync-buildbot.js: Added command line option '--max-retry-factor'. * unit-tests/analysis-task-tests.js: Updated unit tests. * unit-tests/buildbot-syncer-tests.js: Updated unit tests. (createSampleBuildRequest): * unit-tests/measurement-set-analyzer-tests.js: * unit-tests/retry-failed-build-requests-tests.js: Moved unit tests to test-group-tests.js. * unit-tests/test-groups-tests.js: Added unit tests from 'retry-failed-build-requests-tests.js'. Canonical link: https://commits.webkit.org/238151@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@278072 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2021-05-26 01:23:22 +00:00
const maxRetryFactor = options['--max-retry-factor'];
New buildbot syncing scripts that supports multiple builders and slaves https://bugs.webkit.org/show_bug.cgi?id=156269 Reviewed by Chris Dumez. Add sync-buildbot.js that supports scheduling A/B testing jobs on multiple builders and slaves. The old python script (sync-with-buildbot.py) could only support a single builder and slave for each platform, test pair. The main logic is implemented in BuildbotTriggerable.syncOnce. Various helper methods are added throughout the codebase and tests have been refactored. BuildbotSyncer has been updated to support multiple platform, test pairs. It's now responsible for syncing everything on each builder (on a buildbot). Added more unit tests for BuildbotSyncer and server tests for BuildbotTriggerable, and refactored test helpers and mocks as needed. * public/v3/models/build-request.js: (BuildRequest.prototype.status): Added. (BuildRequest.prototype.isScheduled): Added. * public/v3/models/metric.js: (Metric.prototype.fullName): Added. * public/v3/models/platform.js: (Platform): Added the map based on platform name. (Platform.findByName): Added. * public/v3/models/test.js: (Test.topLevelTests): (Test.findByPath): Added. Finds a test based on an array of test names; e.g. ['A', 'B'] would find the test whose name is "B" which has a parent test named "A". (Test.prototype.fullName): Added. * server-tests/api-build-requests-tests.js: (addMockData): Moved to resources/mock-data.js. (addAnotherMockTestGroup): Ditto. * server-tests/resources/mock-data.js: Added. (MockData.resetV3Models): Added. (MockData.addMockData): Moved from api-build-requests-tests.js. (MockData.addAnotherMockTestGroup): Ditto. (MockData.mockTestSyncConfigWithSingleBuilder): Added. (MockData.mockTestSyncConfigWithTwoBuilders): Added. (MockData.pendingBuild): Added. (MockData.runningBuild): Added. (MockData.finishedBuild): Added. * server-tests/resources/test-server.js: (TestServer): (TestServer.prototype.remoteAPI): (TestServer.prototype._ensureTestDatabase): Don't fail even if the test database doesn't exit. (TestServer.prototype._startApache): Create a RemoteAPI instance to access the test sever. (TestServer.prototype._waitForPid): Increase the timeout. (TestServer.prototype.inject): Replace global.RemoteAPI during the test and restore it afterwards. * server-tests/tools-buildbot-triggerable-tests.js: Added. Tests BuildbotTriggerable.syncOnce. (MockLogger): Added. (MockLogger.prototype.log): Added. (MockLogger.prototype.error): Added. * tools/detect-changes.js: (parseArgument): Moved to js/parse-arguments.js. * tools/js/buildbot-syncer.js: (BuildbotBuildEntry): (BuildbotBuildEntry.prototype.syncer): Added. (BuildbotBuildEntry.prototype.buildRequestStatusIfUpdateIsNeeded): Added. Returns a new status for a build request (of the matching build request ID) if it needs to be updated in the server. (BuildbotSyncer): This class (BuildbotSyncer.prototype.addTestConfiguration): Added. (BuildbotSyncer.prototype.testConfigurations): Returns the list of test configurations. (BuildbotSyncer.prototype.matchesConfiguration): Returns true iff the request can be scheduled on this builder. (BuildbotSyncer.prototype.scheduleRequest): Added. Schedules a new job on buildbot for a request. (BuildbotSyncer.prototype.scheduleFirstRequestInGroupIfAvailable): Added. Schedules a new job for the specified build request on the first slave that's available. (BuildbotSyncer.prototype.pullBuildbot): Return a list of BuildbotBuildEntry instead of an object. Also store it on an instance variable so that scheduleFirstRequestInGroupIfAvailable could use it. (BuildbotSyncer.prototype._pullRecentBuilds): (BuildbotSyncer.prototype.pathForPendingBuildsJSON): Renamed from urlForPendingBuildsJSON and now only returns the path instead of the full URL since RemoteAPI takes a path, not full URL. (BuildbotSyncer.prototype.pathForBuildJSON): Ditto from pathForBuildJSON. (BuildbotSyncer.prototype.pathForForceBuild): Added. (BuildbotSyncer.prototype.url): Use RemoteAPI's url method instead of manually constructing URL. (BuildbotSyncer.prototype.urlForBuildNumber): Ditto. (BuildbotSyncer.prototype._propertiesForBuildRequest): Now that each syncer can have multiple test configurations associated with it, find the one matching for this request. (BuildbotSyncer._loadConfig): Create a syncer per builder and add all test configurations to it. (BuildbotSyncer._validateAndMergeConfig): Added the support for 'SlaveList', which is a list of slave names present on this builder. * tools/js/buildbot-triggerable.js: Added. (BuildbotTriggerable): Added. (BuildbotTriggerable.prototype.name): Added. (BuildbotTriggerable.prototype.syncOnce): Added. The main logic for the syncing script. It pulls existing build requests from the perf dashboard, pulls buildbot for pending and running/completed builds on each builder (represented by each syncer), schedules build requests on buildbot if there is any builder/slave available, and updates the status of build requests in the database. (BuildbotTriggerable.prototype._validateRequests): Added. (BuildbotTriggerable.prototype._pullBuildbotOnAllSyncers): Added. (BuildbotTriggerable.prototype._scheduleNextRequestInGroupIfSlaveIsAvailable): Added. (BuildbotTriggerable._testGroupMapForBuildRequests): Added. * tools/js/database.js: * tools/js/parse-arguments.js: Added. Extracted out of tools/detect-changes.js. (parseArguments): * tools/js/remote.js: (RemoteAPI): Now optionally takes the server configuration. (RemoteAPI.prototype.url): Added. (RemoteAPI.prototype.getJSON): Removed the code for specifying request content. (RemoteAPI.prototype.getJSONWithStatus): Ditto. (RemoteAPI.prototype.postJSON): Added. (RemoteAPI.prototype.postFormUrlencodedData): Added. (RemoteAPI.prototype.sendHttpRequest): Fixed the code to specify auth. * tools/js/v3-models.js: Don't include RemoteAPI here as they require a configuration for each host. * tools/sync-buildbot.js: Added. (main): Added. Parse the arguments and start the loop. (syncLoop): Added. * unit-tests/buildbot-syncer-tests.js: Added tests for pullBuildbot, scheduleRequest, as well as scheduleFirstRequestInGroupIfAvailable. Refactored helper functions as needed. (sampleiOSConfig): (smallConfiguration): Added. (smallPendingBuild): Added. (smallInProgressBuild): Added. (smallFinishedBuild): Added. (createSampleBuildRequest): Create a unique build request for each platform. (samplePendingBuild): Optionally specify build time and slave name. (sampleInProgressBuild): Optionally specify slave name. (sampleFinishedBuild): Ditto. * unit-tests/resources/mock-remote-api.js: (assert.notReached.assert.notReached): (MockRemoteAPI.url): Added. (MockRemoteAPI.postFormUrlencodedData): Added. (MockRemoteAPI._addRequest): Extracted from getJSONWithStatus. (MockRemoteAPI.waitForRequest): Extracted from inject. For tools-buildbot-triggerable-tests.js, we need to instantiate a RemoteAPI for buildbot without replacing global.RemoteAPI. (MockRemoteAPI.inject): (MockRemoteAPI.reset): Added. Canonical link: https://commits.webkit.org/174395@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@199123 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-04-06 23:19:29 +00:00
// v3 models use the global RemoteAPI to access the perf dashboard.
global.RemoteAPI = new RemoteAPI(serverConfig.server);
console.log(`Fetching the manifest...`);
sync-buildbot.js should update the list of tests and platforms associated with a triggerable https://bugs.webkit.org/show_bug.cgi?id=158406 <rdar://problem/26185737> Reviewed by Darin Adler. Added /api/update-triggerable to update the list of configurations (platform and test pairs) associated with a given triggerable, and make sync-buildbot.js use this JSON API before each syncing cycle so that the association gets updated automatically by simply updating the JSON. * server-tests/api-manifest.js: Use const for imported modules. * server-tests/api-report-commits-tests.js: Removed unnecessary importing of crypto. * server-tests/resources/mock-data.js: (MockData.someTestId): Added. (MockData.somePlatformId): Added. (MockData.addMockData): * server-tests/tools-buildbot-triggerable-tests.js: Use const for imported modules. Also added a test for BuildbotTriggerable's updateTriggerable. * tools/js/buildbot-triggerable.js: (BuildbotTriggerable.prototype.updateTriggerable): Added. Find the list of all configurations associated with this triggeerable and post it to /api/update-triggerable. * tools/js/database.js: Added triggerable_configurations to the list of tables. * tools/js/remote.js: (RemoteAPI.prototype.postJSON): Print the whole response when JSON parsing fails for debugging. * tools/sync-buildbot.js: (syncLoop): Call BuildbotTriggerable's updateTriggerable before syncing. Canonical link: https://commits.webkit.org/176504@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@201718 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-06-06 18:51:23 +00:00
Add the support for perf try bots to sync-buildbot.js https://bugs.webkit.org/show_bug.cgi?id=172529 Rubber-stamped by Chris Dumez. Make sync-buildbot.js schedule an A/B testing job with a patch or roots to buildbot. Change the buildbot property format in the syncing script's configuration again to use a dictionary with a single key of "revision", "patch", or "roots" to specify a revision, a patch, or a set of roots, and simplified the structure of the configuration by always having "types" and "builders", and make each entry in "configurations" refer to a list of types, platforms, and builders. Since now there are build requests to build patches and run tests, "configurations" has been renamed to "testConfigurations" and "buildConfigurations" have been added. Each entry in "buildConfigurations" specifies a list of platforms and builders. Similarly in repository group configurations, the buildbot properties for testing is now specified as "testProperties" and ones for building a patch is specified in newly introduced "buildProperties". * public/api/build-requests.php: (update_builds): When a build request to build a patch fails, mark all subsequent requests as failed since there is no way to run tests without a successful build. * public/api/update-triggerable.php: (main): Re-generate manifest.json after updating the triggerable. The lack of this re-generation was the reason we had to manually GET /api/manifest in api-update-triggerable-tests.js. * public/v3/models/build-request.js: (BuildRequest.prototype.hasCompleted): Added. * public/v3/models/manifest.js: (Manifest.reset): Added. Extracted from MockData.resetV3Models in unit-tests/mock-data.js and syncLoop in tools/sync-buildbot.js (Manifest.fetch): Reset V3 models before fetching the manifest. This eliminates the need to manually reset V3 models in syncLoop. * public/v3/models/uploaded-file.js: (UploadedFile.prototype.url): Use RemoteAPI.url to get the full URL instead of just a path. * public/v3/remote.js: (BrowserRemoteAPI.prototype.url): Added. Constructs the full URL. * server-tests/api-update-triggerable-tests.js: (.refetchManifest): Deleted. Now that /api/manifest re-generates manifest.json, we can simply call Manifest.fetch instead. * server-tests/resources/mock-data.js: (MockData.resetV3Models): Calls Manifest.reset(). (MockData.addMockConfiguration): Extracted from addMockData. (MockData.addMockData): Updated per the format change. (MockData.mockTestSyncConfigWithSingleBuilder): Ditto. (MockData.mockTestSyncConfigWithTwoBuilders): Ditto. (MockData.runningBuild): Make buildNumber specifiable. (MockData.finishedBuild): Ditto. * server-tests/tools-buildbot-triggerable-tests.js: Updated configurations per the format change. Now that now acceptsCustomRoots() for "system-and-webkit" must be true since we can't have a repository group that which accepts a patch and not take roots. * server-tests/tools-sync-buildbot-integration-tests.js: Added. (createTriggerable): Added. (createTestGroupWihPatch): Added. (uploadRoot): Added. (.assertAndResolveRequest): Added. (.assertTestBuildHasFailed): Added. * tools/js/buildbot-syncer.js: (BuildbotSyncer): Added. _type as an instance variable to identify whether this buildbot builder is a "builder" which builds a patch, builder, or a "tester" which runs a test. Also renamed _testConfigurations to _configurations. (BuildbotSyncer.prototype.addTestConfiguration): Assert that either the type of this syncer hasn't been set or it's a tester. (BuildbotSyncer.prototype.testConfigurations): Return [] when it's a builder. (BuildbotSyncer.prototype.addBuildConfiguration): Added. Adds a platform to a builder. (BuildbotSyncer.prototype.buildConfigurations): Added. Returns the list of configurations if this syncer is a builder. Otherwise returns []. (BuildbotSyncer.prototype.isTester): Added. (BuildbotSyncer.prototype.matchesConfiguration): (BuildbotSyncer.prototype._propertiesForBuildRequest): Updated to support the new format. (BuildbotSyncer._loadConfig): Ditto. Optionally parse buildConfigurations. (BuildbotSyncer._resolveBuildersWithPlatforms): Added. For each test or build configuration entry, creates the list of configurations per builder and platform. (BuildbotSyncer._parseRepositoryGroup): Added the support for parsing the new format with revision, roots, and patch option types with a lot of validations as we're seeing a bit of combinatorial explosion in the number of things that can go wrong. Also parse buildProperties optionally. (BuildbotSyncer._parseRepositoryGroupPropertyTemplate): Added. A helper function to parse a set of buildbot properties, validates its content, and invokes a callback if it's an dynamically resolved type such as "revision" and "patch". (BuildbotSyncer._validateAndMergeConfig): Updated per the format change. No longer allows "types", "type", "platforms", and "platform" as they're explicity resolved in _resolveBuildersWithPlatforms. * tools/js/buildbot-triggerable.js: (BuildbotTriggerable.prototype.syncOnce): (BuildbotTriggerable.prototype._validateRequests): Handle the case when a build request is not associated with any test. (BuildbotTriggerable.prototype._nextRequestInGroup): Return null when there is a build request to build a patch which has not been completed (pending, scheduled, running, or failed). Since all requests to build a patch has a negative order, those requests should all show up at the beginning. (BuildbotTriggerable.prototype._scheduleRequestIfSlaveIsAvailable): Pick a new buildbot syncer when scheduling the first request to build a patch or the first request to run a test. The first request to run a test will always have order of 0, so it's a sufficient condition to find such a request. On the other hand, the first request to build a patch can have a negative order number so we must explicitly check if it's the first item in the ordered list of requests in the test group. * tools/remote-server-relay.log: Added. * tools/sync-buildbot.js: (syncLoop): Fixed a bug we were not re-fetching the triggerable after updating the triggerable so that Triggerable and related objects we have in the memory may not reflect what we just synced to the perf dashboard. Also, we don't reset V3 models manually any more since Manifest.fetch does that. * unit-tests/buildbot-syncer-tests.js: Added more test cases and updated existing test cases to test exception messages explicitly since allowing any exception was resulting in some tests passing a result of unrelated parsing error being thrown, etc... (sampleiOSConfig): Updated per the format change. (sampleiOSConfigWithExpansions): Ditto. (smallConfiguration): Ditto. Canonical link: https://commits.webkit.org/189454@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@217378 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-24 18:47:01 +00:00
const makeTriggerable = function () {
[perf dashboard] Add sequential mode for perf dashboard A/B testing. https://bugs.webkit.org/show_bug.cgi?id=223886 Reviewed by Ryosuke Niwa. Add 'testgroup_repetition_type' to 'analysis_test_groups' table to store repetition type which defaults to 'alternating'. Add support to schedule 'sequential' A/B testing which schedules all iterations for the first configuration (A) before the iterations for the second configuration (B). Before this patch, all A/B testing alternated between two different configurations for each iteration. Add retry logic for 'sequential' A/B testing. Update syncing script to not proceed with next configuration the current configuration successfully completed more iterations than the initially requested or retries exceeded the list. Fix a potential race in syncing script that 'Promise.all' may cause test group not scheduled in order. * browser-tests/test-group-form-tests.js: Updated and added unit tests. * init-database.sql: Added 'testgroup_repetition_type' field to 'analysis_test_groups' table. * migrate-database.sql: Updated migration script to conditionally add 'testgroup_repetition_type' field. * public/api/test-groups.php: Added 'repetitionType' field in api return value. * public/include/build-requests-fetcher.php: Fetching build requests under a triggerable should include test group with all test groups finished but still need retry. This will allow syncing script to resolve the build request by ID and block other request to be scheduled on the builder. * public/include/commit-sets-helpers.php: Added code to create build requests based on test mode. * public/privileged-api/add-build-requests.php: Added support to only add build requests to one commit set. * public/privileged-api/create-analysis-task.php: Added 'repetitionType' support. * public/privileged-api/create-test-group.php: Added 'repetitionType' support. * public/privileged-api/update-test-group.php: * public/shared/common-component-base.js: * public/v3/components/custom-configuration-test-group-form.js: Added 'repetitionType' support. (CustomConfigurationTestGroupForm.prototype.startTesting): * public/v3/components/customizable-test-group-form.js: (CustomizableTestGroupForm.prototype.startTesting): * public/v3/components/test-group-form.js: (TestGroupForm.prototype.startTesting): * public/v3/models/analysis-task.js: (AnalysisTask.async create): (AnalysisTask): * public/v3/models/build-request.js: (BuildRequest): (BuildRequest.prototype.updateSingleton): Allow build request order to be updated. * public/v3/models/test-group.js: (TestGroup): (TestGroup.prototype.updateSingleton): (TestGroup.prototype.repetitionType): (TestGroup.prototype.repetitionCountForCommitSet): Refactored `repetitionCount` since repetition may be different between configurations. (TestGroup.prototype.hasRetries): A helper function to tell if a test group has retry runs. (TestGroup.prototype.additionalRepetitionNeededToReachInitialRepetitionCount): (TestGroup.prototype.successfulTestCount): (TestGroup.prototype.isFirstTestRequest): (TestGroup.prototype.precedingBuildRequest): (TestGroup.prototype.retryCountForCommitSet): (TestGroup.prototype.retryCountsAreSameForAllCommitSets): (TestGroup.prototype.async _createAlternatingRetriesForTestGroup): Moved from 'retry-failed-build-requests' module. (TestGroup.async _createSequentialRetriesForTestGroup): Moved from 'retry-failed-build-requests' module. (TestGroup.prototype.async cancelPendingRequests): (TestGroup.prototype.async didSendNotification): (TestGroup.prototype.async addMoreBuildRequests): (TestGroup.async createWithTask): (TestGroup.async createWithCustomConfiguration): (TestGroup.async createAndRefetchTestGroups): (TestGroup.prototype.async scheduleMoreRequestsOrClearFlag): Moved from 'retry-failed-build-requests' module. (TestGroup.async fetchById): Added support to fetch test group by ID. * public/v3/pages/analysis-task-page.js: (AnalysisTaskChartPane.prototype.didConstructShadowTree): (AnalysisTaskResultsPane.prototype.didConstructShadowTree): (AnalysisTaskConfiguratorPane.prototype.setTestGroups): (AnalysisTaskConfiguratorPane): (AnalysisTaskTestGroupPane.prototype.didConstructShadowTree): (AnalysisTaskTestGroupPane.prototype._renderCurrentTestGroup): (AnalysisTaskTestGroupPane.prototype._retrySummary): (AnalysisTaskPage.prototype.didConstructShadowTree): (AnalysisTaskPage.prototype.async _retryCurrentTestGroup): (AnalysisTaskPage.prototype.async _bisectCurrentTestGroup): (AnalysisTaskPage.set const): (AnalysisTaskPage.prototype.async _createTestGroupAfterVerifyingCommitSetList): (AnalysisTaskPage.prototype.async _createCustomTestGroup): * public/v3/pages/chart-pane.js: (ChartPane.prototype.didConstructShadowTree): (ChartPane.prototype.async _analyzeRange): * public/v3/pages/create-analysis-task-page.js: (CreateAnalysisTaskPage.prototype.async _createAnalysisTaskWithGroup): * server-tests/api-build-requests-tests.js: Added a unit test. * server-tests/api-upload-root-tests.js: Updated unit tests. * server-tests/privileged-api-add-build-requests-tests.js: Updated and added unit tests. * server-tests/privileged-api-create-analysis-task-tests.js: Added unit tests. * server-tests/privileged-api-create-test-group-tests.js: Updated and added unit tests. * server-tests/privileged-api-update-test-group-tests.js: Updated and added unit tests. * server-tests/resources/common-operations.js: (async assertThrows): * server-tests/resources/mock-data.js: (MockData.addMockData): (MockData.set addMockBuildRequestsWithRoots): (MockData.set addAnotherMockTestGroup): (MockData.set mockTestSyncConfigWithSingleBuilder): (MockData.sampleBuildData): * server-tests/tools-buildbot-triggerable-tests.js: Updated and added unit tests. * server-tests/tools-sync-buildbot-integration-tests.js: Updated unit tests. (async createTestGroupWithPatch): (createTestGroupWihOwnedCommit): * tools/js/buildbot-syncer.js: (BuildbotSyncer.prototype.scheduleRequestInGroupIfAvailable): (BuildbotSyncer.prototype.pullBuildbot): * tools/js/buildbot-triggerable.js: (BuildbotTriggerable): (BuildbotTriggerable.prototype.async syncOnce): (BuildbotTriggerable.prototype.async _scheduleRequest): (BuildbotTriggerable.prototype._shouldDeferSequentialTestingRequestWithNewCommitSet): * tools/js/measurement-set-analyzer.js: (MeasurementSetAnalyzer.prototype.async _analyzeMeasurementSet): (MeasurementSetAnalyzer): * tools/js/retry-failed-build-requests.js: Removed. * tools/run-analysis.js: Use updated API to retry. (async analysisLoop): (async processTestGroupMayNeedMoreRequests): * tools/sync-buildbot.js: Added command line option '--max-retry-factor'. * unit-tests/analysis-task-tests.js: Updated unit tests. * unit-tests/buildbot-syncer-tests.js: Updated unit tests. (createSampleBuildRequest): * unit-tests/measurement-set-analyzer-tests.js: * unit-tests/retry-failed-build-requests-tests.js: Moved unit tests to test-group-tests.js. * unit-tests/test-groups-tests.js: Added unit tests from 'retry-failed-build-requests-tests.js'. Canonical link: https://commits.webkit.org/238151@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@278072 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2021-05-26 01:23:22 +00:00
return new BuildbotTriggerable(buildbotConfig, global.RemoteAPI, buildbotRemote, serverConfig.worker, maxRetryFactor, console)
};
Add the support for perf try bots to sync-buildbot.js https://bugs.webkit.org/show_bug.cgi?id=172529 Rubber-stamped by Chris Dumez. Make sync-buildbot.js schedule an A/B testing job with a patch or roots to buildbot. Change the buildbot property format in the syncing script's configuration again to use a dictionary with a single key of "revision", "patch", or "roots" to specify a revision, a patch, or a set of roots, and simplified the structure of the configuration by always having "types" and "builders", and make each entry in "configurations" refer to a list of types, platforms, and builders. Since now there are build requests to build patches and run tests, "configurations" has been renamed to "testConfigurations" and "buildConfigurations" have been added. Each entry in "buildConfigurations" specifies a list of platforms and builders. Similarly in repository group configurations, the buildbot properties for testing is now specified as "testProperties" and ones for building a patch is specified in newly introduced "buildProperties". * public/api/build-requests.php: (update_builds): When a build request to build a patch fails, mark all subsequent requests as failed since there is no way to run tests without a successful build. * public/api/update-triggerable.php: (main): Re-generate manifest.json after updating the triggerable. The lack of this re-generation was the reason we had to manually GET /api/manifest in api-update-triggerable-tests.js. * public/v3/models/build-request.js: (BuildRequest.prototype.hasCompleted): Added. * public/v3/models/manifest.js: (Manifest.reset): Added. Extracted from MockData.resetV3Models in unit-tests/mock-data.js and syncLoop in tools/sync-buildbot.js (Manifest.fetch): Reset V3 models before fetching the manifest. This eliminates the need to manually reset V3 models in syncLoop. * public/v3/models/uploaded-file.js: (UploadedFile.prototype.url): Use RemoteAPI.url to get the full URL instead of just a path. * public/v3/remote.js: (BrowserRemoteAPI.prototype.url): Added. Constructs the full URL. * server-tests/api-update-triggerable-tests.js: (.refetchManifest): Deleted. Now that /api/manifest re-generates manifest.json, we can simply call Manifest.fetch instead. * server-tests/resources/mock-data.js: (MockData.resetV3Models): Calls Manifest.reset(). (MockData.addMockConfiguration): Extracted from addMockData. (MockData.addMockData): Updated per the format change. (MockData.mockTestSyncConfigWithSingleBuilder): Ditto. (MockData.mockTestSyncConfigWithTwoBuilders): Ditto. (MockData.runningBuild): Make buildNumber specifiable. (MockData.finishedBuild): Ditto. * server-tests/tools-buildbot-triggerable-tests.js: Updated configurations per the format change. Now that now acceptsCustomRoots() for "system-and-webkit" must be true since we can't have a repository group that which accepts a patch and not take roots. * server-tests/tools-sync-buildbot-integration-tests.js: Added. (createTriggerable): Added. (createTestGroupWihPatch): Added. (uploadRoot): Added. (.assertAndResolveRequest): Added. (.assertTestBuildHasFailed): Added. * tools/js/buildbot-syncer.js: (BuildbotSyncer): Added. _type as an instance variable to identify whether this buildbot builder is a "builder" which builds a patch, builder, or a "tester" which runs a test. Also renamed _testConfigurations to _configurations. (BuildbotSyncer.prototype.addTestConfiguration): Assert that either the type of this syncer hasn't been set or it's a tester. (BuildbotSyncer.prototype.testConfigurations): Return [] when it's a builder. (BuildbotSyncer.prototype.addBuildConfiguration): Added. Adds a platform to a builder. (BuildbotSyncer.prototype.buildConfigurations): Added. Returns the list of configurations if this syncer is a builder. Otherwise returns []. (BuildbotSyncer.prototype.isTester): Added. (BuildbotSyncer.prototype.matchesConfiguration): (BuildbotSyncer.prototype._propertiesForBuildRequest): Updated to support the new format. (BuildbotSyncer._loadConfig): Ditto. Optionally parse buildConfigurations. (BuildbotSyncer._resolveBuildersWithPlatforms): Added. For each test or build configuration entry, creates the list of configurations per builder and platform. (BuildbotSyncer._parseRepositoryGroup): Added the support for parsing the new format with revision, roots, and patch option types with a lot of validations as we're seeing a bit of combinatorial explosion in the number of things that can go wrong. Also parse buildProperties optionally. (BuildbotSyncer._parseRepositoryGroupPropertyTemplate): Added. A helper function to parse a set of buildbot properties, validates its content, and invokes a callback if it's an dynamically resolved type such as "revision" and "patch". (BuildbotSyncer._validateAndMergeConfig): Updated per the format change. No longer allows "types", "type", "platforms", and "platform" as they're explicity resolved in _resolveBuildersWithPlatforms. * tools/js/buildbot-triggerable.js: (BuildbotTriggerable.prototype.syncOnce): (BuildbotTriggerable.prototype._validateRequests): Handle the case when a build request is not associated with any test. (BuildbotTriggerable.prototype._nextRequestInGroup): Return null when there is a build request to build a patch which has not been completed (pending, scheduled, running, or failed). Since all requests to build a patch has a negative order, those requests should all show up at the beginning. (BuildbotTriggerable.prototype._scheduleRequestIfSlaveIsAvailable): Pick a new buildbot syncer when scheduling the first request to build a patch or the first request to run a test. The first request to run a test will always have order of 0, so it's a sufficient condition to find such a request. On the other hand, the first request to build a patch can have a negative order number so we must explicitly check if it's the first item in the ordered list of requests in the test group. * tools/remote-server-relay.log: Added. * tools/sync-buildbot.js: (syncLoop): Fixed a bug we were not re-fetching the triggerable after updating the triggerable so that Triggerable and related objects we have in the memory may not reflect what we just synced to the perf dashboard. Also, we don't reset V3 models manually any more since Manifest.fetch does that. * unit-tests/buildbot-syncer-tests.js: Added more test cases and updated existing test cases to test exception messages explicitly since allowing any exception was resulting in some tests passing a result of unrelated parsing error being thrown, etc... (sampleiOSConfig): Updated per the format change. (sampleiOSConfigWithExpansions): Ditto. (smallConfiguration): Ditto. Canonical link: https://commits.webkit.org/189454@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@217378 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-24 18:47:01 +00:00
Manifest.fetch().then(() => {
const triggerable = makeTriggerable();
return triggerable.initSyncers().then(() => triggerable.updateTriggerable());
Add the support for perf try bots to sync-buildbot.js https://bugs.webkit.org/show_bug.cgi?id=172529 Rubber-stamped by Chris Dumez. Make sync-buildbot.js schedule an A/B testing job with a patch or roots to buildbot. Change the buildbot property format in the syncing script's configuration again to use a dictionary with a single key of "revision", "patch", or "roots" to specify a revision, a patch, or a set of roots, and simplified the structure of the configuration by always having "types" and "builders", and make each entry in "configurations" refer to a list of types, platforms, and builders. Since now there are build requests to build patches and run tests, "configurations" has been renamed to "testConfigurations" and "buildConfigurations" have been added. Each entry in "buildConfigurations" specifies a list of platforms and builders. Similarly in repository group configurations, the buildbot properties for testing is now specified as "testProperties" and ones for building a patch is specified in newly introduced "buildProperties". * public/api/build-requests.php: (update_builds): When a build request to build a patch fails, mark all subsequent requests as failed since there is no way to run tests without a successful build. * public/api/update-triggerable.php: (main): Re-generate manifest.json after updating the triggerable. The lack of this re-generation was the reason we had to manually GET /api/manifest in api-update-triggerable-tests.js. * public/v3/models/build-request.js: (BuildRequest.prototype.hasCompleted): Added. * public/v3/models/manifest.js: (Manifest.reset): Added. Extracted from MockData.resetV3Models in unit-tests/mock-data.js and syncLoop in tools/sync-buildbot.js (Manifest.fetch): Reset V3 models before fetching the manifest. This eliminates the need to manually reset V3 models in syncLoop. * public/v3/models/uploaded-file.js: (UploadedFile.prototype.url): Use RemoteAPI.url to get the full URL instead of just a path. * public/v3/remote.js: (BrowserRemoteAPI.prototype.url): Added. Constructs the full URL. * server-tests/api-update-triggerable-tests.js: (.refetchManifest): Deleted. Now that /api/manifest re-generates manifest.json, we can simply call Manifest.fetch instead. * server-tests/resources/mock-data.js: (MockData.resetV3Models): Calls Manifest.reset(). (MockData.addMockConfiguration): Extracted from addMockData. (MockData.addMockData): Updated per the format change. (MockData.mockTestSyncConfigWithSingleBuilder): Ditto. (MockData.mockTestSyncConfigWithTwoBuilders): Ditto. (MockData.runningBuild): Make buildNumber specifiable. (MockData.finishedBuild): Ditto. * server-tests/tools-buildbot-triggerable-tests.js: Updated configurations per the format change. Now that now acceptsCustomRoots() for "system-and-webkit" must be true since we can't have a repository group that which accepts a patch and not take roots. * server-tests/tools-sync-buildbot-integration-tests.js: Added. (createTriggerable): Added. (createTestGroupWihPatch): Added. (uploadRoot): Added. (.assertAndResolveRequest): Added. (.assertTestBuildHasFailed): Added. * tools/js/buildbot-syncer.js: (BuildbotSyncer): Added. _type as an instance variable to identify whether this buildbot builder is a "builder" which builds a patch, builder, or a "tester" which runs a test. Also renamed _testConfigurations to _configurations. (BuildbotSyncer.prototype.addTestConfiguration): Assert that either the type of this syncer hasn't been set or it's a tester. (BuildbotSyncer.prototype.testConfigurations): Return [] when it's a builder. (BuildbotSyncer.prototype.addBuildConfiguration): Added. Adds a platform to a builder. (BuildbotSyncer.prototype.buildConfigurations): Added. Returns the list of configurations if this syncer is a builder. Otherwise returns []. (BuildbotSyncer.prototype.isTester): Added. (BuildbotSyncer.prototype.matchesConfiguration): (BuildbotSyncer.prototype._propertiesForBuildRequest): Updated to support the new format. (BuildbotSyncer._loadConfig): Ditto. Optionally parse buildConfigurations. (BuildbotSyncer._resolveBuildersWithPlatforms): Added. For each test or build configuration entry, creates the list of configurations per builder and platform. (BuildbotSyncer._parseRepositoryGroup): Added the support for parsing the new format with revision, roots, and patch option types with a lot of validations as we're seeing a bit of combinatorial explosion in the number of things that can go wrong. Also parse buildProperties optionally. (BuildbotSyncer._parseRepositoryGroupPropertyTemplate): Added. A helper function to parse a set of buildbot properties, validates its content, and invokes a callback if it's an dynamically resolved type such as "revision" and "patch". (BuildbotSyncer._validateAndMergeConfig): Updated per the format change. No longer allows "types", "type", "platforms", and "platform" as they're explicity resolved in _resolveBuildersWithPlatforms. * tools/js/buildbot-triggerable.js: (BuildbotTriggerable.prototype.syncOnce): (BuildbotTriggerable.prototype._validateRequests): Handle the case when a build request is not associated with any test. (BuildbotTriggerable.prototype._nextRequestInGroup): Return null when there is a build request to build a patch which has not been completed (pending, scheduled, running, or failed). Since all requests to build a patch has a negative order, those requests should all show up at the beginning. (BuildbotTriggerable.prototype._scheduleRequestIfSlaveIsAvailable): Pick a new buildbot syncer when scheduling the first request to build a patch or the first request to run a test. The first request to run a test will always have order of 0, so it's a sufficient condition to find such a request. On the other hand, the first request to build a patch can have a negative order number so we must explicitly check if it's the first item in the ordered list of requests in the test group. * tools/remote-server-relay.log: Added. * tools/sync-buildbot.js: (syncLoop): Fixed a bug we were not re-fetching the triggerable after updating the triggerable so that Triggerable and related objects we have in the memory may not reflect what we just synced to the perf dashboard. Also, we don't reset V3 models manually any more since Manifest.fetch does that. * unit-tests/buildbot-syncer-tests.js: Added more test cases and updated existing test cases to test exception messages explicitly since allowing any exception was resulting in some tests passing a result of unrelated parsing error being thrown, etc... (sampleiOSConfig): Updated per the format change. (sampleiOSConfigWithExpansions): Ditto. (smallConfiguration): Ditto. Canonical link: https://commits.webkit.org/189454@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@217378 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-24 18:47:01 +00:00
}).then(() => {
return Manifest.fetch();
}).then(() => {
const triggerable = makeTriggerable();
return triggerable.initSyncers().then(() => triggerable.syncOnce());
Add the support for perf try bots to sync-buildbot.js https://bugs.webkit.org/show_bug.cgi?id=172529 Rubber-stamped by Chris Dumez. Make sync-buildbot.js schedule an A/B testing job with a patch or roots to buildbot. Change the buildbot property format in the syncing script's configuration again to use a dictionary with a single key of "revision", "patch", or "roots" to specify a revision, a patch, or a set of roots, and simplified the structure of the configuration by always having "types" and "builders", and make each entry in "configurations" refer to a list of types, platforms, and builders. Since now there are build requests to build patches and run tests, "configurations" has been renamed to "testConfigurations" and "buildConfigurations" have been added. Each entry in "buildConfigurations" specifies a list of platforms and builders. Similarly in repository group configurations, the buildbot properties for testing is now specified as "testProperties" and ones for building a patch is specified in newly introduced "buildProperties". * public/api/build-requests.php: (update_builds): When a build request to build a patch fails, mark all subsequent requests as failed since there is no way to run tests without a successful build. * public/api/update-triggerable.php: (main): Re-generate manifest.json after updating the triggerable. The lack of this re-generation was the reason we had to manually GET /api/manifest in api-update-triggerable-tests.js. * public/v3/models/build-request.js: (BuildRequest.prototype.hasCompleted): Added. * public/v3/models/manifest.js: (Manifest.reset): Added. Extracted from MockData.resetV3Models in unit-tests/mock-data.js and syncLoop in tools/sync-buildbot.js (Manifest.fetch): Reset V3 models before fetching the manifest. This eliminates the need to manually reset V3 models in syncLoop. * public/v3/models/uploaded-file.js: (UploadedFile.prototype.url): Use RemoteAPI.url to get the full URL instead of just a path. * public/v3/remote.js: (BrowserRemoteAPI.prototype.url): Added. Constructs the full URL. * server-tests/api-update-triggerable-tests.js: (.refetchManifest): Deleted. Now that /api/manifest re-generates manifest.json, we can simply call Manifest.fetch instead. * server-tests/resources/mock-data.js: (MockData.resetV3Models): Calls Manifest.reset(). (MockData.addMockConfiguration): Extracted from addMockData. (MockData.addMockData): Updated per the format change. (MockData.mockTestSyncConfigWithSingleBuilder): Ditto. (MockData.mockTestSyncConfigWithTwoBuilders): Ditto. (MockData.runningBuild): Make buildNumber specifiable. (MockData.finishedBuild): Ditto. * server-tests/tools-buildbot-triggerable-tests.js: Updated configurations per the format change. Now that now acceptsCustomRoots() for "system-and-webkit" must be true since we can't have a repository group that which accepts a patch and not take roots. * server-tests/tools-sync-buildbot-integration-tests.js: Added. (createTriggerable): Added. (createTestGroupWihPatch): Added. (uploadRoot): Added. (.assertAndResolveRequest): Added. (.assertTestBuildHasFailed): Added. * tools/js/buildbot-syncer.js: (BuildbotSyncer): Added. _type as an instance variable to identify whether this buildbot builder is a "builder" which builds a patch, builder, or a "tester" which runs a test. Also renamed _testConfigurations to _configurations. (BuildbotSyncer.prototype.addTestConfiguration): Assert that either the type of this syncer hasn't been set or it's a tester. (BuildbotSyncer.prototype.testConfigurations): Return [] when it's a builder. (BuildbotSyncer.prototype.addBuildConfiguration): Added. Adds a platform to a builder. (BuildbotSyncer.prototype.buildConfigurations): Added. Returns the list of configurations if this syncer is a builder. Otherwise returns []. (BuildbotSyncer.prototype.isTester): Added. (BuildbotSyncer.prototype.matchesConfiguration): (BuildbotSyncer.prototype._propertiesForBuildRequest): Updated to support the new format. (BuildbotSyncer._loadConfig): Ditto. Optionally parse buildConfigurations. (BuildbotSyncer._resolveBuildersWithPlatforms): Added. For each test or build configuration entry, creates the list of configurations per builder and platform. (BuildbotSyncer._parseRepositoryGroup): Added the support for parsing the new format with revision, roots, and patch option types with a lot of validations as we're seeing a bit of combinatorial explosion in the number of things that can go wrong. Also parse buildProperties optionally. (BuildbotSyncer._parseRepositoryGroupPropertyTemplate): Added. A helper function to parse a set of buildbot properties, validates its content, and invokes a callback if it's an dynamically resolved type such as "revision" and "patch". (BuildbotSyncer._validateAndMergeConfig): Updated per the format change. No longer allows "types", "type", "platforms", and "platform" as they're explicity resolved in _resolveBuildersWithPlatforms. * tools/js/buildbot-triggerable.js: (BuildbotTriggerable.prototype.syncOnce): (BuildbotTriggerable.prototype._validateRequests): Handle the case when a build request is not associated with any test. (BuildbotTriggerable.prototype._nextRequestInGroup): Return null when there is a build request to build a patch which has not been completed (pending, scheduled, running, or failed). Since all requests to build a patch has a negative order, those requests should all show up at the beginning. (BuildbotTriggerable.prototype._scheduleRequestIfSlaveIsAvailable): Pick a new buildbot syncer when scheduling the first request to build a patch or the first request to run a test. The first request to run a test will always have order of 0, so it's a sufficient condition to find such a request. On the other hand, the first request to build a patch can have a negative order number so we must explicitly check if it's the first item in the ordered list of requests in the test group. * tools/remote-server-relay.log: Added. * tools/sync-buildbot.js: (syncLoop): Fixed a bug we were not re-fetching the triggerable after updating the triggerable so that Triggerable and related objects we have in the memory may not reflect what we just synced to the perf dashboard. Also, we don't reset V3 models manually any more since Manifest.fetch does that. * unit-tests/buildbot-syncer-tests.js: Added more test cases and updated existing test cases to test exception messages explicitly since allowing any exception was resulting in some tests passing a result of unrelated parsing error being thrown, etc... (sampleiOSConfig): Updated per the format change. (sampleiOSConfigWithExpansions): Ditto. (smallConfiguration): Ditto. Canonical link: https://commits.webkit.org/189454@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@217378 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-24 18:47:01 +00:00
}).catch((error) => {
New buildbot syncing scripts that supports multiple builders and slaves https://bugs.webkit.org/show_bug.cgi?id=156269 Reviewed by Chris Dumez. Add sync-buildbot.js that supports scheduling A/B testing jobs on multiple builders and slaves. The old python script (sync-with-buildbot.py) could only support a single builder and slave for each platform, test pair. The main logic is implemented in BuildbotTriggerable.syncOnce. Various helper methods are added throughout the codebase and tests have been refactored. BuildbotSyncer has been updated to support multiple platform, test pairs. It's now responsible for syncing everything on each builder (on a buildbot). Added more unit tests for BuildbotSyncer and server tests for BuildbotTriggerable, and refactored test helpers and mocks as needed. * public/v3/models/build-request.js: (BuildRequest.prototype.status): Added. (BuildRequest.prototype.isScheduled): Added. * public/v3/models/metric.js: (Metric.prototype.fullName): Added. * public/v3/models/platform.js: (Platform): Added the map based on platform name. (Platform.findByName): Added. * public/v3/models/test.js: (Test.topLevelTests): (Test.findByPath): Added. Finds a test based on an array of test names; e.g. ['A', 'B'] would find the test whose name is "B" which has a parent test named "A". (Test.prototype.fullName): Added. * server-tests/api-build-requests-tests.js: (addMockData): Moved to resources/mock-data.js. (addAnotherMockTestGroup): Ditto. * server-tests/resources/mock-data.js: Added. (MockData.resetV3Models): Added. (MockData.addMockData): Moved from api-build-requests-tests.js. (MockData.addAnotherMockTestGroup): Ditto. (MockData.mockTestSyncConfigWithSingleBuilder): Added. (MockData.mockTestSyncConfigWithTwoBuilders): Added. (MockData.pendingBuild): Added. (MockData.runningBuild): Added. (MockData.finishedBuild): Added. * server-tests/resources/test-server.js: (TestServer): (TestServer.prototype.remoteAPI): (TestServer.prototype._ensureTestDatabase): Don't fail even if the test database doesn't exit. (TestServer.prototype._startApache): Create a RemoteAPI instance to access the test sever. (TestServer.prototype._waitForPid): Increase the timeout. (TestServer.prototype.inject): Replace global.RemoteAPI during the test and restore it afterwards. * server-tests/tools-buildbot-triggerable-tests.js: Added. Tests BuildbotTriggerable.syncOnce. (MockLogger): Added. (MockLogger.prototype.log): Added. (MockLogger.prototype.error): Added. * tools/detect-changes.js: (parseArgument): Moved to js/parse-arguments.js. * tools/js/buildbot-syncer.js: (BuildbotBuildEntry): (BuildbotBuildEntry.prototype.syncer): Added. (BuildbotBuildEntry.prototype.buildRequestStatusIfUpdateIsNeeded): Added. Returns a new status for a build request (of the matching build request ID) if it needs to be updated in the server. (BuildbotSyncer): This class (BuildbotSyncer.prototype.addTestConfiguration): Added. (BuildbotSyncer.prototype.testConfigurations): Returns the list of test configurations. (BuildbotSyncer.prototype.matchesConfiguration): Returns true iff the request can be scheduled on this builder. (BuildbotSyncer.prototype.scheduleRequest): Added. Schedules a new job on buildbot for a request. (BuildbotSyncer.prototype.scheduleFirstRequestInGroupIfAvailable): Added. Schedules a new job for the specified build request on the first slave that's available. (BuildbotSyncer.prototype.pullBuildbot): Return a list of BuildbotBuildEntry instead of an object. Also store it on an instance variable so that scheduleFirstRequestInGroupIfAvailable could use it. (BuildbotSyncer.prototype._pullRecentBuilds): (BuildbotSyncer.prototype.pathForPendingBuildsJSON): Renamed from urlForPendingBuildsJSON and now only returns the path instead of the full URL since RemoteAPI takes a path, not full URL. (BuildbotSyncer.prototype.pathForBuildJSON): Ditto from pathForBuildJSON. (BuildbotSyncer.prototype.pathForForceBuild): Added. (BuildbotSyncer.prototype.url): Use RemoteAPI's url method instead of manually constructing URL. (BuildbotSyncer.prototype.urlForBuildNumber): Ditto. (BuildbotSyncer.prototype._propertiesForBuildRequest): Now that each syncer can have multiple test configurations associated with it, find the one matching for this request. (BuildbotSyncer._loadConfig): Create a syncer per builder and add all test configurations to it. (BuildbotSyncer._validateAndMergeConfig): Added the support for 'SlaveList', which is a list of slave names present on this builder. * tools/js/buildbot-triggerable.js: Added. (BuildbotTriggerable): Added. (BuildbotTriggerable.prototype.name): Added. (BuildbotTriggerable.prototype.syncOnce): Added. The main logic for the syncing script. It pulls existing build requests from the perf dashboard, pulls buildbot for pending and running/completed builds on each builder (represented by each syncer), schedules build requests on buildbot if there is any builder/slave available, and updates the status of build requests in the database. (BuildbotTriggerable.prototype._validateRequests): Added. (BuildbotTriggerable.prototype._pullBuildbotOnAllSyncers): Added. (BuildbotTriggerable.prototype._scheduleNextRequestInGroupIfSlaveIsAvailable): Added. (BuildbotTriggerable._testGroupMapForBuildRequests): Added. * tools/js/database.js: * tools/js/parse-arguments.js: Added. Extracted out of tools/detect-changes.js. (parseArguments): * tools/js/remote.js: (RemoteAPI): Now optionally takes the server configuration. (RemoteAPI.prototype.url): Added. (RemoteAPI.prototype.getJSON): Removed the code for specifying request content. (RemoteAPI.prototype.getJSONWithStatus): Ditto. (RemoteAPI.prototype.postJSON): Added. (RemoteAPI.prototype.postFormUrlencodedData): Added. (RemoteAPI.prototype.sendHttpRequest): Fixed the code to specify auth. * tools/js/v3-models.js: Don't include RemoteAPI here as they require a configuration for each host. * tools/sync-buildbot.js: Added. (main): Added. Parse the arguments and start the loop. (syncLoop): Added. * unit-tests/buildbot-syncer-tests.js: Added tests for pullBuildbot, scheduleRequest, as well as scheduleFirstRequestInGroupIfAvailable. Refactored helper functions as needed. (sampleiOSConfig): (smallConfiguration): Added. (smallPendingBuild): Added. (smallInProgressBuild): Added. (smallFinishedBuild): Added. (createSampleBuildRequest): Create a unique build request for each platform. (samplePendingBuild): Optionally specify build time and slave name. (sampleInProgressBuild): Optionally specify slave name. (sampleFinishedBuild): Ditto. * unit-tests/resources/mock-remote-api.js: (assert.notReached.assert.notReached): (MockRemoteAPI.url): Added. (MockRemoteAPI.postFormUrlencodedData): Added. (MockRemoteAPI._addRequest): Extracted from getJSONWithStatus. (MockRemoteAPI.waitForRequest): Extracted from inject. For tools-buildbot-triggerable-tests.js, we need to instantiate a RemoteAPI for buildbot without replacing global.RemoteAPI. (MockRemoteAPI.inject): (MockRemoteAPI.reset): Added. Canonical link: https://commits.webkit.org/174395@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@199123 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-04-06 23:19:29 +00:00
console.error(error);
if (typeof(error.stack) == 'string') {
for (let line of error.stack.split('\n'))
console.error(line);
}
Add the support for perf try bots to sync-buildbot.js https://bugs.webkit.org/show_bug.cgi?id=172529 Rubber-stamped by Chris Dumez. Make sync-buildbot.js schedule an A/B testing job with a patch or roots to buildbot. Change the buildbot property format in the syncing script's configuration again to use a dictionary with a single key of "revision", "patch", or "roots" to specify a revision, a patch, or a set of roots, and simplified the structure of the configuration by always having "types" and "builders", and make each entry in "configurations" refer to a list of types, platforms, and builders. Since now there are build requests to build patches and run tests, "configurations" has been renamed to "testConfigurations" and "buildConfigurations" have been added. Each entry in "buildConfigurations" specifies a list of platforms and builders. Similarly in repository group configurations, the buildbot properties for testing is now specified as "testProperties" and ones for building a patch is specified in newly introduced "buildProperties". * public/api/build-requests.php: (update_builds): When a build request to build a patch fails, mark all subsequent requests as failed since there is no way to run tests without a successful build. * public/api/update-triggerable.php: (main): Re-generate manifest.json after updating the triggerable. The lack of this re-generation was the reason we had to manually GET /api/manifest in api-update-triggerable-tests.js. * public/v3/models/build-request.js: (BuildRequest.prototype.hasCompleted): Added. * public/v3/models/manifest.js: (Manifest.reset): Added. Extracted from MockData.resetV3Models in unit-tests/mock-data.js and syncLoop in tools/sync-buildbot.js (Manifest.fetch): Reset V3 models before fetching the manifest. This eliminates the need to manually reset V3 models in syncLoop. * public/v3/models/uploaded-file.js: (UploadedFile.prototype.url): Use RemoteAPI.url to get the full URL instead of just a path. * public/v3/remote.js: (BrowserRemoteAPI.prototype.url): Added. Constructs the full URL. * server-tests/api-update-triggerable-tests.js: (.refetchManifest): Deleted. Now that /api/manifest re-generates manifest.json, we can simply call Manifest.fetch instead. * server-tests/resources/mock-data.js: (MockData.resetV3Models): Calls Manifest.reset(). (MockData.addMockConfiguration): Extracted from addMockData. (MockData.addMockData): Updated per the format change. (MockData.mockTestSyncConfigWithSingleBuilder): Ditto. (MockData.mockTestSyncConfigWithTwoBuilders): Ditto. (MockData.runningBuild): Make buildNumber specifiable. (MockData.finishedBuild): Ditto. * server-tests/tools-buildbot-triggerable-tests.js: Updated configurations per the format change. Now that now acceptsCustomRoots() for "system-and-webkit" must be true since we can't have a repository group that which accepts a patch and not take roots. * server-tests/tools-sync-buildbot-integration-tests.js: Added. (createTriggerable): Added. (createTestGroupWihPatch): Added. (uploadRoot): Added. (.assertAndResolveRequest): Added. (.assertTestBuildHasFailed): Added. * tools/js/buildbot-syncer.js: (BuildbotSyncer): Added. _type as an instance variable to identify whether this buildbot builder is a "builder" which builds a patch, builder, or a "tester" which runs a test. Also renamed _testConfigurations to _configurations. (BuildbotSyncer.prototype.addTestConfiguration): Assert that either the type of this syncer hasn't been set or it's a tester. (BuildbotSyncer.prototype.testConfigurations): Return [] when it's a builder. (BuildbotSyncer.prototype.addBuildConfiguration): Added. Adds a platform to a builder. (BuildbotSyncer.prototype.buildConfigurations): Added. Returns the list of configurations if this syncer is a builder. Otherwise returns []. (BuildbotSyncer.prototype.isTester): Added. (BuildbotSyncer.prototype.matchesConfiguration): (BuildbotSyncer.prototype._propertiesForBuildRequest): Updated to support the new format. (BuildbotSyncer._loadConfig): Ditto. Optionally parse buildConfigurations. (BuildbotSyncer._resolveBuildersWithPlatforms): Added. For each test or build configuration entry, creates the list of configurations per builder and platform. (BuildbotSyncer._parseRepositoryGroup): Added the support for parsing the new format with revision, roots, and patch option types with a lot of validations as we're seeing a bit of combinatorial explosion in the number of things that can go wrong. Also parse buildProperties optionally. (BuildbotSyncer._parseRepositoryGroupPropertyTemplate): Added. A helper function to parse a set of buildbot properties, validates its content, and invokes a callback if it's an dynamically resolved type such as "revision" and "patch". (BuildbotSyncer._validateAndMergeConfig): Updated per the format change. No longer allows "types", "type", "platforms", and "platform" as they're explicity resolved in _resolveBuildersWithPlatforms. * tools/js/buildbot-triggerable.js: (BuildbotTriggerable.prototype.syncOnce): (BuildbotTriggerable.prototype._validateRequests): Handle the case when a build request is not associated with any test. (BuildbotTriggerable.prototype._nextRequestInGroup): Return null when there is a build request to build a patch which has not been completed (pending, scheduled, running, or failed). Since all requests to build a patch has a negative order, those requests should all show up at the beginning. (BuildbotTriggerable.prototype._scheduleRequestIfSlaveIsAvailable): Pick a new buildbot syncer when scheduling the first request to build a patch or the first request to run a test. The first request to run a test will always have order of 0, so it's a sufficient condition to find such a request. On the other hand, the first request to build a patch can have a negative order number so we must explicitly check if it's the first item in the ordered list of requests in the test group. * tools/remote-server-relay.log: Added. * tools/sync-buildbot.js: (syncLoop): Fixed a bug we were not re-fetching the triggerable after updating the triggerable so that Triggerable and related objects we have in the memory may not reflect what we just synced to the perf dashboard. Also, we don't reset V3 models manually any more since Manifest.fetch does that. * unit-tests/buildbot-syncer-tests.js: Added more test cases and updated existing test cases to test exception messages explicitly since allowing any exception was resulting in some tests passing a result of unrelated parsing error being thrown, etc... (sampleiOSConfig): Updated per the format change. (sampleiOSConfigWithExpansions): Ditto. (smallConfiguration): Ditto. Canonical link: https://commits.webkit.org/189454@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@217378 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-24 18:47:01 +00:00
}).then(() => {
New buildbot syncing scripts that supports multiple builders and slaves https://bugs.webkit.org/show_bug.cgi?id=156269 Reviewed by Chris Dumez. Add sync-buildbot.js that supports scheduling A/B testing jobs on multiple builders and slaves. The old python script (sync-with-buildbot.py) could only support a single builder and slave for each platform, test pair. The main logic is implemented in BuildbotTriggerable.syncOnce. Various helper methods are added throughout the codebase and tests have been refactored. BuildbotSyncer has been updated to support multiple platform, test pairs. It's now responsible for syncing everything on each builder (on a buildbot). Added more unit tests for BuildbotSyncer and server tests for BuildbotTriggerable, and refactored test helpers and mocks as needed. * public/v3/models/build-request.js: (BuildRequest.prototype.status): Added. (BuildRequest.prototype.isScheduled): Added. * public/v3/models/metric.js: (Metric.prototype.fullName): Added. * public/v3/models/platform.js: (Platform): Added the map based on platform name. (Platform.findByName): Added. * public/v3/models/test.js: (Test.topLevelTests): (Test.findByPath): Added. Finds a test based on an array of test names; e.g. ['A', 'B'] would find the test whose name is "B" which has a parent test named "A". (Test.prototype.fullName): Added. * server-tests/api-build-requests-tests.js: (addMockData): Moved to resources/mock-data.js. (addAnotherMockTestGroup): Ditto. * server-tests/resources/mock-data.js: Added. (MockData.resetV3Models): Added. (MockData.addMockData): Moved from api-build-requests-tests.js. (MockData.addAnotherMockTestGroup): Ditto. (MockData.mockTestSyncConfigWithSingleBuilder): Added. (MockData.mockTestSyncConfigWithTwoBuilders): Added. (MockData.pendingBuild): Added. (MockData.runningBuild): Added. (MockData.finishedBuild): Added. * server-tests/resources/test-server.js: (TestServer): (TestServer.prototype.remoteAPI): (TestServer.prototype._ensureTestDatabase): Don't fail even if the test database doesn't exit. (TestServer.prototype._startApache): Create a RemoteAPI instance to access the test sever. (TestServer.prototype._waitForPid): Increase the timeout. (TestServer.prototype.inject): Replace global.RemoteAPI during the test and restore it afterwards. * server-tests/tools-buildbot-triggerable-tests.js: Added. Tests BuildbotTriggerable.syncOnce. (MockLogger): Added. (MockLogger.prototype.log): Added. (MockLogger.prototype.error): Added. * tools/detect-changes.js: (parseArgument): Moved to js/parse-arguments.js. * tools/js/buildbot-syncer.js: (BuildbotBuildEntry): (BuildbotBuildEntry.prototype.syncer): Added. (BuildbotBuildEntry.prototype.buildRequestStatusIfUpdateIsNeeded): Added. Returns a new status for a build request (of the matching build request ID) if it needs to be updated in the server. (BuildbotSyncer): This class (BuildbotSyncer.prototype.addTestConfiguration): Added. (BuildbotSyncer.prototype.testConfigurations): Returns the list of test configurations. (BuildbotSyncer.prototype.matchesConfiguration): Returns true iff the request can be scheduled on this builder. (BuildbotSyncer.prototype.scheduleRequest): Added. Schedules a new job on buildbot for a request. (BuildbotSyncer.prototype.scheduleFirstRequestInGroupIfAvailable): Added. Schedules a new job for the specified build request on the first slave that's available. (BuildbotSyncer.prototype.pullBuildbot): Return a list of BuildbotBuildEntry instead of an object. Also store it on an instance variable so that scheduleFirstRequestInGroupIfAvailable could use it. (BuildbotSyncer.prototype._pullRecentBuilds): (BuildbotSyncer.prototype.pathForPendingBuildsJSON): Renamed from urlForPendingBuildsJSON and now only returns the path instead of the full URL since RemoteAPI takes a path, not full URL. (BuildbotSyncer.prototype.pathForBuildJSON): Ditto from pathForBuildJSON. (BuildbotSyncer.prototype.pathForForceBuild): Added. (BuildbotSyncer.prototype.url): Use RemoteAPI's url method instead of manually constructing URL. (BuildbotSyncer.prototype.urlForBuildNumber): Ditto. (BuildbotSyncer.prototype._propertiesForBuildRequest): Now that each syncer can have multiple test configurations associated with it, find the one matching for this request. (BuildbotSyncer._loadConfig): Create a syncer per builder and add all test configurations to it. (BuildbotSyncer._validateAndMergeConfig): Added the support for 'SlaveList', which is a list of slave names present on this builder. * tools/js/buildbot-triggerable.js: Added. (BuildbotTriggerable): Added. (BuildbotTriggerable.prototype.name): Added. (BuildbotTriggerable.prototype.syncOnce): Added. The main logic for the syncing script. It pulls existing build requests from the perf dashboard, pulls buildbot for pending and running/completed builds on each builder (represented by each syncer), schedules build requests on buildbot if there is any builder/slave available, and updates the status of build requests in the database. (BuildbotTriggerable.prototype._validateRequests): Added. (BuildbotTriggerable.prototype._pullBuildbotOnAllSyncers): Added. (BuildbotTriggerable.prototype._scheduleNextRequestInGroupIfSlaveIsAvailable): Added. (BuildbotTriggerable._testGroupMapForBuildRequests): Added. * tools/js/database.js: * tools/js/parse-arguments.js: Added. Extracted out of tools/detect-changes.js. (parseArguments): * tools/js/remote.js: (RemoteAPI): Now optionally takes the server configuration. (RemoteAPI.prototype.url): Added. (RemoteAPI.prototype.getJSON): Removed the code for specifying request content. (RemoteAPI.prototype.getJSONWithStatus): Ditto. (RemoteAPI.prototype.postJSON): Added. (RemoteAPI.prototype.postFormUrlencodedData): Added. (RemoteAPI.prototype.sendHttpRequest): Fixed the code to specify auth. * tools/js/v3-models.js: Don't include RemoteAPI here as they require a configuration for each host. * tools/sync-buildbot.js: Added. (main): Added. Parse the arguments and start the loop. (syncLoop): Added. * unit-tests/buildbot-syncer-tests.js: Added tests for pullBuildbot, scheduleRequest, as well as scheduleFirstRequestInGroupIfAvailable. Refactored helper functions as needed. (sampleiOSConfig): (smallConfiguration): Added. (smallPendingBuild): Added. (smallInProgressBuild): Added. (smallFinishedBuild): Added. (createSampleBuildRequest): Create a unique build request for each platform. (samplePendingBuild): Optionally specify build time and slave name. (sampleInProgressBuild): Optionally specify slave name. (sampleFinishedBuild): Ditto. * unit-tests/resources/mock-remote-api.js: (assert.notReached.assert.notReached): (MockRemoteAPI.url): Added. (MockRemoteAPI.postFormUrlencodedData): Added. (MockRemoteAPI._addRequest): Extracted from getJSONWithStatus. (MockRemoteAPI.waitForRequest): Extracted from inject. For tools-buildbot-triggerable-tests.js, we need to instantiate a RemoteAPI for buildbot without replacing global.RemoteAPI. (MockRemoteAPI.inject): (MockRemoteAPI.reset): Added. Canonical link: https://commits.webkit.org/174395@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@199123 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-04-06 23:19:29 +00:00
setTimeout(syncLoop.bind(global, options), options['--seconds-to-sleep'] * 1000);
});
}
main(process.argv);