haikuwebkit/Websites/perf.webkit.org/config.json.sample

46 lines
1.5 KiB
Plaintext
Raw Permalink Normal View History

{
"siteTitle": "WebKit Performance Dashboard",
"debug": true,
"jsonCacheMaxAge": 600,
"dataDirectory": "public/data/",
Add the file uploading capability to the perf dashboard. https://bugs.webkit.org/show_bug.cgi?id=169737 Reviewed by Chris Dumez. Added /privileged-api/upload-file to upload a file, and /api/uploaded-file/ to download the file and retrieve its meta data based on its SHA256. We treat two files with the identical SHA256 as identical since anyone who can upload a file using this mechanism can execute arbitrary code in our bots anyway. This is important for avoiding uploading a large darwinup roots multiple times to the server, saving both user's time/bandwidth and server's disk space. * config.json: Added uploadDirectory, uploadFileLimitInMB, and uploadUserQuotaInMB as options. * init-database.sql: Added uploaded_files table. * public/api/uploaded-file.php: Added. (main): /api/uploaded-file/N would download uploaded_file with id=N. /api/uploaded-file/?sha256=X would return the meta data for uploaded_file with sha256=X. (stream_file_content): Streams the file content in 64KB chunks. We support Range & If-Range HTTP request headers so that browsers can pause and resume downloading of a large root file. (parse_range_header): Parses Range HTTP request header. * public/include/json-header.php: (remote_user_name): Use the default argument of NULL. * public/include/manifest-generator.php: (ManifestGenerator::generate): Include the maximum upload size in the manifest file to let the frontend code preemptively check the file size before attempting to submit a file. * public/include/uploaded-file-helpers.php: Added. (format_uploaded_file): (uploaded_file_path_for_row): * public/privileged-api/upload-file-form.html: Added. For debugging purposes. (fetchCSRFfToken): (upload): * public/privileged-api/upload-file.php: Added. (main): (query_total_file_size): (create_uploaded_file_from_form_data): * public/shared/common-remote.js: (CommonRemoteAPI.prototype.postFormData): Added. (CommonRemoteAPI.prototype.postFormDataWithStatus): Added. (CommonRemoteAPI.prototype.sendHttpRequestWithFormData): Added. (CommonRemoteAPI.prototype._asJSON): Throw an exception instead of calling a non-existent reject. * public/v3/models/uploaded-file.js: Added. (UploadedFile): Added. (UploadedFile.uploadFile): Added. (UploadedFile.fetchUnloadedFileWithIdenticalHash): Added. Finds the file with the same SHA256 in the server to avoid uploading a large custom root multiple times. (UploadedFile._computeSHA256Hash): Added. * public/v3/privileged-api.js: (PrivilegedAPI.prototype.sendRequest): Added the options dictionary as a third argument. For now, only support useFormData boolean. * public/v3/remote.js: (BrowserRemoteAPI.prototype.sendHttpRequestWithFormData): Added. * server-tests/api-manifest.js: Updated per the inclusion of fileUploadSizeLimit in the manifest. * server-tests/api-uploaded-file.js: Added. * server-tests/privileged-api-upload-file-tests.js: Added. * server-tests/resources/temporary-file.js: Added. (TemporaryFile): Added. A helper class for creating a temporary file to upload. (TemporaryFile.makeTemporaryFileOfSizeInMB): (TemporaryFile.makeTemporaryFile): (TemporaryFile.inject): * server-tests/resources/test-server.conf: Set upload_max_filesize and post_max_size for testing. * server-tests/resources/test-server.js: (TestServer.prototype.testConfig): Use uploadFileLimitInMB and uploadUserQuotaInMB of 2MB and 5MB. (TestServer.prototype._ensureDataDirectory): Create a directory to store uploaded files inside the data directory. In a production server, we can place it outside ServerRoot / DocumentRoot. (TestServer.prototype.cleanDataDirectory): Delete the aforementioned directory as needed. * tools/js/database.js: (tableToPrefixMap): Added uploaded_files. * tools/js/remote.js: (NodeRemoteAPI.prototype.sendHttpRequest): Added a dictionary to specify request headers and a callback to process the response as arguments. Fixed the bug that any 2xx code other than 200 was resulting in a rejected promise. Also include the response headers in the result for tests. Finally, when content is a function, call that instead of writing the content since FormData requires a custom logic. (NodeRemoteAPI.prototype.sendHttpRequestWithFormData): Added. * tools/js/v3-models.js: Include uploaded-file.js. * tools/run-tests.py: (main): Add form-data as a new dependency. Canonical link: https://commits.webkit.org/186730@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@214065 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-03-16 20:53:35 +00:00
"uploadDirectory": "uploaded",
"uploadFileLimitInMB": 800,
"uploadUserQuotaInMB": 8192,
"uploadTotalQuotaInMB": 102400,
"database": {
"host": "localhost",
"port": "5432",
"username": "webkit-perf-db-user",
"password": "password",
"name": "webkit-perf-db"
},
"universalSlavePassword": null,
"maintenanceMode": false,
"maintenanceDirectory": "reported/",
"testServer": {
Add mocha server tests for /api/build-requests https://bugs.webkit.org/show_bug.cgi?id=155831 Reviewed by Chris Dumez. Added the new mocha.js based server-tests for /api/build-requests. The new harness automatically: - starts a new Apache instance - switches the database during testing via setting an environmental variable - backups and restores public/data directory during testing As a result, developer no longer has to manually setup Apache, edit config.json manually to use a testing database, or run /api/manifest.php to re-generate the manifest file after testing. This patch also makes ID resolution optional on /api/build-requests so that v3 model based syncing scripts can re-use the same code as the v3 UI to process the JSON. tools/sync-with-buildbot.py has been modified to use this option (useLegacyIdResolution). * config.json: Added configurations for the test httpd server. * init-database.sql: Don't error when tables and types don't exist (when database is empty). * public/api/build-requests.php: (main): Made the ID resolution optional with useLegacyIdResolution. Also removed "updates" from the results JSON since it's never used. * public/include/build-requests-fetcher.php: (BuildRequestsFetcher::__construct): (BuildRequestsFetcher::fetch_roots_for_set_if_needed): Fixed the bug that we would include the same commit multiple times for each root set. * public/include/db.php: (config): If present, use ORG_WEBKIT_PERF_CONFIG_PATH instead of Websites/perf.webkit.org/config.json. * server-tests: Added. * server-tests/api-build-requests-tests.js: Added. Tests for /api/build-requests. (.addMockData): * server-tests/resources: Added. * server-tests/resources/test-server.conf: Added. Apache configuration file for testing. * server-tests/resources/test-server.js: Added. (TestSever): Added. (TestSever.prototype.start): Added. (TestSever.prototype.stop): Added. (TestSever.prototype.remoteAPI): Added. Configures RemoteAPI to be used with the test sever. (TestSever.prototype.database): Added. Returns Database configured to use the test database. (TestSever.prototype._constructTestConfig): Creates config.json for testing. The file is generated by _start and db.php's config() reads it from the environmental variable: ORG_WEBKIT_PERF_CONFIG_PATH. (TestSever.prototype._ensureDataDirectory): Renames public/data to public/original-data if exists, and creates a new empty public/data. (TestSever.prototype._restoreDataDirectory): Deletes public/data and renames public/original-data back to public/data. (TestSever.prototype._ensureTestDatabase): Drops the test database if exists and creates a new one. (TestSever.prototype.initDatabase): Run init-database.sql to start each test with a consistent state. (TestSever.prototype._executePgsqlCommand): Executes a postgres command line tool such as psql. (TestSever.prototype._determinePgsqlDirectory): Finds the directory that contains psql. (TestSever.prototype._startApache): Starts an Apache instance for testing. (TestSever.prototype._stopApache): Stops the Apache instance for testing. (TestSever.prototype._waitForPid): Waits for the Apache pid file to appear or disappear. (before): Start the test server at the beginning. (beforeEach): Re-initialize all tables before each test. (after): Stop the test server at the end. * tools/js/config.js: (Config.prototype.path): (Config.prototype.serverRoot): Added. The path to Websites/perf.webkit.org/public/. (Config.prototype.pathFromRoot): Added. Resolves a path from Websites/perf.webkit.org. * tools/js/database.js: (Database): Now optionally takes the database name to use a different database during testing. (Database.prototype.connect): (Database.prototype.query): Added. (Database.prototype.insert): Added. (tableToPrefixMap): Maps table name to its prefix. Used by Database.insert. * tools/js/remote.js: Added. (RemoteAPI): Added. This is node.js equivalent of RemoteAPI in public/v3/remote.js. (RemoteAPI.prototype.configure): Added. (RemoteAPI.prototype.fetchJSON): Added. (RemoteAPI.prototype.fetchJSONWithStatus): Added. (RemoteAPI.prototype.sendHttpRequest): Added. * tools/sync-with-buildbot.py: (main): Use useLegacyIdResolution as this script relies on the legacy behavior. * unit-tests/checkconfig.js: pg was never directly used in this test. Canonical link: https://commits.webkit.org/173950@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@198642 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-03-24 20:17:01 +00:00
"port": 8180,
"config": "/tmp/org.webkit.perf.test-config.json",
"httpdConfig": "server-tests/resources/test-server.conf",
"httpdPID": "/tmp/org.webkit.perf.test-httpd.pid",
"httpdErrorLog": "server-tests/resources/test-server.log",
"httpdMutexDir": "/tmp/org.webkit.perf.tests/"
The dashboard on new perf monitor should be configurable https://bugs.webkit.org/show_bug.cgi?id=138994 Reviewed by Benjamin Poulain. For now, make it configurable via config.json. We should eventually make it configurable via an administrative page but this will do for now. * config.json: Added the empty dashboard configuration. * public/include/manifest.php: Include the dashboard configuration in the manifest file. * public/v2/app.js: (App.IndexController): Removed defaultTable since this is now dynamically obtained via App.Manifest. (App.IndexController.gridChanged): Use App.Dashboard to parse the dashboard configuration. Also obtain the default configuration from App.Manifest. (App.IndexController._normalizeTable): Moved to App.Dashboard. * public/v2/manifest.js: (App.Repository.urlForRevision): Fixed the position of the open curly bracket. (App.Repository.urlForRevisionRange): Ditto. (App.Dashboard): Added. (App.Dashboard.table): Extracted from App.IndexController.gridChanged. (App.Dashboard.rows): Ditto. (App.Dashboard.headerColumns): Ditto. (App.Dashboard._normalizeTable): Moved from App.IndexController._normalizeTable. (App.MetricSerializer.normalizePayload): Synthesize a dashboard record from the configuration. Since there is exactly one dashboard object per app, it's okay to hard code an id here. (App.Manifest._fetchedManifest): Set defaultDashboard to the one and only one dashboard we have. Canonical link: https://commits.webkit.org/156877@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@176497 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2014-11-22 03:42:26 +00:00
},
Add mocha server tests for /api/build-requests https://bugs.webkit.org/show_bug.cgi?id=155831 Reviewed by Chris Dumez. Added the new mocha.js based server-tests for /api/build-requests. The new harness automatically: - starts a new Apache instance - switches the database during testing via setting an environmental variable - backups and restores public/data directory during testing As a result, developer no longer has to manually setup Apache, edit config.json manually to use a testing database, or run /api/manifest.php to re-generate the manifest file after testing. This patch also makes ID resolution optional on /api/build-requests so that v3 model based syncing scripts can re-use the same code as the v3 UI to process the JSON. tools/sync-with-buildbot.py has been modified to use this option (useLegacyIdResolution). * config.json: Added configurations for the test httpd server. * init-database.sql: Don't error when tables and types don't exist (when database is empty). * public/api/build-requests.php: (main): Made the ID resolution optional with useLegacyIdResolution. Also removed "updates" from the results JSON since it's never used. * public/include/build-requests-fetcher.php: (BuildRequestsFetcher::__construct): (BuildRequestsFetcher::fetch_roots_for_set_if_needed): Fixed the bug that we would include the same commit multiple times for each root set. * public/include/db.php: (config): If present, use ORG_WEBKIT_PERF_CONFIG_PATH instead of Websites/perf.webkit.org/config.json. * server-tests: Added. * server-tests/api-build-requests-tests.js: Added. Tests for /api/build-requests. (.addMockData): * server-tests/resources: Added. * server-tests/resources/test-server.conf: Added. Apache configuration file for testing. * server-tests/resources/test-server.js: Added. (TestSever): Added. (TestSever.prototype.start): Added. (TestSever.prototype.stop): Added. (TestSever.prototype.remoteAPI): Added. Configures RemoteAPI to be used with the test sever. (TestSever.prototype.database): Added. Returns Database configured to use the test database. (TestSever.prototype._constructTestConfig): Creates config.json for testing. The file is generated by _start and db.php's config() reads it from the environmental variable: ORG_WEBKIT_PERF_CONFIG_PATH. (TestSever.prototype._ensureDataDirectory): Renames public/data to public/original-data if exists, and creates a new empty public/data. (TestSever.prototype._restoreDataDirectory): Deletes public/data and renames public/original-data back to public/data. (TestSever.prototype._ensureTestDatabase): Drops the test database if exists and creates a new one. (TestSever.prototype.initDatabase): Run init-database.sql to start each test with a consistent state. (TestSever.prototype._executePgsqlCommand): Executes a postgres command line tool such as psql. (TestSever.prototype._determinePgsqlDirectory): Finds the directory that contains psql. (TestSever.prototype._startApache): Starts an Apache instance for testing. (TestSever.prototype._stopApache): Stops the Apache instance for testing. (TestSever.prototype._waitForPid): Waits for the Apache pid file to appear or disappear. (before): Start the test server at the beginning. (beforeEach): Re-initialize all tables before each test. (after): Stop the test server at the end. * tools/js/config.js: (Config.prototype.path): (Config.prototype.serverRoot): Added. The path to Websites/perf.webkit.org/public/. (Config.prototype.pathFromRoot): Added. Resolves a path from Websites/perf.webkit.org. * tools/js/database.js: (Database): Now optionally takes the database name to use a different database during testing. (Database.prototype.connect): (Database.prototype.query): Added. (Database.prototype.insert): Added. (tableToPrefixMap): Maps table name to its prefix. Used by Database.insert. * tools/js/remote.js: Added. (RemoteAPI): Added. This is node.js equivalent of RemoteAPI in public/v3/remote.js. (RemoteAPI.prototype.configure): Added. (RemoteAPI.prototype.fetchJSON): Added. (RemoteAPI.prototype.fetchJSONWithStatus): Added. (RemoteAPI.prototype.sendHttpRequest): Added. * tools/sync-with-buildbot.py: (main): Use useLegacyIdResolution as this script relies on the legacy behavior. * unit-tests/checkconfig.js: pg was never directly used in this test. Canonical link: https://commits.webkit.org/173950@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@198642 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-03-24 20:17:01 +00:00
"testDatabaseName": "test-db",
Add /api/measurement-set for v3 UI https://bugs.webkit.org/show_bug.cgi?id=152312 Rubber-stamped by Chris Dumez. The new API JSON allows the front end to fetch measured data in chunks called a "cluster" as specified in config.json for each measurement set specified by the pair of a platform and a metric. When the front end needs measured data in a given time range (t_0, t_1) for a measurement set, it first fetches the primary cluster by /api/measurement-set/?platform=<platform-id>&metric=<metric-id>. The primary cluster is the last cluster in the set (returning the first cluster here is not useful since we don't typically show very old data), and provides the information needed to fetch other clusters. Fetching the primary cluster also creates JSON files at: /data/measurement-set-<platform-id>-<metric-id>-<cluster-end-time>.json to allow latency free access for secondary clusters. The front end code can also fetch the cache of the primary cluster at: /data/measurement-set-<platform-id>-<metric-id>.json. Because the front end code has to behave as if all data is fetched, each cluster contains one data point immediately before the first data point and one immediately after the last data point. This avoids having to fetch multiple empty clusters for manually specified baseline data. To support this behavior, we generate all clusters for a given measurement set at once when the primary cluster is requested. Furthermore, all measurement sets are divided at the same time into clusters so that the boundary of clusters won't shift as more data are reported to the server. * config.json: Added clusterStart and clusterSize as options. * public/api/measurement-set.php: Added. (main): (MeasurementSetFetcher::__construct): (MeasurementSetFetcher::fetch_config_list): Finds configurations that belongs to this (platform, metric) pair. (MeasurementSetFetcher::at_end): Returns true if we've reached the end of all clusters for this set. (MeasurementSetFetcher::fetch_next_cluster): Generates the JSON data for the next cluster. We generate clusters in increasing chronological order (the oldest first and the newest last). (MeasurementSetFetcher::execute_query): Executes the main query. (MeasurementSetFetcher::format_map): Returns the mapping of a measurement field to an array index. This removes the need to have key names for each measurement and reduces the JSON size by ~10%. (MeasurementSetFetcher::format_run): Creates an array that contains data for a single measurement. The order matches that of keys in format_map. (MeasurementSetFetcher::parse_revisions_array): Added. Copied from runs.php. * tests/api-measurement-set.js: Added. Added tests for /api/measurement-set. Canonical link: https://commits.webkit.org/170424@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@194120 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-12-15 23:57:25 +00:00
"clusterStart": [2000, 1, 1, 0, 0],
"clusterSize": [0, 2, 0],
"cacheDirectory": "public/data/remote-cache/",
"remoteServer": {
"httpdConfig": "tools/remote-server-relay.conf",
"httpdPID": "tools/remote-server-relay.pid",
"httpdErrorLog": "tools/remote-server-relay.log",
"httpdMutexDir": "/tmp/org.webkit.perf.remote/",
"url": "https://perf.webkit.org",
"basicAuth": {
"username": "username",
"password": "password"
}
},
"defaultDashboard": [[]],
"dashboards": {}
}