2015-08-11 19:51:35 +00:00
|
|
|
/*
|
2016-04-20 04:25:02 +00:00
|
|
|
* Copyright (C) 2015-2016 Apple Inc. All rights reserved.
|
2015-08-11 19:51:35 +00:00
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
|
|
|
|
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
|
|
|
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
|
|
|
|
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
|
|
|
|
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
|
|
|
|
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
|
|
|
|
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
|
|
|
|
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
|
|
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
|
|
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
*/
|
|
|
|
|
2018-10-15 14:24:49 +00:00
|
|
|
#pragma once
|
2015-08-11 19:51:35 +00:00
|
|
|
|
|
|
|
#include <wtf/Atomics.h>
|
2016-05-26 21:58:42 +00:00
|
|
|
#include <wtf/ScopedLambda.h>
|
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows
https://bugs.webkit.org/show_bug.cgi?id=152045
Reviewed by Andy Estes.
Source/JavaScriptCore:
Probably the nicest example of why this patch is a good idea is the change in
AtomicsObject.cpp.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
* runtime/AtomicsObject.cpp:
(JSC::atomicsFuncWait):
Source/WebCore:
No new layout tests because no new behavior. The new WTF time classes have some unit tests
in TestWebKitAPI.
* fileapi/ThreadableBlobRegistry.cpp:
(WebCore::ThreadableBlobRegistry::blobSize):
* platform/MainThreadSharedTimer.h:
* platform/SharedTimer.h:
* platform/ThreadTimers.cpp:
(WebCore::ThreadTimers::updateSharedTimer):
* platform/cf/MainThreadSharedTimerCF.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/efl/MainThreadSharedTimerEfl.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/glib/MainThreadSharedTimerGLib.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/win/MainThreadSharedTimerWin.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* workers/WorkerRunLoop.cpp:
(WebCore::WorkerRunLoop::runInMode):
Source/WebKit2:
* Platform/IPC/Connection.cpp:
(IPC::Connection::SyncMessageState::wait):
(IPC::Connection::sendMessage):
(IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting):
(IPC::Connection::waitForMessage):
(IPC::Connection::sendSyncMessage):
(IPC::Connection::waitForSyncReply):
* Platform/IPC/Connection.h:
(IPC::Connection::sendSync):
(IPC::Connection::waitForAndDispatchImmediately):
* Platform/IPC/MessageSender.h:
(IPC::MessageSender::sendSync):
* UIProcess/ChildProcessProxy.h:
(WebKit::ChildProcessProxy::sendSync):
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/Storage/StorageManager.cpp:
(WebKit::StorageManager::applicationWillTerminate):
* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/WebResourceLoadStatisticsStore.cpp:
(WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate):
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h:
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm:
(-[WKOneShotDisplayLinkHandler displayLinkFired:]):
(WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree):
(WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay):
(WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm:
(WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/WKImmediateActionController.mm:
(-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]):
* UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::stringSelectionForPasteboard):
(WebKit::WebPageProxy::dataSelectionForPasteboard):
(WebKit::WebPageProxy::readSelectionFromPasteboard):
(WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent):
(WebKit::WebPageProxy::acceptsFirstMouse):
* WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::runBeforeUnloadConfirmPanel):
(WebKit::WebChromeClient::runJavaScriptAlert):
(WebKit::WebChromeClient::runJavaScriptConfirm):
(WebKit::WebChromeClient::runJavaScriptPrompt):
(WebKit::WebChromeClient::print):
(WebKit::WebChromeClient::exceededDatabaseQuota):
(WebKit::WebChromeClient::reachedApplicationCacheOriginQuota):
* WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp:
(WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse):
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::postSynchronousMessageForTesting):
Source/WTF:
We used to use 'double' for all time measurements. Sometimes it was milliseconds,
sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time
since some epoch. When we spoke of time since epoch, we either used a monotonic clock or
a wall clock. The type - always 'double' - never told us what kind of time we had, even
though there were roughly six of them (sec interval, ms interval, sec since epoch on wall,
ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic).
At some point, we thought that it would be a good idea to replace these doubles with
std::chrono. But since replacing some things with std::chrono, we found it to be terribly
inconvenient:
- Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say
std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally
duration_cast would not even be a thing.
- No overflow protection. std::chrono uses integers by default and using anything else is
clumsy. But the integer math is done without regard for the rough edges of integer math,
so any cast between std::chrono types risks overflow. Any comparison risks overflow
because it may do conversions silently. We have even found bugs where some C++
implementations had more overflows than others, which ends up being a special kind of
hell. In many cases, the overflow also has nasal demons.
It's an error to represent time using integers. It would have been excusable back when
floating point math was not guaranteed to be supported on all platforms, but that would
have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit
for floating point:
- Floating point preserves precision under multiplication in all but extreme cases, so
using floating point for time means that unit conversions are almost completely
lossless. This means that we don't have to think very hard about what units to use. In
this patch, we use seconds almost everywhere. We only convert at boundaries, like an API
boundary that wants something other than seconds.
- Floating point makes it easy to reason about infinity, which is something that time code
wants to do a lot. Example: when would you like to timeout? Infinity please! This is the
most elegant way of having an API support both a timeout variant and a no-timeout
variant.
- Floating point does well-understood things when math goes wrong, and these things are
pretty well optimized to match what a mathematician would do when computing with real
numbers represented using scientific notation with a finite number of significant
digits. This means that time math under floating point looks like normal math. On the
other hand, std::chrono time math looks like garbage because you have to always check
for multiple possible UB corners whenever you touch large integers. Integers that
represent time are very likely to be large and you don't have to do much to overflow
them. At this time, based on the number of bugs we have already seen due to chrono
overflows, I am not certain that we even understand what are all of the corner cases
that we should even check for.
This patch introduces a new set of timekeeping classes that are all based on double, and
all internally use seconds. These classes support algebraic typing. The classes are:
- Seconds: this is for measuring a duration.
- WallTime: time since epoch according to a wall clock (aka real time clock).
- MonotonicTime: time since epoch according to a monotonic clock.
- ClockType: enum that says either Wall or Monotonic.
- TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a
wall time or a monotonic time.
All of these classes behave like C++ values and are cheap to copy around since they are
very nearly POD. This supports comprehensive conversions between the various time types.
Most of this is by way of algebra. Here are just some of the rules we recognize:
WallTime = WallTime + Seconds
Seconds = WallTime - WallTime
MonotonicTime = MonotonicTime + Seconds
etc...
We support negative, infinite, and NaN times because math.
We support conversions between MonotonicTime and WallTime, like:
WallTime wt = mt.approximateWallTime()
This is called this "approximate" because the only way to do it is to get the current time
on both clocks and convert relative to that.
Many of our APIs would be happy using whatever notion of time the user wanted to use. For
those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You
can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This
means that if you use a WallTime with Condition::waitUntil, then Condition's internal
logic for when it should wake up makes its decision based on the current WallTime - but if
you use MonotonicTime then waitUntil will make its decision based on current
MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono
did not have the concept of a dynamic clock type.
This patch does not include conversions between std::chrono and these new time classes,
because past experience shows that we're quite bad at getting conversions between
std::chrono and anything else right. Also, I didn't need such conversion code because this
patch only converts code that transitively touches ParkingLot and Condition. It was easy
to get all of that code onto the new time classes.
* WTF.xcodeproj/project.pbxproj:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/ClockType.cpp: Added.
(WTF::printInternal):
* wtf/ClockType.h: Added.
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::waitFor):
(WTF::ConditionBase::wait):
(WTF::ConditionBase::waitUntilWallClockSeconds): Deleted.
(WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted.
(WTF::ConditionBase::waitForSeconds): Deleted.
(WTF::ConditionBase::waitForSecondsImpl): Deleted.
(WTF::ConditionBase::waitForImpl): Deleted.
(WTF::ConditionBase::absoluteFromRelative): Deleted.
* wtf/CrossThreadQueue.h:
(WTF::CrossThreadQueue<DataType>::waitForMessage):
* wtf/CurrentTime.cpp:
(WTF::sleep):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime): Deleted.
* wtf/MonotonicTime.cpp: Added.
(WTF::MonotonicTime::now):
(WTF::MonotonicTime::approximateWallTime):
(WTF::MonotonicTime::dump):
(WTF::MonotonicTime::sleep):
* wtf/MonotonicTime.h: Added.
(WTF::MonotonicTime::MonotonicTime):
(WTF::MonotonicTime::fromRawDouble):
(WTF::MonotonicTime::infinity):
(WTF::MonotonicTime::secondsSinceEpoch):
(WTF::MonotonicTime::approximateMonotonicTime):
(WTF::MonotonicTime::operator bool):
(WTF::MonotonicTime::operator+):
(WTF::MonotonicTime::operator-):
(WTF::MonotonicTime::operator+=):
(WTF::MonotonicTime::operator-=):
(WTF::MonotonicTime::operator==):
(WTF::MonotonicTime::operator!=):
(WTF::MonotonicTime::operator<):
(WTF::MonotonicTime::operator>):
(WTF::MonotonicTime::operator<=):
(WTF::MonotonicTime::operator>=):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkCount):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
* wtf/Seconds.cpp: Added.
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::dump):
(WTF::Seconds::sleep):
* wtf/Seconds.h: Added.
(WTF::Seconds::Seconds):
(WTF::Seconds::value):
(WTF::Seconds::seconds):
(WTF::Seconds::milliseconds):
(WTF::Seconds::microseconds):
(WTF::Seconds::nanoseconds):
(WTF::Seconds::fromMilliseconds):
(WTF::Seconds::fromMicroseconds):
(WTF::Seconds::fromNanoseconds):
(WTF::Seconds::infinity):
(WTF::Seconds::operator bool):
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::operator*):
(WTF::Seconds::operator/):
(WTF::Seconds::operator+=):
(WTF::Seconds::operator-=):
(WTF::Seconds::operator*=):
(WTF::Seconds::operator/=):
(WTF::Seconds::operator==):
(WTF::Seconds::operator!=):
(WTF::Seconds::operator<):
(WTF::Seconds::operator>):
(WTF::Seconds::operator<=):
(WTF::Seconds::operator>=):
* wtf/TimeWithDynamicClockType.cpp: Added.
(WTF::TimeWithDynamicClockType::now):
(WTF::TimeWithDynamicClockType::nowWithSameClock):
(WTF::TimeWithDynamicClockType::wallTime):
(WTF::TimeWithDynamicClockType::monotonicTime):
(WTF::TimeWithDynamicClockType::approximateWallTime):
(WTF::TimeWithDynamicClockType::approximateMonotonicTime):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator<):
(WTF::TimeWithDynamicClockType::operator>):
(WTF::TimeWithDynamicClockType::operator<=):
(WTF::TimeWithDynamicClockType::operator>=):
(WTF::TimeWithDynamicClockType::dump):
(WTF::TimeWithDynamicClockType::sleep):
* wtf/TimeWithDynamicClockType.h: Added.
(WTF::TimeWithDynamicClockType::TimeWithDynamicClockType):
(WTF::TimeWithDynamicClockType::fromRawDouble):
(WTF::TimeWithDynamicClockType::secondsSinceEpoch):
(WTF::TimeWithDynamicClockType::clockType):
(WTF::TimeWithDynamicClockType::withSameClockAndRawDouble):
(WTF::TimeWithDynamicClockType::operator bool):
(WTF::TimeWithDynamicClockType::operator+):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator+=):
(WTF::TimeWithDynamicClockType::operator-=):
(WTF::TimeWithDynamicClockType::operator==):
(WTF::TimeWithDynamicClockType::operator!=):
* wtf/WallTime.cpp: Added.
(WTF::WallTime::now):
(WTF::WallTime::approximateMonotonicTime):
(WTF::WallTime::dump):
(WTF::WallTime::sleep):
* wtf/WallTime.h: Added.
(WTF::WallTime::WallTime):
(WTF::WallTime::fromRawDouble):
(WTF::WallTime::infinity):
(WTF::WallTime::secondsSinceEpoch):
(WTF::WallTime::approximateWallTime):
(WTF::WallTime::operator bool):
(WTF::WallTime::operator+):
(WTF::WallTime::operator-):
(WTF::WallTime::operator+=):
(WTF::WallTime::operator-=):
(WTF::WallTime::operator==):
(WTF::WallTime::operator!=):
(WTF::WallTime::operator<):
(WTF::WallTime::operator>):
(WTF::WallTime::operator<=):
(WTF::WallTime::operator>=):
* wtf/threads/BinarySemaphore.cpp:
(WTF::BinarySemaphore::wait):
* wtf/threads/BinarySemaphore.h:
Tools:
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/Condition.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp:
(TestWebKitAPI::ToUpperConverter::stopProducing):
(TestWebKitAPI::ToUpperConverter::stopConsuming):
* TestWebKitAPI/Tests/WTF/Time.cpp: Added.
(WTF::operator<<):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/182152@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
|
|
|
#include <wtf/TimeWithDynamicClockType.h>
|
2015-08-11 19:51:35 +00:00
|
|
|
|
|
|
|
namespace WTF {
|
|
|
|
|
[WTF] Implement WTF::ThreadGroup
https://bugs.webkit.org/show_bug.cgi?id=174081
Reviewed by Mark Lam.
Source/JavaScriptCore:
Large part of MachineThreads are now removed and replaced with WTF::ThreadGroup.
And SamplingProfiler and others interact with WTF::Thread directly.
* API/tests/ExecutionTimeLimitTest.cpp:
* heap/MachineStackMarker.cpp:
(JSC::MachineThreads::MachineThreads):
(JSC::captureStack):
(JSC::MachineThreads::tryCopyOtherThreadStack):
(JSC::MachineThreads::tryCopyOtherThreadStacks):
(JSC::MachineThreads::gatherConservativeRoots):
(JSC::ActiveMachineThreadsManager::Locker::Locker): Deleted.
(JSC::ActiveMachineThreadsManager::add): Deleted.
(JSC::ActiveMachineThreadsManager::remove): Deleted.
(JSC::ActiveMachineThreadsManager::contains): Deleted.
(JSC::ActiveMachineThreadsManager::ActiveMachineThreadsManager): Deleted.
(JSC::activeMachineThreadsManager): Deleted.
(JSC::MachineThreads::~MachineThreads): Deleted.
(JSC::MachineThreads::addCurrentThread): Deleted.
(): Deleted.
(JSC::MachineThreads::removeThread): Deleted.
(JSC::MachineThreads::removeThreadIfFound): Deleted.
(JSC::MachineThreads::MachineThread::MachineThread): Deleted.
(JSC::MachineThreads::MachineThread::getRegisters): Deleted.
(JSC::MachineThreads::MachineThread::Registers::stackPointer): Deleted.
(JSC::MachineThreads::MachineThread::Registers::framePointer): Deleted.
(JSC::MachineThreads::MachineThread::Registers::instructionPointer): Deleted.
(JSC::MachineThreads::MachineThread::Registers::llintPC): Deleted.
(JSC::MachineThreads::MachineThread::captureStack): Deleted.
* heap/MachineStackMarker.h:
(JSC::MachineThreads::addCurrentThread):
(JSC::MachineThreads::getLock):
(JSC::MachineThreads::threads):
(JSC::MachineThreads::MachineThread::suspend): Deleted.
(JSC::MachineThreads::MachineThread::resume): Deleted.
(JSC::MachineThreads::MachineThread::threadID): Deleted.
(JSC::MachineThreads::MachineThread::stackBase): Deleted.
(JSC::MachineThreads::MachineThread::stackEnd): Deleted.
(JSC::MachineThreads::threadsListHead): Deleted.
* runtime/SamplingProfiler.cpp:
(JSC::FrameWalker::isValidFramePointer):
(JSC::SamplingProfiler::SamplingProfiler):
(JSC::SamplingProfiler::takeSample):
(JSC::SamplingProfiler::noticeCurrentThreadAsJSCExecutionThread):
* runtime/SamplingProfiler.h:
* wasm/WasmMachineThreads.cpp:
(JSC::Wasm::resetInstructionCacheOnAllThreads):
Source/WebCore:
* page/ResourceUsageThread.h:
Source/WebKit:
* Shared/AsyncRequest.h:
Source/WTF:
This patch implements WTF::ThreadGroup. It implements core of JSC::MachineThreads with more reliable way.
JSC::MachineThreads was complicated because of managing dead threads. Each JSC::MachineThreads has its
own TLS with a registered destructor. And everytime a thread dies, the registered TLS destructor is called.
And this destructor will remove the current dying thread from JSC::MachineThreads.
However the above implementation is tricky. And each JSC::MachineThreads requires own TLS space, which is
not considered in WTF's Windows ThreadSpecific implementation. Current design works well since we only
have small number of MachineThreads right now.
Instead, we use more reliable way. After introducing WTF::Thread, WTF::Thread has WTF::Thread::didExit,
which is called when associated TLS (with WTF::Thread) is destroyed. We leverage this mechanism to remove
WTF::Thread from MachineThreads.
This patch introduces WTF::ThreadGroup. It is tightly integrated with WTF::Thread: WTF::Thread knows
ThreadGroups which includes this thread. And WTF::ThreadGroup of course knows WTF::Threads added to it.
WTF::Thread::didExit carefully remove itself from WTF::ThreadGroups.
The most important part of this patch is locking. WTF::Thread can die. And WTF::ThreadGroup can die.
If we take a design using two fine grain locks in WTF::Thread and WTF::ThreadGroup, we easily encounter
dead lock. Consider the following case.
1. When adding WTF::Thread (TH) to WTF::ThreadGroup (THG), we first hold a lock of THG, and hold a lock of TH (locking order is THG -> TH).
2. When TH dies, TH need to hold a lock of TH to iterate THGs. And we hold a lock of THG to unregister TH from it (locking order is TH -> THG).
3. When suspending and resuming THs in THG, we first hold a lock of THG. And then, we hold a lock of TH to suspend and resume it (locking order is THG -> TH).
4. When destroying THG, we need to hold a lock of TH to unregister THG from TH. We can hold a lock of THG before that (locking order is THG -> TH).
Then, it easily causes dead lock. We cannot swap the locking order of (2) since iterating THG requires a lock of TH.
To solve this problem, we use std::shared_ptr and std::weak_ptr.
1. When adding WTF::Thread (TH) to WTF::ThreadGroup (THG), we first hold THG, and hold a lock of TH. (THG -> TH)
2. When TH dies, TH first hold lock of TH. And we use std::weak_ptr<>::lock() to retain non-destructed ThreadGroups.
If some of ThreadGroups are dying, we just ignore them. It is ok because such a ThreadGroup will be destructed. So we do not need to unregister this thread from
such a ThreadGroup. Then, we have Vector<std::shared_ptr<ThreadGroup>>. So we unlock a lock of TH. To unregister a thread from thread group, we first hold a
lock of THG and then hold a lock of TH. Both lifetime is ensured: THG is retained by std::shared_ptr. And TH is itself. (TH), (THG -> TH).
3. When suspending and resuming THs in THG, we first hold a lock of THG. And then, we hold a lock of TH to suspend and resume it (THG -> TH).
4. When destroying THG, we hold a lock of THG. And hold a lock of TH. During holding THG's lock, registered thread never dies because (2) holds THG lock. (THG -> TH).
We also fix suspend and resume locking mechanism to avoid dead lock. We should hold the global lock when suspending and resuming.
If we use per-thread lock, the suspended thread can hold the lock of the other threads. It causes dead lock.
* WTF.xcodeproj/project.pbxproj:
* wtf/AutomaticThread.cpp:
* wtf/CMakeLists.txt:
* wtf/CrossThreadCopier.h:
* wtf/ParkingLot.h:
* wtf/ThreadGroup.cpp: Copied from Source/JavaScriptCore/wasm/WasmMachineThreads.cpp.
(WTF::ThreadGroup::~ThreadGroup):
(WTF::ThreadGroup::add):
(WTF::ThreadGroup::addCurrentThread):
* wtf/ThreadGroup.h: Copied from Source/JavaScriptCore/wasm/WasmMachineThreads.cpp.
(WTF::ThreadGroup::create):
(WTF::ThreadGroup::threads):
(WTF::ThreadGroup::getLock):
(WTF::ThreadGroup::weakFromThis):
* wtf/Threading.cpp:
(WTF::shouldRemoveThreadFromThreadGroup):
(WTF::Thread::didExit):
(WTF::Thread::addToThreadGroup):
(WTF::Thread::removeFromThreadGroup):
* wtf/Threading.h:
* wtf/ThreadingPthreads.cpp:
(WTF::Thread::resume):
(WTF::Thread::getRegisters):
* wtf/ThreadingWin.cpp:
(WTF::Thread::resume):
(WTF::Thread::getRegisters):
Tools:
Add WTF::ThreadGroup tests.
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/ThreadGroup.cpp: Added.
(TestWebKitAPI::testThreadGroup):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/191464@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@219653 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-07-19 08:43:57 +00:00
|
|
|
class Thread;
|
|
|
|
|
2015-08-11 19:51:35 +00:00
|
|
|
class ParkingLot {
|
2019-08-12 20:57:15 +00:00
|
|
|
WTF_MAKE_FAST_ALLOCATED;
|
|
|
|
|
2015-08-11 19:51:35 +00:00
|
|
|
ParkingLot() = delete;
|
|
|
|
ParkingLot(const ParkingLot&) = delete;
|
|
|
|
|
|
|
|
public:
|
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows
https://bugs.webkit.org/show_bug.cgi?id=152045
Reviewed by Andy Estes.
Source/JavaScriptCore:
Probably the nicest example of why this patch is a good idea is the change in
AtomicsObject.cpp.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
* runtime/AtomicsObject.cpp:
(JSC::atomicsFuncWait):
Source/WebCore:
No new layout tests because no new behavior. The new WTF time classes have some unit tests
in TestWebKitAPI.
* fileapi/ThreadableBlobRegistry.cpp:
(WebCore::ThreadableBlobRegistry::blobSize):
* platform/MainThreadSharedTimer.h:
* platform/SharedTimer.h:
* platform/ThreadTimers.cpp:
(WebCore::ThreadTimers::updateSharedTimer):
* platform/cf/MainThreadSharedTimerCF.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/efl/MainThreadSharedTimerEfl.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/glib/MainThreadSharedTimerGLib.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/win/MainThreadSharedTimerWin.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* workers/WorkerRunLoop.cpp:
(WebCore::WorkerRunLoop::runInMode):
Source/WebKit2:
* Platform/IPC/Connection.cpp:
(IPC::Connection::SyncMessageState::wait):
(IPC::Connection::sendMessage):
(IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting):
(IPC::Connection::waitForMessage):
(IPC::Connection::sendSyncMessage):
(IPC::Connection::waitForSyncReply):
* Platform/IPC/Connection.h:
(IPC::Connection::sendSync):
(IPC::Connection::waitForAndDispatchImmediately):
* Platform/IPC/MessageSender.h:
(IPC::MessageSender::sendSync):
* UIProcess/ChildProcessProxy.h:
(WebKit::ChildProcessProxy::sendSync):
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/Storage/StorageManager.cpp:
(WebKit::StorageManager::applicationWillTerminate):
* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/WebResourceLoadStatisticsStore.cpp:
(WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate):
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h:
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm:
(-[WKOneShotDisplayLinkHandler displayLinkFired:]):
(WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree):
(WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay):
(WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm:
(WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/WKImmediateActionController.mm:
(-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]):
* UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::stringSelectionForPasteboard):
(WebKit::WebPageProxy::dataSelectionForPasteboard):
(WebKit::WebPageProxy::readSelectionFromPasteboard):
(WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent):
(WebKit::WebPageProxy::acceptsFirstMouse):
* WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::runBeforeUnloadConfirmPanel):
(WebKit::WebChromeClient::runJavaScriptAlert):
(WebKit::WebChromeClient::runJavaScriptConfirm):
(WebKit::WebChromeClient::runJavaScriptPrompt):
(WebKit::WebChromeClient::print):
(WebKit::WebChromeClient::exceededDatabaseQuota):
(WebKit::WebChromeClient::reachedApplicationCacheOriginQuota):
* WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp:
(WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse):
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::postSynchronousMessageForTesting):
Source/WTF:
We used to use 'double' for all time measurements. Sometimes it was milliseconds,
sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time
since some epoch. When we spoke of time since epoch, we either used a monotonic clock or
a wall clock. The type - always 'double' - never told us what kind of time we had, even
though there were roughly six of them (sec interval, ms interval, sec since epoch on wall,
ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic).
At some point, we thought that it would be a good idea to replace these doubles with
std::chrono. But since replacing some things with std::chrono, we found it to be terribly
inconvenient:
- Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say
std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally
duration_cast would not even be a thing.
- No overflow protection. std::chrono uses integers by default and using anything else is
clumsy. But the integer math is done without regard for the rough edges of integer math,
so any cast between std::chrono types risks overflow. Any comparison risks overflow
because it may do conversions silently. We have even found bugs where some C++
implementations had more overflows than others, which ends up being a special kind of
hell. In many cases, the overflow also has nasal demons.
It's an error to represent time using integers. It would have been excusable back when
floating point math was not guaranteed to be supported on all platforms, but that would
have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit
for floating point:
- Floating point preserves precision under multiplication in all but extreme cases, so
using floating point for time means that unit conversions are almost completely
lossless. This means that we don't have to think very hard about what units to use. In
this patch, we use seconds almost everywhere. We only convert at boundaries, like an API
boundary that wants something other than seconds.
- Floating point makes it easy to reason about infinity, which is something that time code
wants to do a lot. Example: when would you like to timeout? Infinity please! This is the
most elegant way of having an API support both a timeout variant and a no-timeout
variant.
- Floating point does well-understood things when math goes wrong, and these things are
pretty well optimized to match what a mathematician would do when computing with real
numbers represented using scientific notation with a finite number of significant
digits. This means that time math under floating point looks like normal math. On the
other hand, std::chrono time math looks like garbage because you have to always check
for multiple possible UB corners whenever you touch large integers. Integers that
represent time are very likely to be large and you don't have to do much to overflow
them. At this time, based on the number of bugs we have already seen due to chrono
overflows, I am not certain that we even understand what are all of the corner cases
that we should even check for.
This patch introduces a new set of timekeeping classes that are all based on double, and
all internally use seconds. These classes support algebraic typing. The classes are:
- Seconds: this is for measuring a duration.
- WallTime: time since epoch according to a wall clock (aka real time clock).
- MonotonicTime: time since epoch according to a monotonic clock.
- ClockType: enum that says either Wall or Monotonic.
- TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a
wall time or a monotonic time.
All of these classes behave like C++ values and are cheap to copy around since they are
very nearly POD. This supports comprehensive conversions between the various time types.
Most of this is by way of algebra. Here are just some of the rules we recognize:
WallTime = WallTime + Seconds
Seconds = WallTime - WallTime
MonotonicTime = MonotonicTime + Seconds
etc...
We support negative, infinite, and NaN times because math.
We support conversions between MonotonicTime and WallTime, like:
WallTime wt = mt.approximateWallTime()
This is called this "approximate" because the only way to do it is to get the current time
on both clocks and convert relative to that.
Many of our APIs would be happy using whatever notion of time the user wanted to use. For
those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You
can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This
means that if you use a WallTime with Condition::waitUntil, then Condition's internal
logic for when it should wake up makes its decision based on the current WallTime - but if
you use MonotonicTime then waitUntil will make its decision based on current
MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono
did not have the concept of a dynamic clock type.
This patch does not include conversions between std::chrono and these new time classes,
because past experience shows that we're quite bad at getting conversions between
std::chrono and anything else right. Also, I didn't need such conversion code because this
patch only converts code that transitively touches ParkingLot and Condition. It was easy
to get all of that code onto the new time classes.
* WTF.xcodeproj/project.pbxproj:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/ClockType.cpp: Added.
(WTF::printInternal):
* wtf/ClockType.h: Added.
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::waitFor):
(WTF::ConditionBase::wait):
(WTF::ConditionBase::waitUntilWallClockSeconds): Deleted.
(WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted.
(WTF::ConditionBase::waitForSeconds): Deleted.
(WTF::ConditionBase::waitForSecondsImpl): Deleted.
(WTF::ConditionBase::waitForImpl): Deleted.
(WTF::ConditionBase::absoluteFromRelative): Deleted.
* wtf/CrossThreadQueue.h:
(WTF::CrossThreadQueue<DataType>::waitForMessage):
* wtf/CurrentTime.cpp:
(WTF::sleep):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime): Deleted.
* wtf/MonotonicTime.cpp: Added.
(WTF::MonotonicTime::now):
(WTF::MonotonicTime::approximateWallTime):
(WTF::MonotonicTime::dump):
(WTF::MonotonicTime::sleep):
* wtf/MonotonicTime.h: Added.
(WTF::MonotonicTime::MonotonicTime):
(WTF::MonotonicTime::fromRawDouble):
(WTF::MonotonicTime::infinity):
(WTF::MonotonicTime::secondsSinceEpoch):
(WTF::MonotonicTime::approximateMonotonicTime):
(WTF::MonotonicTime::operator bool):
(WTF::MonotonicTime::operator+):
(WTF::MonotonicTime::operator-):
(WTF::MonotonicTime::operator+=):
(WTF::MonotonicTime::operator-=):
(WTF::MonotonicTime::operator==):
(WTF::MonotonicTime::operator!=):
(WTF::MonotonicTime::operator<):
(WTF::MonotonicTime::operator>):
(WTF::MonotonicTime::operator<=):
(WTF::MonotonicTime::operator>=):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkCount):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
* wtf/Seconds.cpp: Added.
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::dump):
(WTF::Seconds::sleep):
* wtf/Seconds.h: Added.
(WTF::Seconds::Seconds):
(WTF::Seconds::value):
(WTF::Seconds::seconds):
(WTF::Seconds::milliseconds):
(WTF::Seconds::microseconds):
(WTF::Seconds::nanoseconds):
(WTF::Seconds::fromMilliseconds):
(WTF::Seconds::fromMicroseconds):
(WTF::Seconds::fromNanoseconds):
(WTF::Seconds::infinity):
(WTF::Seconds::operator bool):
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::operator*):
(WTF::Seconds::operator/):
(WTF::Seconds::operator+=):
(WTF::Seconds::operator-=):
(WTF::Seconds::operator*=):
(WTF::Seconds::operator/=):
(WTF::Seconds::operator==):
(WTF::Seconds::operator!=):
(WTF::Seconds::operator<):
(WTF::Seconds::operator>):
(WTF::Seconds::operator<=):
(WTF::Seconds::operator>=):
* wtf/TimeWithDynamicClockType.cpp: Added.
(WTF::TimeWithDynamicClockType::now):
(WTF::TimeWithDynamicClockType::nowWithSameClock):
(WTF::TimeWithDynamicClockType::wallTime):
(WTF::TimeWithDynamicClockType::monotonicTime):
(WTF::TimeWithDynamicClockType::approximateWallTime):
(WTF::TimeWithDynamicClockType::approximateMonotonicTime):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator<):
(WTF::TimeWithDynamicClockType::operator>):
(WTF::TimeWithDynamicClockType::operator<=):
(WTF::TimeWithDynamicClockType::operator>=):
(WTF::TimeWithDynamicClockType::dump):
(WTF::TimeWithDynamicClockType::sleep):
* wtf/TimeWithDynamicClockType.h: Added.
(WTF::TimeWithDynamicClockType::TimeWithDynamicClockType):
(WTF::TimeWithDynamicClockType::fromRawDouble):
(WTF::TimeWithDynamicClockType::secondsSinceEpoch):
(WTF::TimeWithDynamicClockType::clockType):
(WTF::TimeWithDynamicClockType::withSameClockAndRawDouble):
(WTF::TimeWithDynamicClockType::operator bool):
(WTF::TimeWithDynamicClockType::operator+):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator+=):
(WTF::TimeWithDynamicClockType::operator-=):
(WTF::TimeWithDynamicClockType::operator==):
(WTF::TimeWithDynamicClockType::operator!=):
* wtf/WallTime.cpp: Added.
(WTF::WallTime::now):
(WTF::WallTime::approximateMonotonicTime):
(WTF::WallTime::dump):
(WTF::WallTime::sleep):
* wtf/WallTime.h: Added.
(WTF::WallTime::WallTime):
(WTF::WallTime::fromRawDouble):
(WTF::WallTime::infinity):
(WTF::WallTime::secondsSinceEpoch):
(WTF::WallTime::approximateWallTime):
(WTF::WallTime::operator bool):
(WTF::WallTime::operator+):
(WTF::WallTime::operator-):
(WTF::WallTime::operator+=):
(WTF::WallTime::operator-=):
(WTF::WallTime::operator==):
(WTF::WallTime::operator!=):
(WTF::WallTime::operator<):
(WTF::WallTime::operator>):
(WTF::WallTime::operator<=):
(WTF::WallTime::operator>=):
* wtf/threads/BinarySemaphore.cpp:
(WTF::BinarySemaphore::wait):
* wtf/threads/BinarySemaphore.h:
Tools:
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/Condition.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp:
(TestWebKitAPI::ToUpperConverter::stopProducing):
(TestWebKitAPI::ToUpperConverter::stopConsuming):
* TestWebKitAPI/Tests/WTF/Time.cpp: Added.
(WTF::operator<<):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/182152@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
|
|
|
// ParkingLot will accept any kind of time and convert it internally, but this typedef tells
|
|
|
|
// you what kind of time ParkingLot would be able to use without conversions. It's sad that
|
|
|
|
// this is WallTime not MonotonicTime, but that's just how OS wait functions work. However,
|
|
|
|
// because ParkingLot evaluates whether it should wait by checking if your time has passed
|
|
|
|
// using whatever clock you used, specifying timeouts in MonotonicTime is semantically better.
|
|
|
|
// For example, if the user sets his computer's clock back during the time that you wanted to
|
|
|
|
// wait for one second, and you specified the timeout using the MonotonicTime, then ParkingLot
|
|
|
|
// will be smart enough to know that your one second has elapsed.
|
|
|
|
typedef WallTime Time;
|
Replace all remaining uses of WTF::Mutex with WTF::Lock
https://bugs.webkit.org/show_bug.cgi?id=148089
Reviewed by Geoffrey Garen.
Source/WebCore:
No new tests because no new behavior.
* Modules/webaudio/AsyncAudioDecoder.cpp:
(WebCore::AsyncAudioDecoder::AsyncAudioDecoder):
(WebCore::AsyncAudioDecoder::runLoop):
* Modules/webaudio/AsyncAudioDecoder.h:
* Modules/webaudio/AudioContext.h:
* Modules/webaudio/MediaStreamAudioSource.cpp:
(WebCore::MediaStreamAudioSource::addAudioConsumer):
(WebCore::MediaStreamAudioSource::removeAudioConsumer):
(WebCore::MediaStreamAudioSource::setAudioFormat):
(WebCore::MediaStreamAudioSource::consumeAudio):
* Modules/webaudio/MediaStreamAudioSource.h:
* Modules/webdatabase/Database.cpp:
(WebCore::Database::close):
(WebCore::Database::runTransaction):
(WebCore::Database::inProgressTransactionCompleted):
(WebCore::Database::hasPendingTransaction):
* Modules/webdatabase/Database.h:
* Modules/webdatabase/DatabaseTask.cpp:
(WebCore::DatabaseTaskSynchronizer::taskCompleted):
* Modules/webdatabase/DatabaseTask.h:
* Modules/webdatabase/DatabaseThread.cpp:
(WebCore::DatabaseThread::start):
(WebCore::DatabaseThread::databaseThread):
* Modules/webdatabase/DatabaseThread.h:
* Modules/webdatabase/DatabaseTracker.cpp:
(WebCore::DatabaseTracker::setDatabaseDirectoryPath):
(WebCore::DatabaseTracker::canEstablishDatabase):
(WebCore::DatabaseTracker::retryCanEstablishDatabase):
(WebCore::DatabaseTracker::hasEntryForOrigin):
(WebCore::DatabaseTracker::getMaxSizeForDatabase):
(WebCore::DatabaseTracker::closeAllDatabases):
(WebCore::DatabaseTracker::fullPathForDatabase):
(WebCore::DatabaseTracker::origins):
(WebCore::DatabaseTracker::databaseNamesForOrigin):
(WebCore::DatabaseTracker::detailsForNameAndOrigin):
(WebCore::DatabaseTracker::setDatabaseDetails):
(WebCore::DatabaseTracker::doneCreatingDatabase):
(WebCore::DatabaseTracker::addOpenDatabase):
(WebCore::DatabaseTracker::removeOpenDatabase):
(WebCore::DatabaseTracker::getOpenDatabases):
(WebCore::DatabaseTracker::originLockFor):
(WebCore::DatabaseTracker::quotaForOrigin):
(WebCore::DatabaseTracker::setQuota):
(WebCore::DatabaseTracker::deleteOrigin):
(WebCore::DatabaseTracker::deleteDatabase):
(WebCore::DatabaseTracker::deleteDatabaseFile):
(WebCore::DatabaseTracker::removeDeletedOpenedDatabases):
(WebCore::DatabaseTracker::deleteDatabaseFileIfEmpty):
(WebCore::DatabaseTracker::openDatabaseMutex):
(WebCore::DatabaseTracker::setClient):
(WebCore::notificationMutex):
(WebCore::DatabaseTracker::scheduleNotifyDatabaseChanged):
(WebCore::DatabaseTracker::notifyDatabasesChanged):
* Modules/webdatabase/DatabaseTracker.h:
* Modules/webdatabase/OriginLock.h:
* Modules/webdatabase/SQLCallbackWrapper.h:
(WebCore::SQLCallbackWrapper::clear):
(WebCore::SQLCallbackWrapper::unwrap):
(WebCore::SQLCallbackWrapper::hasCallback):
* Modules/webdatabase/SQLTransactionBackend.cpp:
(WebCore::SQLTransactionBackend::doCleanup):
(WebCore::SQLTransactionBackend::enqueueStatementBackend):
(WebCore::SQLTransactionBackend::getNextStatement):
* Modules/webdatabase/SQLTransactionBackend.h:
* bindings/js/WorkerScriptController.cpp:
(WebCore::WorkerScriptController::scheduleExecutionTermination):
(WebCore::WorkerScriptController::isExecutionTerminating):
* bindings/js/WorkerScriptController.h:
* dom/default/PlatformMessagePortChannel.cpp:
(WebCore::MessagePortChannel::postMessageToRemote):
(WebCore::MessagePortChannel::tryGetMessageFromRemote):
(WebCore::MessagePortChannel::isConnectedTo):
(WebCore::MessagePortChannel::hasPendingActivity):
(WebCore::MessagePortChannel::locallyEntangledPort):
(WebCore::PlatformMessagePortChannel::setRemotePort):
(WebCore::PlatformMessagePortChannel::entangledChannel):
(WebCore::PlatformMessagePortChannel::closeInternal):
* dom/default/PlatformMessagePortChannel.h:
* loader/icon/IconDatabase.cpp:
(WebCore::IconDatabase::removeAllIcons):
(WebCore::IconDatabase::synchronousIconForPageURL):
(WebCore::IconDatabase::synchronousNativeIconForPageURL):
(WebCore::IconDatabase::synchronousIconURLForPageURL):
(WebCore::IconDatabase::retainIconForPageURL):
(WebCore::IconDatabase::performRetainIconForPageURL):
(WebCore::IconDatabase::releaseIconForPageURL):
(WebCore::IconDatabase::performReleaseIconForPageURL):
(WebCore::IconDatabase::setIconDataForIconURL):
(WebCore::IconDatabase::setIconURLForPageURL):
(WebCore::IconDatabase::synchronousLoadDecisionForIconURL):
(WebCore::IconDatabase::synchronousIconDataKnownForIconURL):
(WebCore::IconDatabase::pageURLMappingCount):
(WebCore::IconDatabase::retainedPageURLCount):
(WebCore::IconDatabase::iconRecordCount):
(WebCore::IconDatabase::iconRecordCountWithData):
(WebCore::IconDatabase::wakeSyncThread):
(WebCore::IconDatabase::scheduleOrDeferSyncTimer):
(WebCore::IconDatabase::isOpenBesidesMainThreadCallbacks):
(WebCore::IconDatabase::databasePath):
(WebCore::IconDatabase::getOrCreatePageURLRecord):
(WebCore::IconDatabase::iconDatabaseSyncThread):
(WebCore::IconDatabase::performOpenInitialization):
(WebCore::IconDatabase::performURLImport):
(WebCore::IconDatabase::syncThreadMainLoop):
(WebCore::IconDatabase::performPendingRetainAndReleaseOperations):
(WebCore::IconDatabase::readFromDatabase):
(WebCore::IconDatabase::writeToDatabase):
(WebCore::IconDatabase::pruneUnretainedIcons):
(WebCore::IconDatabase::cleanupSyncThread):
* loader/icon/IconDatabase.h:
* page/scrolling/ScrollingTree.cpp:
(WebCore::ScrollingTree::shouldHandleWheelEventSynchronously):
(WebCore::ScrollingTree::commitNewTreeState):
(WebCore::ScrollingTree::setMainFramePinState):
(WebCore::ScrollingTree::mainFrameScrollPosition):
(WebCore::ScrollingTree::setMainFrameScrollPosition):
(WebCore::ScrollingTree::isPointInNonFastScrollableRegion):
(WebCore::ScrollingTree::isRubberBandInProgress):
(WebCore::ScrollingTree::setMainFrameIsRubberBanding):
(WebCore::ScrollingTree::isScrollSnapInProgress):
(WebCore::ScrollingTree::setMainFrameIsScrollSnapping):
(WebCore::ScrollingTree::setCanRubberBandState):
(WebCore::ScrollingTree::rubberBandsAtLeft):
(WebCore::ScrollingTree::rubberBandsAtRight):
(WebCore::ScrollingTree::rubberBandsAtBottom):
(WebCore::ScrollingTree::rubberBandsAtTop):
(WebCore::ScrollingTree::setScrollPinningBehavior):
(WebCore::ScrollingTree::scrollPinningBehavior):
(WebCore::ScrollingTree::willWheelEventStartSwipeGesture):
(WebCore::ScrollingTree::latchedNode):
(WebCore::ScrollingTree::setLatchedNode):
(WebCore::ScrollingTree::clearLatchedNode):
* page/scrolling/ScrollingTree.h:
* platform/MemoryPressureHandler.h:
* platform/audio/HRTFDatabaseLoader.cpp:
(WebCore::HRTFDatabaseLoader::loadAsynchronously):
(WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion):
* platform/audio/HRTFDatabaseLoader.h:
* platform/cocoa/MemoryPressureHandlerCocoa.mm:
(WebCore::MemoryPressureHandler::setReceivedMemoryPressure):
(WebCore::MemoryPressureHandler::clearMemoryPressure):
(WebCore::MemoryPressureHandler::shouldWaitForMemoryClearMessage):
(WebCore::MemoryPressureHandler::respondToMemoryPressureIfNeeded):
* platform/graphics/DisplayRefreshMonitor.cpp:
(WebCore::DisplayRefreshMonitor::displayDidRefresh):
* platform/graphics/DisplayRefreshMonitor.h:
(WebCore::DisplayRefreshMonitor::setMonotonicAnimationStartTime):
(WebCore::DisplayRefreshMonitor::mutex):
* platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp:
(WebCore::MediaPlayerPrivateAVFoundation::setDelayCallbacks):
(WebCore::MediaPlayerPrivateAVFoundation::clearMainThreadPendingFlag):
(WebCore::MediaPlayerPrivateAVFoundation::dispatchNotification):
* platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h:
* platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp:
(WebCore::AVFWrapper::callbackContext):
(WebCore::AVFWrapper::~AVFWrapper):
(WebCore::AVFWrapper::mapLock):
(WebCore::AVFWrapper::addToMap):
(WebCore::AVFWrapper::removeFromMap):
(WebCore::AVFWrapper::periodicTimeObserverCallback):
(WebCore::AVFWrapper::processNotification):
(WebCore::AVFWrapper::loadPlayableCompletionCallback):
(WebCore::AVFWrapper::loadMetadataCompletionCallback):
(WebCore::AVFWrapper::seekCompletedCallback):
(WebCore::AVFWrapper::processCue):
(WebCore::AVFWrapper::legibleOutputCallback):
(WebCore::AVFWrapper::processShouldWaitForLoadingOfResource):
(WebCore::AVFWrapper::resourceLoaderShouldWaitForLoadingOfRequestedResource):
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp:
(WebCore::InbandTextTrackPrivateGStreamer::handleSample):
(WebCore::InbandTextTrackPrivateGStreamer::notifyTrackOfSample):
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h:
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:
(WebCore::TrackPrivateBaseGStreamer::tagsChanged):
(WebCore::TrackPrivateBaseGStreamer::notifyTrackOfTagsChanged):
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h:
* platform/graphics/mac/DisplayRefreshMonitorMac.cpp:
(WebCore::DisplayRefreshMonitorMac::requestRefreshCallback):
(WebCore::DisplayRefreshMonitorMac::displayLinkFired):
* platform/graphics/win/MediaPlayerPrivateMediaFoundation.cpp:
(WebCore::MediaPlayerPrivateMediaFoundation::addListener):
(WebCore::MediaPlayerPrivateMediaFoundation::removeListener):
(WebCore::MediaPlayerPrivateMediaFoundation::notifyDeleted):
(WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::Invoke):
(WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::onMediaPlayerDeleted):
* platform/graphics/win/MediaPlayerPrivateMediaFoundation.h:
* platform/ios/LegacyTileCache.h:
* platform/ios/LegacyTileCache.mm:
(WebCore::LegacyTileCache::setTilesOpaque):
(WebCore::LegacyTileCache::doLayoutTiles):
(WebCore::LegacyTileCache::setCurrentScale):
(WebCore::LegacyTileCache::commitScaleChange):
(WebCore::LegacyTileCache::layoutTilesNow):
(WebCore::LegacyTileCache::layoutTilesNowForRect):
(WebCore::LegacyTileCache::removeAllNonVisibleTiles):
(WebCore::LegacyTileCache::removeAllTiles):
(WebCore::LegacyTileCache::removeForegroundTiles):
(WebCore::LegacyTileCache::setContentReplacementImage):
(WebCore::LegacyTileCache::contentReplacementImage):
(WebCore::LegacyTileCache::tileCreationTimerFired):
(WebCore::LegacyTileCache::setNeedsDisplayInRect):
(WebCore::LegacyTileCache::updateTilingMode):
(WebCore::LegacyTileCache::setTilingMode):
(WebCore::LegacyTileCache::doPendingRepaints):
(WebCore::LegacyTileCache::flushSavedDisplayRects):
(WebCore::LegacyTileCache::prepareToDraw):
* platform/ios/LegacyTileLayerPool.h:
* platform/ios/LegacyTileLayerPool.mm:
(WebCore::LegacyTileLayerPool::addLayer):
(WebCore::LegacyTileLayerPool::takeLayerWithSize):
(WebCore::LegacyTileLayerPool::setCapacity):
(WebCore::LegacyTileLayerPool::prune):
(WebCore::LegacyTileLayerPool::drain):
* platform/network/curl/CurlDownload.cpp:
(WebCore::CurlDownloadManager::add):
(WebCore::CurlDownloadManager::remove):
(WebCore::CurlDownloadManager::getActiveDownloadCount):
(WebCore::CurlDownloadManager::getPendingDownloadCount):
(WebCore::CurlDownloadManager::stopThreadIfIdle):
(WebCore::CurlDownloadManager::updateHandleList):
(WebCore::CurlDownload::~CurlDownload):
(WebCore::CurlDownload::init):
(WebCore::CurlDownload::getTempPath):
(WebCore::CurlDownload::getUrl):
(WebCore::CurlDownload::getResponse):
(WebCore::CurlDownload::closeFile):
(WebCore::CurlDownload::didReceiveHeader):
(WebCore::CurlDownload::didReceiveData):
(WebCore::CurlDownload::didFail):
* platform/network/curl/CurlDownload.h:
* platform/network/curl/ResourceHandleManager.cpp:
(WebCore::cookieJarPath):
(WebCore::sharedResourceMutex):
(WebCore::curl_lock_callback):
(WebCore::curl_unlock_callback):
* platform/network/ios/QuickLook.mm:
(WebCore::QLDirectoryAttributes):
(qlPreviewConverterDictionaryMutex):
(WebCore::addQLPreviewConverterWithFileForURL):
(WebCore::qlPreviewConverterUTIForURL):
(WebCore::removeQLPreviewConverterForURL):
(WebCore::safeQLURLForDocumentURLAndResourceURL):
* platform/sql/SQLiteDatabase.cpp:
(WebCore::SQLiteDatabase::close):
(WebCore::SQLiteDatabase::maximumSize):
(WebCore::SQLiteDatabase::setMaximumSize):
(WebCore::SQLiteDatabase::pageSize):
(WebCore::SQLiteDatabase::freeSpaceSize):
(WebCore::SQLiteDatabase::totalSize):
(WebCore::SQLiteDatabase::runIncrementalVacuumCommand):
(WebCore::SQLiteDatabase::setAuthorizer):
* platform/sql/SQLiteDatabase.h:
(WebCore::SQLiteDatabase::databaseMutex):
* platform/sql/SQLiteStatement.cpp:
(WebCore::SQLiteStatement::prepare):
(WebCore::SQLiteStatement::step):
* workers/WorkerThread.cpp:
(WebCore::WorkerThread::start):
(WebCore::WorkerThread::workerThread):
(WebCore::WorkerThread::stop):
* workers/WorkerThread.h:
Source/WebKit:
* Storage/StorageAreaSync.cpp:
(WebCore::StorageAreaSync::syncTimerFired):
(WebCore::StorageAreaSync::markImported):
(WebCore::StorageAreaSync::blockUntilImportComplete):
(WebCore::StorageAreaSync::performSync):
* Storage/StorageAreaSync.h:
* Storage/StorageTracker.cpp:
(WebCore::StorageTracker::setDatabaseDirectoryPath):
(WebCore::StorageTracker::finishedImportingOriginIdentifiers):
(WebCore::StorageTracker::syncImportOriginIdentifiers):
(WebCore::StorageTracker::syncFileSystemAndTrackerDatabase):
(WebCore::StorageTracker::setOriginDetails):
(WebCore::StorageTracker::syncSetOriginDetails):
(WebCore::StorageTracker::origins):
(WebCore::StorageTracker::deleteAllOrigins):
(WebCore::StorageTracker::syncDeleteAllOrigins):
(WebCore::StorageTracker::deleteOrigin):
(WebCore::StorageTracker::syncDeleteOrigin):
(WebCore::StorageTracker::canDeleteOrigin):
(WebCore::StorageTracker::cancelDeletingOrigin):
(WebCore::StorageTracker::diskUsageForOrigin):
* Storage/StorageTracker.h:
Source/WebKit/ios:
* WebCoreSupport/WebFixedPositionContent.mm:
(WebFixedPositionContentDataLock):
(-[WebFixedPositionContent scrollOrZoomChanged:]):
(-[WebFixedPositionContent overflowScrollPositionForLayer:changedTo:]):
(-[WebFixedPositionContent setViewportConstrainedLayers:stickyContainerMap:]):
(-[WebFixedPositionContent hasFixedOrStickyPositionLayers]):
(-[WebFixedPositionContent minimumOffsetFromFixedPositionLayersToAnchorEdge:ofRect:inLayer:]):
Source/WebKit/mac:
* Storage/WebDatabaseManager.mm:
(transactionBackgroundTaskIdentifierLock):
(+[WebDatabaseManager startBackgroundTask]):
(+[WebDatabaseManager endBackgroundTask]):
* WebView/WebView.mm:
(-[WebView _synchronizeCustomFixedPositionLayoutRect]):
(-[WebView _setCustomFixedPositionLayoutRectInWebThread:synchronize:]):
(-[WebView _setCustomFixedPositionLayoutRect:]):
(-[WebView _fetchCustomFixedPositionLayoutRect:]):
* WebView/WebViewData.h:
Source/WebKit/win:
* Plugins/PluginMainThreadScheduler.cpp:
(WebCore::PluginMainThreadScheduler::scheduleCall):
(WebCore::PluginMainThreadScheduler::registerPlugin):
(WebCore::PluginMainThreadScheduler::unregisterPlugin):
(WebCore::PluginMainThreadScheduler::dispatchCallsForPlugin):
* Plugins/PluginMainThreadScheduler.h:
* WebIconDatabase.cpp:
(WebIconDatabase::didRemoveAllIcons):
(WebIconDatabase::didImportIconURLForPageURL):
(WebIconDatabase::deliverNotifications):
* WebIconDatabase.h:
* WebLocalizableStrings.cpp:
(mainBundleLocStrings):
(frameworkLocStringsMutex):
(findCachedString):
(cacheString):
Source/WebKit2:
* DatabaseProcess/DatabaseProcess.cpp:
(WebKit::DatabaseProcess::postDatabaseTask):
(WebKit::DatabaseProcess::performNextDatabaseTask):
* DatabaseProcess/DatabaseProcess.h:
* DatabaseProcess/IndexedDB/UniqueIDBDatabase.cpp:
(WebKit::UniqueIDBDatabase::shutdown):
(WebKit::UniqueIDBDatabase::postMainThreadTask):
(WebKit::UniqueIDBDatabase::performNextMainThreadTask):
(WebKit::UniqueIDBDatabase::postDatabaseTask):
(WebKit::UniqueIDBDatabase::performNextDatabaseTask):
* DatabaseProcess/IndexedDB/UniqueIDBDatabase.h:
* Platform/IPC/Connection.cpp:
(IPC::Connection::sendSyncMessage):
(IPC::Connection::sendSyncMessageFromSecondaryThread):
(IPC::Connection::waitForSyncReply):
(IPC::Connection::processIncomingSyncReply):
(IPC::Connection::connectionDidClose):
* Platform/IPC/Connection.h:
* Shared/CoordinatedGraphics/CoordinatedGraphicsScene.cpp:
(WebKit::CoordinatedGraphicsScene::appendUpdate):
* Shared/CoordinatedGraphics/CoordinatedGraphicsScene.h:
* Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.cpp:
(WebKit::ThreadedCompositor::createCompositingThread):
(WebKit::ThreadedCompositor::runCompositingThread):
(WebKit::ThreadedCompositor::terminateCompositingThread):
* Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.h:
* Shared/Network/CustomProtocols/Cocoa/CustomProtocolManagerCocoa.mm:
(WebKit::CustomProtocolManager::addCustomProtocol):
(WebKit::CustomProtocolManager::removeCustomProtocol):
(WebKit::CustomProtocolManager::registerScheme):
(WebKit::CustomProtocolManager::unregisterScheme):
(WebKit::CustomProtocolManager::supportsScheme):
(WebKit::CustomProtocolManager::protocolForID):
* Shared/Network/CustomProtocols/CustomProtocolManager.h:
* Shared/linux/SeccompFilters/SeccompBroker.cpp:
* WebProcess/Plugins/PluginProcessConnectionManager.cpp:
(WebKit::PluginProcessConnectionManager::getPluginProcessConnection):
(WebKit::PluginProcessConnectionManager::removePluginProcessConnection):
(WebKit::PluginProcessConnectionManager::pluginProcessCrashed):
* WebProcess/Plugins/PluginProcessConnectionManager.h:
* WebProcess/WebPage/EventDispatcher.cpp:
(WebKit::EventDispatcher::addScrollingTreeForPage):
(WebKit::EventDispatcher::removeScrollingTreeForPage):
(WebKit::EventDispatcher::wheelEvent):
* WebProcess/WebPage/EventDispatcher.h:
* WebProcess/soup/WebKitSoupRequestInputStream.cpp:
(webkitSoupRequestInputStreamReadAsync):
(webkitSoupRequestInputStreamAddData):
Source/WTF:
This also beefs up and rationalizes the Condition API, so that it can deal with units of time
other than just steady_clock. This makes it easier to port ThreadCondition and
std::condition_variable code over to Condition. This patch does not take a position on what
kind of time is best; from reading a lot of the uses in WebCore, it seems like our use of
double to measure seconds is often nicer than the many different classes in std::chrono.
Also added a Condition speed test, to make sure that all of this is a good idea. And indeed it
is. The 1-producer/1-consumer scenario with a 100-element queue runs 36x faster using
Lock/Condition than Mutex/ThreadCondition when you use the notifyOne()-per-enqueue style. It
runs 58x faster with Lock/Condition when you use the notifyAll()-at-boundary style. Note that
I have a bug open for making the notifyOne()-per-enqueue style even faster:
https://bugs.webkit.org/show_bug.cgi?id=148090. Also, the 10-consumer/10-producer scenario with
a 100-element queue runs 20x faster with Lock/Condition for notifyOne()-per-enqueue and 30x
faster with notifyAll()-at-boundary. The only way to tweak the test to get
Mutex/ThreadCondition to win is to have one producer, one consumer, a 1-element queue, and use
the notifyOne()-per-enqueue style. In that case, one of the two threads is going to be waiting
most of the time and the test basically measures wake-up latency and nothing else. Because
Condition::wait() does a little bit more work than ThreadCondition::wait(),
Mutex/ThreadCondition end up running 3% faster in this test case. But if you vary any of the
parameters of the test, Mutex/ThreadCondition ends up losing - all it takes is more threads or
a queue size of 5 or more. To my knowledge, we never do producer/consumer with a queue bounded
to one element precisely because that approach is the least efficient regardless of locking
algorithm. For example, neither WTF::MessageQueue nor DFG::Worklist have any bounds on their
queue size. So, it seems that replacing all uses of system mutexes and condition variables with
our own thing is a great idea.
* benchmarks/LockSpeedTest.cpp:
* benchmarks/ConditionSpeedTest.cpp: Added.
* wtf/Condition.h:
(WTF::Condition::Condition):
(WTF::Condition::waitUntil):
(WTF::Condition::waitFor):
(WTF::Condition::wait):
(WTF::Condition::waitUntilWallClockSeconds):
(WTF::Condition::waitUntilMonotonicClockSeconds):
(WTF::Condition::notifyOne):
(WTF::Condition::notifyAll):
(WTF::Condition::waitForSecondsImpl):
(WTF::Condition::waitForImpl):
(WTF::Condition::absoluteFromRelative):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime):
(WTF::MessageQueue<DataType>::append):
(WTF::MessageQueue<DataType>::appendAndKill):
(WTF::MessageQueue<DataType>::appendAndCheckEmpty):
(WTF::MessageQueue<DataType>::prepend):
(WTF::MessageQueue<DataType>::removeIf):
(WTF::MessageQueue<DataType>::isEmpty):
(WTF::MessageQueue<DataType>::kill):
(WTF::MessageQueue<DataType>::killed):
* wtf/ParallelJobsGeneric.cpp:
(WTF::ParallelEnvironment::ThreadPrivate::execute):
(WTF::ParallelEnvironment::ThreadPrivate::waitForFinish):
(WTF::ParallelEnvironment::ThreadPrivate::workerThread):
* wtf/ParallelJobsGeneric.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionally):
* wtf/ParkingLot.h:
(WTF::ParkingLot::compareAndPark):
* wtf/ThreadingPthreads.cpp:
(WTF::initializeThreading):
* wtf/ThreadingWin.cpp:
(WTF::initializeThreading):
* wtf/dtoa.cpp:
(WTF::pow5mult):
* wtf/dtoa.h:
Tools:
* DumpRenderTree/JavaScriptThreading.cpp:
(javaScriptThreadsMutex):
(runJavaScriptThread):
(startJavaScriptThreads):
(stopJavaScriptThreads):
* TestWebKitAPI/Tests/WTF/Condition.cpp: Fixed a bug in the test that I found from turning the test into a benchmark.
* TestWebKitAPI/Tests/WTF/WorkQueue.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/glib/WorkQueueGLib.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/166252@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188594 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-18 19:31:28 +00:00
|
|
|
|
2015-08-11 19:51:35 +00:00
|
|
|
// Parks the thread in a queue associated with the given address, which cannot be null. The
|
|
|
|
// parking only succeeds if the validation function returns true while the queue lock is held.
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
//
|
2015-08-13 20:42:11 +00:00
|
|
|
// If validation returns false, it will unlock the internal parking queue and then it will
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// return a null ParkResult (wasUnparked = false, token = 0) without doing anything else.
|
|
|
|
//
|
|
|
|
// If validation returns true, it will enqueue the thread, unlock the parking queue lock, call
|
|
|
|
// the beforeSleep function, and then it will sleep so long as the thread continues to be on the
|
|
|
|
// queue and the timeout hasn't fired. Finally, this returns wasUnparked = true if we actually
|
|
|
|
// got unparked or wasUnparked = false if the timeout was hit. When wasUnparked = true, the
|
|
|
|
// token will contain whatever token was returned from the callback to unparkOne(), or 0 if the
|
|
|
|
// thread was unparked using unparkAll() or the form of unparkOne() that doesn't take a
|
|
|
|
// callback.
|
|
|
|
//
|
|
|
|
// Note that beforeSleep is called with no locks held, so it's OK to do pretty much anything so
|
|
|
|
// long as you don't recursively call parkConditionally(). You can call unparkOne()/unparkAll()
|
|
|
|
// though. It's useful to use beforeSleep() to unlock some mutex in the implementation of
|
2015-08-13 20:42:11 +00:00
|
|
|
// Condition::wait().
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
struct ParkResult {
|
|
|
|
bool wasUnparked { false };
|
|
|
|
intptr_t token { 0 };
|
|
|
|
};
|
2016-04-20 04:25:02 +00:00
|
|
|
template<typename ValidationFunctor, typename BeforeSleepFunctor>
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
static ParkResult parkConditionally(
|
2015-08-13 20:42:11 +00:00
|
|
|
const void* address,
|
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
https://bugs.webkit.org/show_bug.cgi?id=160125
Reviewed by Geoffrey Garen and Keith Miller.
JSTests:
Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.
* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.
Source/bmalloc:
I needed to tryMemalign, so I added such a thing.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):
Source/JavaScriptCore:
In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.
Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.
At a high level the mechanical changes are:
- We use AuxiliaryBarrier instead of CopyBarrier.
- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
CheckedBoolean stuff, since it's so much more trouble than it's worth.
- The JITs have to emit inlined marked space allocations instead of inline copy space
allocations.
- Everyone has to get used to zeroing their butterflies after allocation instead of relying
on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
space doesn't.
That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:
- MarkedSpace always causes internal fragmentation for array allocations because the vector
length we choose when we resize usually leads to a cell size that doesn't correspond to any
size class. I got around this by making array allocations usually round up vectorLength to
the maximum allowed by the size class that we would have allocated in. Also,
ensureLengthSlow() and friends first make sure that the requested length can't just be
fulfilled with the current allocation size. This safeguard means that not every array
allocation has to do size class queries. For example, the fast path of new Array(length)
never does any size class queries, under the assumption that (1) the speed gained from
avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
size class query, is too small to offset the speed lost by doing the query on every
allocation and (2) new Array(length) is a pretty good hint that resizing is not very
likely.
- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
changes MarkedSpace size classes to use a linear progression for very small sizes followed
by a geometric progression that naturally transitions to a hyperbolic progression. We want
hyperbolic sizes when we get close to blockSize: for example the largest size we want is
payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
next size down should be payloadSize / 3 rounded down, and so on. After the last precise
size (80 bytes), we proceed using a geometric progression, but round up each size to
minimize slop at the end of the block. This naturally causes the geometric progression to
turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
- Large allocations caused massive internal fragmentation, since the smallest large
allocation had to use exactly blockSize, and the largest small allocation used
blockSize / 2. The next size up - the first large allocation size to require two blocks -
also had 50% internal fragmentation. This is because we required large allocations to be
blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
out great since (1) typed arrays won't use large allocations anyway since they have their
own malloc fallback and (2) large array butterflies already have a 8 byte header, which
means that the 8 byte base misalignment aligns the large array payload on a 16 byte
boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
rare as possible; for example, ExecState::vm() skips the check because we know that callees
must be small allocations. It's also possible to use template tricks to do one check for
cell container kind, and then invoke a function specialized for MarkedBlock or a function
specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
that get used from functions that are template-specialized like this. That's mostly to
speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
cutoff for large allocations is runtime-configurable, so long as you don't choose something
so small that callees end up large. I found that 400 bytes is roughly optimal. This means
that the MarkedBlock size classes end up being:
16, 32, 48, 64, 80, 112, 160, 224, 320
The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.
- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
MarkedBlocks because the list that contains the interesting meta-data is allocated on the
side, for better locality during a sequential walk. It's no longer necessary to scan
MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
bits because we now use versioned mark bits: to clear then, just increment the 64-bit
heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
together, these optimizations allowed me to reduce block size to 16KB without losing much
performance. There is some small perf loss on JetStream/splay, but not enough to hurt
JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
progression on membuster. That's not possible yet, since there is still enough per-block
overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
- Even after all of that, copying butterflies was still faster because it allowed us to skip
sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
larger than O(live), especially in an eden collection. Copying satisfies this premise while
mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
that we walk the block to build a free list. This means walking dead space. The new
allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
is completely empty, which is the common case because of the generational hypothesis: the
number of objects that survive an eden collection is a tiny fraction of the number of
objects that had been allocated, and this fraction is so small that there are typically
fewer than one survivors per MarkedBlock. This change was enough to make this change a net
win over tip-of-tree.
- FTL now shares the same allocation fast paths as everything else, which is great, because
bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
control flow, since it won't be able to improve the machine code we write ourselves. GC
fast paths are best written in assembly. So, I've empowered B3 to have even better support
for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
ability to constant-fold some allocator logic (in case we first assume that we're doing a
variable-length allocation but then realize that the length is known). I think it makes
sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
since this makes lowering easier (you can constant fold during lowering more easily) and it
reduces the amount of malloc traffic. In the future, we could teach B3 how to better
constant-fold this code. That would require allowing loads to be constant-folded, which is
doable but hella tricky.
- It used to be that if a logical object allocation required two physical allocations (first
the butterfly and then the cell), then the JIT would emit the code in such a way that a
failure in the second fast path would cause us to forget the successful first physical
allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
register to storing either the butterfly or null, because the butterfly register is anyway
going to be free inside the first allocation. The only overhead here is zeroing the
butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
path, which can then either allocate a butterfly or not. So now we never waste a successful
allocation. This patch implements such a solution both in DFG (where it's easy to do this
since we control registers already) and in FTL (where it's annoying, because mutable
"butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
one version of this patch, this wastage would resonate with some Kraken benchmark: the
benchmark would always allocate N small things followed by one bigger thing. The problem
was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
the JSObject size class, which both the small and big thing shared for their cell, could
hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
it allocated the big thing. So, it would end up having to allocate the big thing's large
butterfly twice, every single time! Ouch!
- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
of the allocator to initialize the object to whatever state they need. It used to be that
we would just use a dumb loop. I initially changed this so that we would end up in memset
for large allocations, but this didn't actually help performance that much. I got a much
better result by playing with different memsets written in assembly. First I wrote one
using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
"rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
of stores (if the size is known), but those paths in the JIT won't trigger for
NewArrayWithSize() if the size is large, since that takes us to the
operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
optimizations here are all in JSArray::create() - that's the hot place for large arrays
that need to be filled with holes.
All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.
We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.
This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):
Source/WebCore:
No new tests because no new WebCore behavior.
Just rewiring #includes.
* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:
Source/WebKit2:
Just rewiring some #includes.
* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:
Source/WTF:
I needed tryFastAlignedMalloc() so I added it.
* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.
Tools:
* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/179778@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
|
|
|
const ValidationFunctor& validation,
|
|
|
|
const BeforeSleepFunctor& beforeSleep,
|
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows
https://bugs.webkit.org/show_bug.cgi?id=152045
Reviewed by Andy Estes.
Source/JavaScriptCore:
Probably the nicest example of why this patch is a good idea is the change in
AtomicsObject.cpp.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
* runtime/AtomicsObject.cpp:
(JSC::atomicsFuncWait):
Source/WebCore:
No new layout tests because no new behavior. The new WTF time classes have some unit tests
in TestWebKitAPI.
* fileapi/ThreadableBlobRegistry.cpp:
(WebCore::ThreadableBlobRegistry::blobSize):
* platform/MainThreadSharedTimer.h:
* platform/SharedTimer.h:
* platform/ThreadTimers.cpp:
(WebCore::ThreadTimers::updateSharedTimer):
* platform/cf/MainThreadSharedTimerCF.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/efl/MainThreadSharedTimerEfl.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/glib/MainThreadSharedTimerGLib.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/win/MainThreadSharedTimerWin.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* workers/WorkerRunLoop.cpp:
(WebCore::WorkerRunLoop::runInMode):
Source/WebKit2:
* Platform/IPC/Connection.cpp:
(IPC::Connection::SyncMessageState::wait):
(IPC::Connection::sendMessage):
(IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting):
(IPC::Connection::waitForMessage):
(IPC::Connection::sendSyncMessage):
(IPC::Connection::waitForSyncReply):
* Platform/IPC/Connection.h:
(IPC::Connection::sendSync):
(IPC::Connection::waitForAndDispatchImmediately):
* Platform/IPC/MessageSender.h:
(IPC::MessageSender::sendSync):
* UIProcess/ChildProcessProxy.h:
(WebKit::ChildProcessProxy::sendSync):
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/Storage/StorageManager.cpp:
(WebKit::StorageManager::applicationWillTerminate):
* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/WebResourceLoadStatisticsStore.cpp:
(WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate):
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h:
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm:
(-[WKOneShotDisplayLinkHandler displayLinkFired:]):
(WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree):
(WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay):
(WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm:
(WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/WKImmediateActionController.mm:
(-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]):
* UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::stringSelectionForPasteboard):
(WebKit::WebPageProxy::dataSelectionForPasteboard):
(WebKit::WebPageProxy::readSelectionFromPasteboard):
(WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent):
(WebKit::WebPageProxy::acceptsFirstMouse):
* WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::runBeforeUnloadConfirmPanel):
(WebKit::WebChromeClient::runJavaScriptAlert):
(WebKit::WebChromeClient::runJavaScriptConfirm):
(WebKit::WebChromeClient::runJavaScriptPrompt):
(WebKit::WebChromeClient::print):
(WebKit::WebChromeClient::exceededDatabaseQuota):
(WebKit::WebChromeClient::reachedApplicationCacheOriginQuota):
* WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp:
(WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse):
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::postSynchronousMessageForTesting):
Source/WTF:
We used to use 'double' for all time measurements. Sometimes it was milliseconds,
sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time
since some epoch. When we spoke of time since epoch, we either used a monotonic clock or
a wall clock. The type - always 'double' - never told us what kind of time we had, even
though there were roughly six of them (sec interval, ms interval, sec since epoch on wall,
ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic).
At some point, we thought that it would be a good idea to replace these doubles with
std::chrono. But since replacing some things with std::chrono, we found it to be terribly
inconvenient:
- Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say
std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally
duration_cast would not even be a thing.
- No overflow protection. std::chrono uses integers by default and using anything else is
clumsy. But the integer math is done without regard for the rough edges of integer math,
so any cast between std::chrono types risks overflow. Any comparison risks overflow
because it may do conversions silently. We have even found bugs where some C++
implementations had more overflows than others, which ends up being a special kind of
hell. In many cases, the overflow also has nasal demons.
It's an error to represent time using integers. It would have been excusable back when
floating point math was not guaranteed to be supported on all platforms, but that would
have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit
for floating point:
- Floating point preserves precision under multiplication in all but extreme cases, so
using floating point for time means that unit conversions are almost completely
lossless. This means that we don't have to think very hard about what units to use. In
this patch, we use seconds almost everywhere. We only convert at boundaries, like an API
boundary that wants something other than seconds.
- Floating point makes it easy to reason about infinity, which is something that time code
wants to do a lot. Example: when would you like to timeout? Infinity please! This is the
most elegant way of having an API support both a timeout variant and a no-timeout
variant.
- Floating point does well-understood things when math goes wrong, and these things are
pretty well optimized to match what a mathematician would do when computing with real
numbers represented using scientific notation with a finite number of significant
digits. This means that time math under floating point looks like normal math. On the
other hand, std::chrono time math looks like garbage because you have to always check
for multiple possible UB corners whenever you touch large integers. Integers that
represent time are very likely to be large and you don't have to do much to overflow
them. At this time, based on the number of bugs we have already seen due to chrono
overflows, I am not certain that we even understand what are all of the corner cases
that we should even check for.
This patch introduces a new set of timekeeping classes that are all based on double, and
all internally use seconds. These classes support algebraic typing. The classes are:
- Seconds: this is for measuring a duration.
- WallTime: time since epoch according to a wall clock (aka real time clock).
- MonotonicTime: time since epoch according to a monotonic clock.
- ClockType: enum that says either Wall or Monotonic.
- TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a
wall time or a monotonic time.
All of these classes behave like C++ values and are cheap to copy around since they are
very nearly POD. This supports comprehensive conversions between the various time types.
Most of this is by way of algebra. Here are just some of the rules we recognize:
WallTime = WallTime + Seconds
Seconds = WallTime - WallTime
MonotonicTime = MonotonicTime + Seconds
etc...
We support negative, infinite, and NaN times because math.
We support conversions between MonotonicTime and WallTime, like:
WallTime wt = mt.approximateWallTime()
This is called this "approximate" because the only way to do it is to get the current time
on both clocks and convert relative to that.
Many of our APIs would be happy using whatever notion of time the user wanted to use. For
those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You
can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This
means that if you use a WallTime with Condition::waitUntil, then Condition's internal
logic for when it should wake up makes its decision based on the current WallTime - but if
you use MonotonicTime then waitUntil will make its decision based on current
MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono
did not have the concept of a dynamic clock type.
This patch does not include conversions between std::chrono and these new time classes,
because past experience shows that we're quite bad at getting conversions between
std::chrono and anything else right. Also, I didn't need such conversion code because this
patch only converts code that transitively touches ParkingLot and Condition. It was easy
to get all of that code onto the new time classes.
* WTF.xcodeproj/project.pbxproj:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/ClockType.cpp: Added.
(WTF::printInternal):
* wtf/ClockType.h: Added.
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::waitFor):
(WTF::ConditionBase::wait):
(WTF::ConditionBase::waitUntilWallClockSeconds): Deleted.
(WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted.
(WTF::ConditionBase::waitForSeconds): Deleted.
(WTF::ConditionBase::waitForSecondsImpl): Deleted.
(WTF::ConditionBase::waitForImpl): Deleted.
(WTF::ConditionBase::absoluteFromRelative): Deleted.
* wtf/CrossThreadQueue.h:
(WTF::CrossThreadQueue<DataType>::waitForMessage):
* wtf/CurrentTime.cpp:
(WTF::sleep):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime): Deleted.
* wtf/MonotonicTime.cpp: Added.
(WTF::MonotonicTime::now):
(WTF::MonotonicTime::approximateWallTime):
(WTF::MonotonicTime::dump):
(WTF::MonotonicTime::sleep):
* wtf/MonotonicTime.h: Added.
(WTF::MonotonicTime::MonotonicTime):
(WTF::MonotonicTime::fromRawDouble):
(WTF::MonotonicTime::infinity):
(WTF::MonotonicTime::secondsSinceEpoch):
(WTF::MonotonicTime::approximateMonotonicTime):
(WTF::MonotonicTime::operator bool):
(WTF::MonotonicTime::operator+):
(WTF::MonotonicTime::operator-):
(WTF::MonotonicTime::operator+=):
(WTF::MonotonicTime::operator-=):
(WTF::MonotonicTime::operator==):
(WTF::MonotonicTime::operator!=):
(WTF::MonotonicTime::operator<):
(WTF::MonotonicTime::operator>):
(WTF::MonotonicTime::operator<=):
(WTF::MonotonicTime::operator>=):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkCount):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
* wtf/Seconds.cpp: Added.
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::dump):
(WTF::Seconds::sleep):
* wtf/Seconds.h: Added.
(WTF::Seconds::Seconds):
(WTF::Seconds::value):
(WTF::Seconds::seconds):
(WTF::Seconds::milliseconds):
(WTF::Seconds::microseconds):
(WTF::Seconds::nanoseconds):
(WTF::Seconds::fromMilliseconds):
(WTF::Seconds::fromMicroseconds):
(WTF::Seconds::fromNanoseconds):
(WTF::Seconds::infinity):
(WTF::Seconds::operator bool):
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::operator*):
(WTF::Seconds::operator/):
(WTF::Seconds::operator+=):
(WTF::Seconds::operator-=):
(WTF::Seconds::operator*=):
(WTF::Seconds::operator/=):
(WTF::Seconds::operator==):
(WTF::Seconds::operator!=):
(WTF::Seconds::operator<):
(WTF::Seconds::operator>):
(WTF::Seconds::operator<=):
(WTF::Seconds::operator>=):
* wtf/TimeWithDynamicClockType.cpp: Added.
(WTF::TimeWithDynamicClockType::now):
(WTF::TimeWithDynamicClockType::nowWithSameClock):
(WTF::TimeWithDynamicClockType::wallTime):
(WTF::TimeWithDynamicClockType::monotonicTime):
(WTF::TimeWithDynamicClockType::approximateWallTime):
(WTF::TimeWithDynamicClockType::approximateMonotonicTime):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator<):
(WTF::TimeWithDynamicClockType::operator>):
(WTF::TimeWithDynamicClockType::operator<=):
(WTF::TimeWithDynamicClockType::operator>=):
(WTF::TimeWithDynamicClockType::dump):
(WTF::TimeWithDynamicClockType::sleep):
* wtf/TimeWithDynamicClockType.h: Added.
(WTF::TimeWithDynamicClockType::TimeWithDynamicClockType):
(WTF::TimeWithDynamicClockType::fromRawDouble):
(WTF::TimeWithDynamicClockType::secondsSinceEpoch):
(WTF::TimeWithDynamicClockType::clockType):
(WTF::TimeWithDynamicClockType::withSameClockAndRawDouble):
(WTF::TimeWithDynamicClockType::operator bool):
(WTF::TimeWithDynamicClockType::operator+):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator+=):
(WTF::TimeWithDynamicClockType::operator-=):
(WTF::TimeWithDynamicClockType::operator==):
(WTF::TimeWithDynamicClockType::operator!=):
* wtf/WallTime.cpp: Added.
(WTF::WallTime::now):
(WTF::WallTime::approximateMonotonicTime):
(WTF::WallTime::dump):
(WTF::WallTime::sleep):
* wtf/WallTime.h: Added.
(WTF::WallTime::WallTime):
(WTF::WallTime::fromRawDouble):
(WTF::WallTime::infinity):
(WTF::WallTime::secondsSinceEpoch):
(WTF::WallTime::approximateWallTime):
(WTF::WallTime::operator bool):
(WTF::WallTime::operator+):
(WTF::WallTime::operator-):
(WTF::WallTime::operator+=):
(WTF::WallTime::operator-=):
(WTF::WallTime::operator==):
(WTF::WallTime::operator!=):
(WTF::WallTime::operator<):
(WTF::WallTime::operator>):
(WTF::WallTime::operator<=):
(WTF::WallTime::operator>=):
* wtf/threads/BinarySemaphore.cpp:
(WTF::BinarySemaphore::wait):
* wtf/threads/BinarySemaphore.h:
Tools:
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/Condition.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp:
(TestWebKitAPI::ToUpperConverter::stopProducing):
(TestWebKitAPI::ToUpperConverter::stopConsuming):
* TestWebKitAPI/Tests/WTF/Time.cpp: Added.
(WTF::operator<<):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/182152@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
|
|
|
const TimeWithDynamicClockType& timeout)
|
2016-04-20 04:25:02 +00:00
|
|
|
{
|
|
|
|
return parkConditionallyImpl(
|
|
|
|
address,
|
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
https://bugs.webkit.org/show_bug.cgi?id=160125
Reviewed by Geoffrey Garen and Keith Miller.
JSTests:
Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.
* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.
Source/bmalloc:
I needed to tryMemalign, so I added such a thing.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):
Source/JavaScriptCore:
In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.
Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.
At a high level the mechanical changes are:
- We use AuxiliaryBarrier instead of CopyBarrier.
- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
CheckedBoolean stuff, since it's so much more trouble than it's worth.
- The JITs have to emit inlined marked space allocations instead of inline copy space
allocations.
- Everyone has to get used to zeroing their butterflies after allocation instead of relying
on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
space doesn't.
That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:
- MarkedSpace always causes internal fragmentation for array allocations because the vector
length we choose when we resize usually leads to a cell size that doesn't correspond to any
size class. I got around this by making array allocations usually round up vectorLength to
the maximum allowed by the size class that we would have allocated in. Also,
ensureLengthSlow() and friends first make sure that the requested length can't just be
fulfilled with the current allocation size. This safeguard means that not every array
allocation has to do size class queries. For example, the fast path of new Array(length)
never does any size class queries, under the assumption that (1) the speed gained from
avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
size class query, is too small to offset the speed lost by doing the query on every
allocation and (2) new Array(length) is a pretty good hint that resizing is not very
likely.
- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
changes MarkedSpace size classes to use a linear progression for very small sizes followed
by a geometric progression that naturally transitions to a hyperbolic progression. We want
hyperbolic sizes when we get close to blockSize: for example the largest size we want is
payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
next size down should be payloadSize / 3 rounded down, and so on. After the last precise
size (80 bytes), we proceed using a geometric progression, but round up each size to
minimize slop at the end of the block. This naturally causes the geometric progression to
turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
- Large allocations caused massive internal fragmentation, since the smallest large
allocation had to use exactly blockSize, and the largest small allocation used
blockSize / 2. The next size up - the first large allocation size to require two blocks -
also had 50% internal fragmentation. This is because we required large allocations to be
blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
out great since (1) typed arrays won't use large allocations anyway since they have their
own malloc fallback and (2) large array butterflies already have a 8 byte header, which
means that the 8 byte base misalignment aligns the large array payload on a 16 byte
boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
rare as possible; for example, ExecState::vm() skips the check because we know that callees
must be small allocations. It's also possible to use template tricks to do one check for
cell container kind, and then invoke a function specialized for MarkedBlock or a function
specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
that get used from functions that are template-specialized like this. That's mostly to
speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
cutoff for large allocations is runtime-configurable, so long as you don't choose something
so small that callees end up large. I found that 400 bytes is roughly optimal. This means
that the MarkedBlock size classes end up being:
16, 32, 48, 64, 80, 112, 160, 224, 320
The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.
- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
MarkedBlocks because the list that contains the interesting meta-data is allocated on the
side, for better locality during a sequential walk. It's no longer necessary to scan
MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
bits because we now use versioned mark bits: to clear then, just increment the 64-bit
heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
together, these optimizations allowed me to reduce block size to 16KB without losing much
performance. There is some small perf loss on JetStream/splay, but not enough to hurt
JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
progression on membuster. That's not possible yet, since there is still enough per-block
overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
- Even after all of that, copying butterflies was still faster because it allowed us to skip
sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
larger than O(live), especially in an eden collection. Copying satisfies this premise while
mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
that we walk the block to build a free list. This means walking dead space. The new
allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
is completely empty, which is the common case because of the generational hypothesis: the
number of objects that survive an eden collection is a tiny fraction of the number of
objects that had been allocated, and this fraction is so small that there are typically
fewer than one survivors per MarkedBlock. This change was enough to make this change a net
win over tip-of-tree.
- FTL now shares the same allocation fast paths as everything else, which is great, because
bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
control flow, since it won't be able to improve the machine code we write ourselves. GC
fast paths are best written in assembly. So, I've empowered B3 to have even better support
for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
ability to constant-fold some allocator logic (in case we first assume that we're doing a
variable-length allocation but then realize that the length is known). I think it makes
sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
since this makes lowering easier (you can constant fold during lowering more easily) and it
reduces the amount of malloc traffic. In the future, we could teach B3 how to better
constant-fold this code. That would require allowing loads to be constant-folded, which is
doable but hella tricky.
- It used to be that if a logical object allocation required two physical allocations (first
the butterfly and then the cell), then the JIT would emit the code in such a way that a
failure in the second fast path would cause us to forget the successful first physical
allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
register to storing either the butterfly or null, because the butterfly register is anyway
going to be free inside the first allocation. The only overhead here is zeroing the
butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
path, which can then either allocate a butterfly or not. So now we never waste a successful
allocation. This patch implements such a solution both in DFG (where it's easy to do this
since we control registers already) and in FTL (where it's annoying, because mutable
"butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
one version of this patch, this wastage would resonate with some Kraken benchmark: the
benchmark would always allocate N small things followed by one bigger thing. The problem
was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
the JSObject size class, which both the small and big thing shared for their cell, could
hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
it allocated the big thing. So, it would end up having to allocate the big thing's large
butterfly twice, every single time! Ouch!
- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
of the allocator to initialize the object to whatever state they need. It used to be that
we would just use a dumb loop. I initially changed this so that we would end up in memset
for large allocations, but this didn't actually help performance that much. I got a much
better result by playing with different memsets written in assembly. First I wrote one
using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
"rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
of stores (if the size is known), but those paths in the JIT won't trigger for
NewArrayWithSize() if the size is large, since that takes us to the
operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
optimizations here are all in JSArray::create() - that's the hot place for large arrays
that need to be filled with holes.
All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.
We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.
This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):
Source/WebCore:
No new tests because no new WebCore behavior.
Just rewiring #includes.
* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:
Source/WebKit2:
Just rewiring some #includes.
* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:
Source/WTF:
I needed tryFastAlignedMalloc() so I added it.
* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.
Tools:
* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/179778@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
|
|
|
scopedLambdaRef<bool()>(validation),
|
|
|
|
scopedLambdaRef<void()>(beforeSleep),
|
2016-04-20 04:25:02 +00:00
|
|
|
timeout);
|
|
|
|
}
|
2015-08-11 19:51:35 +00:00
|
|
|
|
2015-08-13 20:42:11 +00:00
|
|
|
// Simple version of parkConditionally() that covers the most common case: you want to park
|
|
|
|
// indefinitely so long as the value at the given address hasn't changed.
|
2015-08-11 19:51:35 +00:00
|
|
|
template<typename T, typename U>
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
static ParkResult compareAndPark(const Atomic<T>* address, U expected)
|
2015-08-11 19:51:35 +00:00
|
|
|
{
|
|
|
|
return parkConditionally(
|
|
|
|
address,
|
|
|
|
[address, expected] () -> bool {
|
|
|
|
U value = address->load();
|
|
|
|
return value == expected;
|
2015-08-13 20:42:11 +00:00
|
|
|
},
|
|
|
|
[] () { },
|
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows
https://bugs.webkit.org/show_bug.cgi?id=152045
Reviewed by Andy Estes.
Source/JavaScriptCore:
Probably the nicest example of why this patch is a good idea is the change in
AtomicsObject.cpp.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
* runtime/AtomicsObject.cpp:
(JSC::atomicsFuncWait):
Source/WebCore:
No new layout tests because no new behavior. The new WTF time classes have some unit tests
in TestWebKitAPI.
* fileapi/ThreadableBlobRegistry.cpp:
(WebCore::ThreadableBlobRegistry::blobSize):
* platform/MainThreadSharedTimer.h:
* platform/SharedTimer.h:
* platform/ThreadTimers.cpp:
(WebCore::ThreadTimers::updateSharedTimer):
* platform/cf/MainThreadSharedTimerCF.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/efl/MainThreadSharedTimerEfl.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/glib/MainThreadSharedTimerGLib.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/win/MainThreadSharedTimerWin.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* workers/WorkerRunLoop.cpp:
(WebCore::WorkerRunLoop::runInMode):
Source/WebKit2:
* Platform/IPC/Connection.cpp:
(IPC::Connection::SyncMessageState::wait):
(IPC::Connection::sendMessage):
(IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting):
(IPC::Connection::waitForMessage):
(IPC::Connection::sendSyncMessage):
(IPC::Connection::waitForSyncReply):
* Platform/IPC/Connection.h:
(IPC::Connection::sendSync):
(IPC::Connection::waitForAndDispatchImmediately):
* Platform/IPC/MessageSender.h:
(IPC::MessageSender::sendSync):
* UIProcess/ChildProcessProxy.h:
(WebKit::ChildProcessProxy::sendSync):
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/Storage/StorageManager.cpp:
(WebKit::StorageManager::applicationWillTerminate):
* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/WebResourceLoadStatisticsStore.cpp:
(WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate):
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h:
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm:
(-[WKOneShotDisplayLinkHandler displayLinkFired:]):
(WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree):
(WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay):
(WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm:
(WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/WKImmediateActionController.mm:
(-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]):
* UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::stringSelectionForPasteboard):
(WebKit::WebPageProxy::dataSelectionForPasteboard):
(WebKit::WebPageProxy::readSelectionFromPasteboard):
(WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent):
(WebKit::WebPageProxy::acceptsFirstMouse):
* WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::runBeforeUnloadConfirmPanel):
(WebKit::WebChromeClient::runJavaScriptAlert):
(WebKit::WebChromeClient::runJavaScriptConfirm):
(WebKit::WebChromeClient::runJavaScriptPrompt):
(WebKit::WebChromeClient::print):
(WebKit::WebChromeClient::exceededDatabaseQuota):
(WebKit::WebChromeClient::reachedApplicationCacheOriginQuota):
* WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp:
(WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse):
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::postSynchronousMessageForTesting):
Source/WTF:
We used to use 'double' for all time measurements. Sometimes it was milliseconds,
sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time
since some epoch. When we spoke of time since epoch, we either used a monotonic clock or
a wall clock. The type - always 'double' - never told us what kind of time we had, even
though there were roughly six of them (sec interval, ms interval, sec since epoch on wall,
ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic).
At some point, we thought that it would be a good idea to replace these doubles with
std::chrono. But since replacing some things with std::chrono, we found it to be terribly
inconvenient:
- Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say
std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally
duration_cast would not even be a thing.
- No overflow protection. std::chrono uses integers by default and using anything else is
clumsy. But the integer math is done without regard for the rough edges of integer math,
so any cast between std::chrono types risks overflow. Any comparison risks overflow
because it may do conversions silently. We have even found bugs where some C++
implementations had more overflows than others, which ends up being a special kind of
hell. In many cases, the overflow also has nasal demons.
It's an error to represent time using integers. It would have been excusable back when
floating point math was not guaranteed to be supported on all platforms, but that would
have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit
for floating point:
- Floating point preserves precision under multiplication in all but extreme cases, so
using floating point for time means that unit conversions are almost completely
lossless. This means that we don't have to think very hard about what units to use. In
this patch, we use seconds almost everywhere. We only convert at boundaries, like an API
boundary that wants something other than seconds.
- Floating point makes it easy to reason about infinity, which is something that time code
wants to do a lot. Example: when would you like to timeout? Infinity please! This is the
most elegant way of having an API support both a timeout variant and a no-timeout
variant.
- Floating point does well-understood things when math goes wrong, and these things are
pretty well optimized to match what a mathematician would do when computing with real
numbers represented using scientific notation with a finite number of significant
digits. This means that time math under floating point looks like normal math. On the
other hand, std::chrono time math looks like garbage because you have to always check
for multiple possible UB corners whenever you touch large integers. Integers that
represent time are very likely to be large and you don't have to do much to overflow
them. At this time, based on the number of bugs we have already seen due to chrono
overflows, I am not certain that we even understand what are all of the corner cases
that we should even check for.
This patch introduces a new set of timekeeping classes that are all based on double, and
all internally use seconds. These classes support algebraic typing. The classes are:
- Seconds: this is for measuring a duration.
- WallTime: time since epoch according to a wall clock (aka real time clock).
- MonotonicTime: time since epoch according to a monotonic clock.
- ClockType: enum that says either Wall or Monotonic.
- TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a
wall time or a monotonic time.
All of these classes behave like C++ values and are cheap to copy around since they are
very nearly POD. This supports comprehensive conversions between the various time types.
Most of this is by way of algebra. Here are just some of the rules we recognize:
WallTime = WallTime + Seconds
Seconds = WallTime - WallTime
MonotonicTime = MonotonicTime + Seconds
etc...
We support negative, infinite, and NaN times because math.
We support conversions between MonotonicTime and WallTime, like:
WallTime wt = mt.approximateWallTime()
This is called this "approximate" because the only way to do it is to get the current time
on both clocks and convert relative to that.
Many of our APIs would be happy using whatever notion of time the user wanted to use. For
those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You
can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This
means that if you use a WallTime with Condition::waitUntil, then Condition's internal
logic for when it should wake up makes its decision based on the current WallTime - but if
you use MonotonicTime then waitUntil will make its decision based on current
MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono
did not have the concept of a dynamic clock type.
This patch does not include conversions between std::chrono and these new time classes,
because past experience shows that we're quite bad at getting conversions between
std::chrono and anything else right. Also, I didn't need such conversion code because this
patch only converts code that transitively touches ParkingLot and Condition. It was easy
to get all of that code onto the new time classes.
* WTF.xcodeproj/project.pbxproj:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/ClockType.cpp: Added.
(WTF::printInternal):
* wtf/ClockType.h: Added.
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::waitFor):
(WTF::ConditionBase::wait):
(WTF::ConditionBase::waitUntilWallClockSeconds): Deleted.
(WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted.
(WTF::ConditionBase::waitForSeconds): Deleted.
(WTF::ConditionBase::waitForSecondsImpl): Deleted.
(WTF::ConditionBase::waitForImpl): Deleted.
(WTF::ConditionBase::absoluteFromRelative): Deleted.
* wtf/CrossThreadQueue.h:
(WTF::CrossThreadQueue<DataType>::waitForMessage):
* wtf/CurrentTime.cpp:
(WTF::sleep):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime): Deleted.
* wtf/MonotonicTime.cpp: Added.
(WTF::MonotonicTime::now):
(WTF::MonotonicTime::approximateWallTime):
(WTF::MonotonicTime::dump):
(WTF::MonotonicTime::sleep):
* wtf/MonotonicTime.h: Added.
(WTF::MonotonicTime::MonotonicTime):
(WTF::MonotonicTime::fromRawDouble):
(WTF::MonotonicTime::infinity):
(WTF::MonotonicTime::secondsSinceEpoch):
(WTF::MonotonicTime::approximateMonotonicTime):
(WTF::MonotonicTime::operator bool):
(WTF::MonotonicTime::operator+):
(WTF::MonotonicTime::operator-):
(WTF::MonotonicTime::operator+=):
(WTF::MonotonicTime::operator-=):
(WTF::MonotonicTime::operator==):
(WTF::MonotonicTime::operator!=):
(WTF::MonotonicTime::operator<):
(WTF::MonotonicTime::operator>):
(WTF::MonotonicTime::operator<=):
(WTF::MonotonicTime::operator>=):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkCount):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
* wtf/Seconds.cpp: Added.
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::dump):
(WTF::Seconds::sleep):
* wtf/Seconds.h: Added.
(WTF::Seconds::Seconds):
(WTF::Seconds::value):
(WTF::Seconds::seconds):
(WTF::Seconds::milliseconds):
(WTF::Seconds::microseconds):
(WTF::Seconds::nanoseconds):
(WTF::Seconds::fromMilliseconds):
(WTF::Seconds::fromMicroseconds):
(WTF::Seconds::fromNanoseconds):
(WTF::Seconds::infinity):
(WTF::Seconds::operator bool):
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::operator*):
(WTF::Seconds::operator/):
(WTF::Seconds::operator+=):
(WTF::Seconds::operator-=):
(WTF::Seconds::operator*=):
(WTF::Seconds::operator/=):
(WTF::Seconds::operator==):
(WTF::Seconds::operator!=):
(WTF::Seconds::operator<):
(WTF::Seconds::operator>):
(WTF::Seconds::operator<=):
(WTF::Seconds::operator>=):
* wtf/TimeWithDynamicClockType.cpp: Added.
(WTF::TimeWithDynamicClockType::now):
(WTF::TimeWithDynamicClockType::nowWithSameClock):
(WTF::TimeWithDynamicClockType::wallTime):
(WTF::TimeWithDynamicClockType::monotonicTime):
(WTF::TimeWithDynamicClockType::approximateWallTime):
(WTF::TimeWithDynamicClockType::approximateMonotonicTime):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator<):
(WTF::TimeWithDynamicClockType::operator>):
(WTF::TimeWithDynamicClockType::operator<=):
(WTF::TimeWithDynamicClockType::operator>=):
(WTF::TimeWithDynamicClockType::dump):
(WTF::TimeWithDynamicClockType::sleep):
* wtf/TimeWithDynamicClockType.h: Added.
(WTF::TimeWithDynamicClockType::TimeWithDynamicClockType):
(WTF::TimeWithDynamicClockType::fromRawDouble):
(WTF::TimeWithDynamicClockType::secondsSinceEpoch):
(WTF::TimeWithDynamicClockType::clockType):
(WTF::TimeWithDynamicClockType::withSameClockAndRawDouble):
(WTF::TimeWithDynamicClockType::operator bool):
(WTF::TimeWithDynamicClockType::operator+):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator+=):
(WTF::TimeWithDynamicClockType::operator-=):
(WTF::TimeWithDynamicClockType::operator==):
(WTF::TimeWithDynamicClockType::operator!=):
* wtf/WallTime.cpp: Added.
(WTF::WallTime::now):
(WTF::WallTime::approximateMonotonicTime):
(WTF::WallTime::dump):
(WTF::WallTime::sleep):
* wtf/WallTime.h: Added.
(WTF::WallTime::WallTime):
(WTF::WallTime::fromRawDouble):
(WTF::WallTime::infinity):
(WTF::WallTime::secondsSinceEpoch):
(WTF::WallTime::approximateWallTime):
(WTF::WallTime::operator bool):
(WTF::WallTime::operator+):
(WTF::WallTime::operator-):
(WTF::WallTime::operator+=):
(WTF::WallTime::operator-=):
(WTF::WallTime::operator==):
(WTF::WallTime::operator!=):
(WTF::WallTime::operator<):
(WTF::WallTime::operator>):
(WTF::WallTime::operator<=):
(WTF::WallTime::operator>=):
* wtf/threads/BinarySemaphore.cpp:
(WTF::BinarySemaphore::wait):
* wtf/threads/BinarySemaphore.h:
Tools:
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/Condition.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp:
(TestWebKitAPI::ToUpperConverter::stopProducing):
(TestWebKitAPI::ToUpperConverter::stopConsuming):
* TestWebKitAPI/Tests/WTF/Time.cpp: Added.
(WTF::operator<<):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/182152@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
|
|
|
Time::infinity());
|
2015-08-11 19:51:35 +00:00
|
|
|
}
|
|
|
|
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// Unparking status given to you anytime you unparkOne().
|
2016-04-20 04:25:02 +00:00
|
|
|
struct UnparkResult {
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// True if some thread was unparked.
|
2016-04-20 04:25:02 +00:00
|
|
|
bool didUnparkThread { false };
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// True if there may be more threads on this address. This may be conservatively true.
|
2016-04-20 04:25:02 +00:00
|
|
|
bool mayHaveMoreThreads { false };
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// This bit is randomly set to true indicating that it may be profitable to unlock the lock
|
|
|
|
// using a fair unlocking protocol. This is most useful when used in conjunction with
|
|
|
|
// unparkOne(address, callback).
|
|
|
|
bool timeToBeFair { false };
|
2016-04-20 04:25:02 +00:00
|
|
|
};
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
|
|
|
|
// Unparks one thread from the queue associated with the given address, which cannot be null.
|
|
|
|
// Returns true if there may still be other threads on that queue, or false if there definitely
|
|
|
|
// are no more threads on the queue.
|
2016-04-20 04:25:02 +00:00
|
|
|
WTF_EXPORT_PRIVATE static UnparkResult unparkOne(const void* address);
|
2015-08-11 19:51:35 +00:00
|
|
|
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// This is an expert-mode version of unparkOne() that allows for really good thundering herd
|
|
|
|
// avoidance and eventual stochastic fairness in adaptive mutexes.
|
|
|
|
//
|
WTF::Lock should not suffer from the thundering herd
https://bugs.webkit.org/show_bug.cgi?id=147947
Reviewed by Geoffrey Garen.
Source/WTF:
This changes Lock::unlockSlow() to use unparkOne() instead of unparkAll(). The problem with
doing this is that it's not obvious after calling unparkOne() if there are any other threads
that are still parked on the lock's queue. If we assume that there are and leave the
hasParkedBit set, then future calls to unlock() will take the slow path. We don't want that
if there aren't actually any threads parked. On the other hand, if we assume that there
aren't any threads parked and clear the hasParkedBit, then if there actually were some
threads parked, then they may never be awoken since future calls to unlock() won't take slow
path and so won't call unparkOne(). In other words, we need a way to be very precise about
when we clear the hasParkedBit and we need to do it in a race-free way: it can't be the case
that we clear the bit just as some thread gets parked on the queue.
A similar problem arises in futexes, and one of the solutions is to have a thread that
acquires a lock after parking sets the hasParkedBit. This is what Rusty Russel's usersem
does. It's a subtle algorithm. Also, it means that if a thread barges in before the unparked
thread runs, then that barging thread will not know that there are threads parked. This
could increase the severity of barging.
Since ParkingLot is a user-level API, we don't have to worry about the kernel-user security
issues and so we can expose callbacks while ParkingLot is holding its internal locks. This
change does exactly that for unparkOne(). The new variant of unparkOne() will call a user
function while the queue from which we are unparking is locked. The callback is told basic
stats about the queue: did we unpark a thread this time, and could there be more threads to
unpark in the future. The callback runs while it's impossible for the queue state to change,
since the ParkingLot's internal locks for the queue is held. This means that
Lock::unlockSlow() can either clear, or leave, the hasParkedBit while releasing the lock
inside the callback from unparkOne(). This takes care of the thundering herd problem while
also reducing the greed that arises from barging threads.
This required some careful reworking of the ParkingLot algorithm. The first thing I noticed
was that the ThreadData::shouldPark flag was useless, since it's set exactly when
ThreadData::address is non-null. Then I had to make sure that dequeue() could lazily create
both hashtables and buckets, since the "callback is called while queue is locked" invariant
requires that we didn't exit early due to the hashtable or bucket not being present. Note
that all of this is done in such a way that the old unparkOne() and unparkAll() don't have
to create any buckets, though they now may create the hashtable. We don't care as much about
the hashtable being created by unpark since it's just such an unlikely scenario and it would
only happen once.
This change reduces the kernel CPU usage of WTF::Lock for the long critical section test by
about 8x and makes it always perform as well as WTF::WordLock and WTF::Mutex for that
benchmark.
* benchmarks/LockSpeedTest.cpp:
* wtf/Lock.cpp:
(WTF::LockBase::unlockSlow):
* wtf/Lock.h:
(WTF::LockBase::isLocked):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
* wtf/WordLock.h:
(WTF::WordLock::isLocked):
(WTF::WordLock::isFullyReset):
Tools:
Add testing that checks that locks return to a pristine state after contention is over.
* TestWebKitAPI/Tests/WTF/Lock.cpp:
(TestWebKitAPI::LockInspector::isFullyReset):
(TestWebKitAPI::runLockTest):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/166072@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188374 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 03:51:25 +00:00
|
|
|
// Unparks one thread from the queue associated with the given address, and calls the given
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
// callback while the address is locked. Reports to the callback whether any thread got
|
|
|
|
// unparked, whether there may be any other threads still on the queue, and whether this may be
|
|
|
|
// a good time to do fair unlocking. The callback returns an intptr_t token, which is returned
|
|
|
|
// to the unparked thread via ParkResult::token.
|
|
|
|
//
|
|
|
|
// WTF::Lock and WTF::Condition both use this form of unparkOne() because it allows them to use
|
|
|
|
// the ParkingLot's internal queue lock to serialize some decision-making. For example, if
|
|
|
|
// UnparkResult::mayHaveMoreThreads is false inside the callback, then we know that at that
|
|
|
|
// moment nobody can add any threads to the queue because the queue lock is still held. Also,
|
|
|
|
// WTF::Lock uses the timeToBeFair and token mechanism to implement eventual fairness.
|
2016-05-26 21:58:42 +00:00
|
|
|
template<typename Callback>
|
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
https://bugs.webkit.org/show_bug.cgi?id=160125
Reviewed by Geoffrey Garen and Keith Miller.
JSTests:
Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.
* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.
Source/bmalloc:
I needed to tryMemalign, so I added such a thing.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):
Source/JavaScriptCore:
In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.
Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.
At a high level the mechanical changes are:
- We use AuxiliaryBarrier instead of CopyBarrier.
- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
CheckedBoolean stuff, since it's so much more trouble than it's worth.
- The JITs have to emit inlined marked space allocations instead of inline copy space
allocations.
- Everyone has to get used to zeroing their butterflies after allocation instead of relying
on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
space doesn't.
That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:
- MarkedSpace always causes internal fragmentation for array allocations because the vector
length we choose when we resize usually leads to a cell size that doesn't correspond to any
size class. I got around this by making array allocations usually round up vectorLength to
the maximum allowed by the size class that we would have allocated in. Also,
ensureLengthSlow() and friends first make sure that the requested length can't just be
fulfilled with the current allocation size. This safeguard means that not every array
allocation has to do size class queries. For example, the fast path of new Array(length)
never does any size class queries, under the assumption that (1) the speed gained from
avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
size class query, is too small to offset the speed lost by doing the query on every
allocation and (2) new Array(length) is a pretty good hint that resizing is not very
likely.
- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
changes MarkedSpace size classes to use a linear progression for very small sizes followed
by a geometric progression that naturally transitions to a hyperbolic progression. We want
hyperbolic sizes when we get close to blockSize: for example the largest size we want is
payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
next size down should be payloadSize / 3 rounded down, and so on. After the last precise
size (80 bytes), we proceed using a geometric progression, but round up each size to
minimize slop at the end of the block. This naturally causes the geometric progression to
turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
- Large allocations caused massive internal fragmentation, since the smallest large
allocation had to use exactly blockSize, and the largest small allocation used
blockSize / 2. The next size up - the first large allocation size to require two blocks -
also had 50% internal fragmentation. This is because we required large allocations to be
blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
out great since (1) typed arrays won't use large allocations anyway since they have their
own malloc fallback and (2) large array butterflies already have a 8 byte header, which
means that the 8 byte base misalignment aligns the large array payload on a 16 byte
boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
rare as possible; for example, ExecState::vm() skips the check because we know that callees
must be small allocations. It's also possible to use template tricks to do one check for
cell container kind, and then invoke a function specialized for MarkedBlock or a function
specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
that get used from functions that are template-specialized like this. That's mostly to
speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
cutoff for large allocations is runtime-configurable, so long as you don't choose something
so small that callees end up large. I found that 400 bytes is roughly optimal. This means
that the MarkedBlock size classes end up being:
16, 32, 48, 64, 80, 112, 160, 224, 320
The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.
- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
MarkedBlocks because the list that contains the interesting meta-data is allocated on the
side, for better locality during a sequential walk. It's no longer necessary to scan
MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
bits because we now use versioned mark bits: to clear then, just increment the 64-bit
heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
together, these optimizations allowed me to reduce block size to 16KB without losing much
performance. There is some small perf loss on JetStream/splay, but not enough to hurt
JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
progression on membuster. That's not possible yet, since there is still enough per-block
overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
- Even after all of that, copying butterflies was still faster because it allowed us to skip
sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
larger than O(live), especially in an eden collection. Copying satisfies this premise while
mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
that we walk the block to build a free list. This means walking dead space. The new
allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
is completely empty, which is the common case because of the generational hypothesis: the
number of objects that survive an eden collection is a tiny fraction of the number of
objects that had been allocated, and this fraction is so small that there are typically
fewer than one survivors per MarkedBlock. This change was enough to make this change a net
win over tip-of-tree.
- FTL now shares the same allocation fast paths as everything else, which is great, because
bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
control flow, since it won't be able to improve the machine code we write ourselves. GC
fast paths are best written in assembly. So, I've empowered B3 to have even better support
for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
ability to constant-fold some allocator logic (in case we first assume that we're doing a
variable-length allocation but then realize that the length is known). I think it makes
sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
since this makes lowering easier (you can constant fold during lowering more easily) and it
reduces the amount of malloc traffic. In the future, we could teach B3 how to better
constant-fold this code. That would require allowing loads to be constant-folded, which is
doable but hella tricky.
- It used to be that if a logical object allocation required two physical allocations (first
the butterfly and then the cell), then the JIT would emit the code in such a way that a
failure in the second fast path would cause us to forget the successful first physical
allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
register to storing either the butterfly or null, because the butterfly register is anyway
going to be free inside the first allocation. The only overhead here is zeroing the
butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
path, which can then either allocate a butterfly or not. So now we never waste a successful
allocation. This patch implements such a solution both in DFG (where it's easy to do this
since we control registers already) and in FTL (where it's annoying, because mutable
"butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
one version of this patch, this wastage would resonate with some Kraken benchmark: the
benchmark would always allocate N small things followed by one bigger thing. The problem
was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
the JSObject size class, which both the small and big thing shared for their cell, could
hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
it allocated the big thing. So, it would end up having to allocate the big thing's large
butterfly twice, every single time! Ouch!
- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
of the allocator to initialize the object to whatever state they need. It used to be that
we would just use a dumb loop. I initially changed this so that we would end up in memset
for large allocations, but this didn't actually help performance that much. I got a much
better result by playing with different memsets written in assembly. First I wrote one
using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
"rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
of stores (if the size is known), but those paths in the JIT won't trigger for
NewArrayWithSize() if the size is large, since that takes us to the
operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
optimizations here are all in JSArray::create() - that's the hot place for large arrays
that need to be filled with holes.
All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.
We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.
This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):
Source/WebCore:
No new tests because no new WebCore behavior.
Just rewiring #includes.
* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:
Source/WebKit2:
Just rewiring some #includes.
* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:
Source/WTF:
I needed tryFastAlignedMalloc() so I added it.
* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.
Tools:
* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/179778@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
|
|
|
static void unparkOne(const void* address, const Callback& callback)
|
2016-05-26 21:58:42 +00:00
|
|
|
{
|
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
https://bugs.webkit.org/show_bug.cgi?id=160125
Reviewed by Geoffrey Garen and Keith Miller.
JSTests:
Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.
* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.
Source/bmalloc:
I needed to tryMemalign, so I added such a thing.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):
Source/JavaScriptCore:
In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.
Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.
At a high level the mechanical changes are:
- We use AuxiliaryBarrier instead of CopyBarrier.
- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
CheckedBoolean stuff, since it's so much more trouble than it's worth.
- The JITs have to emit inlined marked space allocations instead of inline copy space
allocations.
- Everyone has to get used to zeroing their butterflies after allocation instead of relying
on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
space doesn't.
That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:
- MarkedSpace always causes internal fragmentation for array allocations because the vector
length we choose when we resize usually leads to a cell size that doesn't correspond to any
size class. I got around this by making array allocations usually round up vectorLength to
the maximum allowed by the size class that we would have allocated in. Also,
ensureLengthSlow() and friends first make sure that the requested length can't just be
fulfilled with the current allocation size. This safeguard means that not every array
allocation has to do size class queries. For example, the fast path of new Array(length)
never does any size class queries, under the assumption that (1) the speed gained from
avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
size class query, is too small to offset the speed lost by doing the query on every
allocation and (2) new Array(length) is a pretty good hint that resizing is not very
likely.
- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
changes MarkedSpace size classes to use a linear progression for very small sizes followed
by a geometric progression that naturally transitions to a hyperbolic progression. We want
hyperbolic sizes when we get close to blockSize: for example the largest size we want is
payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
next size down should be payloadSize / 3 rounded down, and so on. After the last precise
size (80 bytes), we proceed using a geometric progression, but round up each size to
minimize slop at the end of the block. This naturally causes the geometric progression to
turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
- Large allocations caused massive internal fragmentation, since the smallest large
allocation had to use exactly blockSize, and the largest small allocation used
blockSize / 2. The next size up - the first large allocation size to require two blocks -
also had 50% internal fragmentation. This is because we required large allocations to be
blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
out great since (1) typed arrays won't use large allocations anyway since they have their
own malloc fallback and (2) large array butterflies already have a 8 byte header, which
means that the 8 byte base misalignment aligns the large array payload on a 16 byte
boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
rare as possible; for example, ExecState::vm() skips the check because we know that callees
must be small allocations. It's also possible to use template tricks to do one check for
cell container kind, and then invoke a function specialized for MarkedBlock or a function
specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
that get used from functions that are template-specialized like this. That's mostly to
speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
cutoff for large allocations is runtime-configurable, so long as you don't choose something
so small that callees end up large. I found that 400 bytes is roughly optimal. This means
that the MarkedBlock size classes end up being:
16, 32, 48, 64, 80, 112, 160, 224, 320
The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.
- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
MarkedBlocks because the list that contains the interesting meta-data is allocated on the
side, for better locality during a sequential walk. It's no longer necessary to scan
MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
bits because we now use versioned mark bits: to clear then, just increment the 64-bit
heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
together, these optimizations allowed me to reduce block size to 16KB without losing much
performance. There is some small perf loss on JetStream/splay, but not enough to hurt
JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
progression on membuster. That's not possible yet, since there is still enough per-block
overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
- Even after all of that, copying butterflies was still faster because it allowed us to skip
sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
larger than O(live), especially in an eden collection. Copying satisfies this premise while
mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
that we walk the block to build a free list. This means walking dead space. The new
allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
is completely empty, which is the common case because of the generational hypothesis: the
number of objects that survive an eden collection is a tiny fraction of the number of
objects that had been allocated, and this fraction is so small that there are typically
fewer than one survivors per MarkedBlock. This change was enough to make this change a net
win over tip-of-tree.
- FTL now shares the same allocation fast paths as everything else, which is great, because
bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
control flow, since it won't be able to improve the machine code we write ourselves. GC
fast paths are best written in assembly. So, I've empowered B3 to have even better support
for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
ability to constant-fold some allocator logic (in case we first assume that we're doing a
variable-length allocation but then realize that the length is known). I think it makes
sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
since this makes lowering easier (you can constant fold during lowering more easily) and it
reduces the amount of malloc traffic. In the future, we could teach B3 how to better
constant-fold this code. That would require allowing loads to be constant-folded, which is
doable but hella tricky.
- It used to be that if a logical object allocation required two physical allocations (first
the butterfly and then the cell), then the JIT would emit the code in such a way that a
failure in the second fast path would cause us to forget the successful first physical
allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
register to storing either the butterfly or null, because the butterfly register is anyway
going to be free inside the first allocation. The only overhead here is zeroing the
butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
path, which can then either allocate a butterfly or not. So now we never waste a successful
allocation. This patch implements such a solution both in DFG (where it's easy to do this
since we control registers already) and in FTL (where it's annoying, because mutable
"butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
one version of this patch, this wastage would resonate with some Kraken benchmark: the
benchmark would always allocate N small things followed by one bigger thing. The problem
was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
the JSObject size class, which both the small and big thing shared for their cell, could
hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
it allocated the big thing. So, it would end up having to allocate the big thing's large
butterfly twice, every single time! Ouch!
- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
of the allocator to initialize the object to whatever state they need. It used to be that
we would just use a dumb loop. I initially changed this so that we would end up in memset
for large allocations, but this didn't actually help performance that much. I got a much
better result by playing with different memsets written in assembly. First I wrote one
using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
"rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
of stores (if the size is known), but those paths in the JIT won't trigger for
NewArrayWithSize() if the size is large, since that takes us to the
operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
optimizations here are all in JSArray::create() - that's the hot place for large arrays
that need to be filled with holes.
All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.
We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.
This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):
Source/WebCore:
No new tests because no new WebCore behavior.
Just rewiring #includes.
* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:
Source/WebKit2:
Just rewiring some #includes.
* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:
Source/WTF:
I needed tryFastAlignedMalloc() so I added it.
* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.
Tools:
* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/179778@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
|
|
|
unparkOneImpl(address, scopedLambdaRef<intptr_t(UnparkResult)>(callback));
|
2016-05-26 21:58:42 +00:00
|
|
|
}
|
JSC should support SharedArrayBuffer
https://bugs.webkit.org/show_bug.cgi?id=163986
Reviewed by Keith Miller.
JSTests:
This adds our own test for the various corner cases of SharedArrayBuffer. This test is meant to
check all of the things that don't require concurrency.
* stress/SharedArrayBuffer.js: Added.
(checkAtomics):
(shouldFail):
(Symbol):
(runAtomic):
Source/JavaScriptCore:
This implements https://tc39.github.io/ecmascript_sharedmem/shmem.html.
There is now a new SharedArrayBuffer type. In the JS runtime, which includes typed array
types, the SharedArrayBuffer is a drop-in replacement for ArrayBuffer, even though they are
distinct types (new SharedArrayBuffer() instanceof ArrayBuffer == false and vice versa). The
DOM will not recognize SharedArrayBuffer, or any typed array that wraps it, to ensure safety.
This matches what other browsers intend to do, see
https://github.com/tc39/ecmascript_sharedmem/issues/38. API is provided for the DOM to opt
into SharedArrayBuffer. One notable place is postMessage, which will share the
SharedArrayBuffer's underlying data storage with other workers. This creates a pool of shared
memory that the workers can use to talk to each other.
There is also an Atomics object in global scope, which exposes sequentially consistent atomic
operations: add, and, compareExchange, exchange, load, or, store, sub, and xor. Additionally
it exposes a Atomics.isLockFree utility, which takes a byte amount and returns true or false.
Also there is Atomics.wake/wait, which neatly map to ParkingLot.
Accesses to typed arrays that wrap SharedArrayBuffer are optimized by JSC the same way as
always. I believe that DFG and B3 already obey the following memory model, which I believe is
a bit weaker than Cambridge and a bit stronger than what is being proposed for
SharedArrayBuffer. To predict a program's behavior under the B3 memory model, imagine the
space of all possible programs that would result from running an optimizer that adversarially
follows B3's transformation rules. B3 transformations are correct if the newly created
program is equivalent to the old one, assuming that any opaque effect in IR (like the reads
and writes of a patchpoint/call/fence) could perform any load/store that satisfies the
B3::Effects summary. Opaque effects are a way of describing an infinite set of programs: any
program that only does the effects summarized in B3::Effects belongs to the set. For example,
this prevents motion of operations across fences since fences are summarized as opaque
effects that could read or write memory. This rule alone is not enough, because it leaves the
door open for turning an atomic operation (like a load) into a non-atomic one (like a load
followed by a store of the same value back to the same location or multiple loads). This is
not an optimization that either our compiler or the CPU would want to do. One way to think of
what exactly is forbidden is that B3 transformations that mess with memory accesses can only
reorder them or remove them. This means that for any execution of the untransformed program,
the corresponding execution of the transformed program (i.e. with the same input arguments
and the same programs filled in for the opaque effects) must have the same loads and stores,
with some removed and some reordered. This is a fairly simple mental model that B3 and DFG
already follow and it's based on existing abstractions for the infinite set of programs
inside an opaque effect (DFG's AbstractHeaps and B3's Effects).
This patch makes all atomics operations intrinsic, but the DFG doesn't know about any of them
yet. That's covered by bug 164108.
This ought to be perf-neutral, but I am still running tests to confirm this. I'm also still
writing new tests to cover all of the Atomics functionality and the behavior of SAB objects.
* API/JSTypedArray.cpp:
(JSObjectGetTypedArrayBytesPtr):
(JSObjectGetTypedArrayBuffer):
(JSObjectMakeArrayBufferWithBytesNoCopy):
* API/tests/CompareAndSwapTest.cpp:
(Bitmap::concurrentTestAndSet):
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* dfg/DFGDesiredWatchpoints.cpp:
(JSC::DFG::ArrayBufferViewWatchpointAdaptor::add):
* heap/Heap.cpp:
(JSC::Heap::reportExtraMemoryVisited):
(JSC::Heap::reportExternalMemoryVisited):
* jsc.cpp:
(functionTransferArrayBuffer):
* runtime/ArrayBuffer.cpp:
(JSC::SharedArrayBufferContents::SharedArrayBufferContents):
(JSC::SharedArrayBufferContents::~SharedArrayBufferContents):
(JSC::ArrayBufferContents::ArrayBufferContents):
(JSC::ArrayBufferContents::operator=):
(JSC::ArrayBufferContents::~ArrayBufferContents):
(JSC::ArrayBufferContents::clear):
(JSC::ArrayBufferContents::destroy):
(JSC::ArrayBufferContents::reset):
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBufferContents::makeShared):
(JSC::ArrayBufferContents::transferTo):
(JSC::ArrayBufferContents::copyTo):
(JSC::ArrayBufferContents::shareWith):
(JSC::ArrayBuffer::create):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
(JSC::ArrayBuffer::createUninitialized):
(JSC::ArrayBuffer::tryCreateUninitialized):
(JSC::ArrayBuffer::createInternal):
(JSC::ArrayBuffer::ArrayBuffer):
(JSC::ArrayBuffer::slice):
(JSC::ArrayBuffer::sliceImpl):
(JSC::ArrayBuffer::makeShared):
(JSC::ArrayBuffer::setSharingMode):
(JSC::ArrayBuffer::transferTo):
(JSC::ArrayBuffer::transfer): Deleted.
* runtime/ArrayBuffer.h:
(JSC::arrayBufferSharingModeName):
(JSC::SharedArrayBufferContents::data):
(JSC::ArrayBufferContents::data):
(JSC::ArrayBufferContents::sizeInBytes):
(JSC::ArrayBufferContents::isShared):
(JSC::ArrayBuffer::sharingMode):
(JSC::ArrayBuffer::isShared):
(JSC::ArrayBuffer::gcSizeEstimateInBytes):
(JSC::arrayBufferDestructorNull): Deleted.
(JSC::arrayBufferDestructorDefault): Deleted.
(JSC::ArrayBufferContents::ArrayBufferContents): Deleted.
(JSC::ArrayBufferContents::transfer): Deleted.
(JSC::ArrayBufferContents::copyTo): Deleted.
(JSC::ArrayBuffer::create): Deleted.
(JSC::ArrayBuffer::createAdopted): Deleted.
(JSC::ArrayBuffer::createFromBytes): Deleted.
(JSC::ArrayBuffer::tryCreate): Deleted.
(JSC::ArrayBuffer::createUninitialized): Deleted.
(JSC::ArrayBuffer::tryCreateUninitialized): Deleted.
(JSC::ArrayBuffer::createInternal): Deleted.
(JSC::ArrayBuffer::ArrayBuffer): Deleted.
(JSC::ArrayBuffer::slice): Deleted.
(JSC::ArrayBuffer::sliceImpl): Deleted.
(JSC::ArrayBufferContents::tryAllocate): Deleted.
(JSC::ArrayBufferContents::~ArrayBufferContents): Deleted.
* runtime/ArrayBufferSharingMode.h: Added.
* runtime/ArrayBufferView.h:
(JSC::ArrayBufferView::possiblySharedBuffer):
(JSC::ArrayBufferView::unsharedBuffer):
(JSC::ArrayBufferView::isShared):
(JSC::ArrayBufferView::buffer): Deleted.
* runtime/AtomicsObject.cpp: Added.
(JSC::AtomicsObject::AtomicsObject):
(JSC::AtomicsObject::create):
(JSC::AtomicsObject::createStructure):
(JSC::AtomicsObject::finishCreation):
(JSC::atomicsFuncAdd):
(JSC::atomicsFuncAnd):
(JSC::atomicsFuncCompareExchange):
(JSC::atomicsFuncExchange):
(JSC::atomicsFuncIsLockFree):
(JSC::atomicsFuncLoad):
(JSC::atomicsFuncOr):
(JSC::atomicsFuncStore):
(JSC::atomicsFuncSub):
(JSC::atomicsFuncWait):
(JSC::atomicsFuncWake):
(JSC::atomicsFuncXor):
* runtime/AtomicsObject.h: Added.
* runtime/CommonIdentifiers.h:
* runtime/DataView.cpp:
(JSC::DataView::wrap):
* runtime/GenericTypedArrayViewInlines.h:
(JSC::GenericTypedArrayView<Adaptor>::subarray):
* runtime/Intrinsic.h:
* runtime/JSArrayBuffer.cpp:
(JSC::JSArrayBuffer::finishCreation):
(JSC::JSArrayBuffer::isShared):
(JSC::JSArrayBuffer::sharingMode):
* runtime/JSArrayBuffer.h:
(JSC::toPossiblySharedArrayBuffer):
(JSC::toUnsharedArrayBuffer):
(JSC::JSArrayBuffer::toWrapped):
(JSC::toArrayBuffer): Deleted.
* runtime/JSArrayBufferConstructor.cpp:
(JSC::JSArrayBufferConstructor::JSArrayBufferConstructor):
(JSC::JSArrayBufferConstructor::finishCreation):
(JSC::JSArrayBufferConstructor::create):
(JSC::constructArrayBuffer):
* runtime/JSArrayBufferConstructor.h:
(JSC::JSArrayBufferConstructor::sharingMode):
* runtime/JSArrayBufferPrototype.cpp:
(JSC::arrayBufferProtoFuncSlice):
(JSC::JSArrayBufferPrototype::JSArrayBufferPrototype):
(JSC::JSArrayBufferPrototype::finishCreation):
(JSC::JSArrayBufferPrototype::create):
* runtime/JSArrayBufferPrototype.h:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::finishCreation):
(JSC::JSArrayBufferView::visitChildren):
(JSC::JSArrayBufferView::unsharedBuffer):
(JSC::JSArrayBufferView::unsharedJSBuffer):
(JSC::JSArrayBufferView::possiblySharedJSBuffer):
(JSC::JSArrayBufferView::neuter):
(JSC::JSArrayBufferView::toWrapped): Deleted.
* runtime/JSArrayBufferView.h:
(JSC::JSArrayBufferView::jsBuffer): Deleted.
* runtime/JSArrayBufferViewInlines.h:
(JSC::JSArrayBufferView::isShared):
(JSC::JSArrayBufferView::possiblySharedBuffer):
(JSC::JSArrayBufferView::possiblySharedImpl):
(JSC::JSArrayBufferView::unsharedImpl):
(JSC::JSArrayBufferView::byteOffset):
(JSC::JSArrayBufferView::toWrapped):
(JSC::JSArrayBufferView::buffer): Deleted.
(JSC::JSArrayBufferView::impl): Deleted.
(JSC::JSArrayBufferView::neuter): Deleted.
* runtime/JSDataView.cpp:
(JSC::JSDataView::possiblySharedTypedImpl):
(JSC::JSDataView::unsharedTypedImpl):
(JSC::JSDataView::getTypedArrayImpl):
(JSC::JSDataView::typedImpl): Deleted.
* runtime/JSDataView.h:
(JSC::JSDataView::possiblySharedBuffer):
(JSC::JSDataView::unsharedBuffer):
(JSC::JSDataView::buffer): Deleted.
* runtime/JSDataViewPrototype.cpp:
(JSC::dataViewProtoGetterBuffer):
* runtime/JSGenericTypedArrayView.h:
(JSC::toPossiblySharedNativeTypedView):
(JSC::toUnsharedNativeTypedView):
(JSC::JSGenericTypedArrayView<Adaptor>::toWrapped):
(JSC::JSGenericTypedArrayView::typedImpl): Deleted.
(JSC::toNativeTypedView): Deleted.
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::create):
(JSC::JSGenericTypedArrayView<Adaptor>::possiblySharedTypedImpl):
(JSC::JSGenericTypedArrayView<Adaptor>::unsharedTypedImpl):
(JSC::JSGenericTypedArrayView<Adaptor>::getTypedArrayImpl):
* runtime/JSGenericTypedArrayViewPrototypeFunctions.h:
(JSC::genericTypedArrayViewProtoGetterFuncBuffer):
(JSC::genericTypedArrayViewPrivateFuncSubarrayCreate):
* runtime/JSGlobalObject.cpp:
(JSC::createAtomicsProperty):
(JSC::JSGlobalObject::init):
(JSC::JSGlobalObject::visitChildren):
* runtime/JSGlobalObject.h:
(JSC::JSGlobalObject::arrayBufferPrototype):
(JSC::JSGlobalObject::arrayBufferStructure):
* runtime/MathObject.cpp:
* runtime/RuntimeFlags.h:
* runtime/SimpleTypedArrayController.cpp:
(JSC::SimpleTypedArrayController::toJS):
* runtime/TypedArrayType.h:
(JSC::typedArrayTypeForType):
Source/WebCore:
New tests added in the LayoutTests/workers/sab directory.
This teaches WebCore that a typed array could be shared or not. By default, WebCore will
reject shared typed arrays as if they were not typed arrays. This ensures that we don't get
race conditions in code that can't handle it.
If you postMessage a SharedArrayBuffer or something that wraps it, you will send the shared
memory to the other worker.
* Modules/encryptedmedia/CDMSessionClearKey.cpp:
(WebCore::CDMSessionClearKey::cachedKeyForKeyID):
* Modules/fetch/FetchBody.cpp:
(WebCore::FetchBody::extract):
* Modules/mediastream/RTCDataChannel.cpp:
(WebCore::RTCDataChannel::send):
* Modules/webaudio/AudioBuffer.cpp:
(WebCore::AudioBuffer::getChannelData):
* Modules/websockets/WebSocket.cpp:
(WebCore::WebSocket::send):
* bindings/js/JSBlobCustom.cpp:
(WebCore::constructJSBlob):
* bindings/js/JSCryptoAlgorithmDictionary.cpp:
(WebCore::createRsaKeyGenParams):
* bindings/js/JSCryptoCustom.cpp:
(WebCore::JSCrypto::getRandomValues):
* bindings/js/JSCryptoOperationData.cpp:
(WebCore::cryptoOperationDataFromJSValue):
* bindings/js/JSDOMBinding.h:
(WebCore::toJS):
(WebCore::toPossiblySharedArrayBufferView):
(WebCore::toUnsharedArrayBufferView):
(WebCore::toPossiblySharedInt8Array):
(WebCore::toPossiblySharedInt16Array):
(WebCore::toPossiblySharedInt32Array):
(WebCore::toPossiblySharedUint8Array):
(WebCore::toPossiblySharedUint8ClampedArray):
(WebCore::toPossiblySharedUint16Array):
(WebCore::toPossiblySharedUint32Array):
(WebCore::toPossiblySharedFloat32Array):
(WebCore::toPossiblySharedFloat64Array):
(WebCore::toUnsharedInt8Array):
(WebCore::toUnsharedInt16Array):
(WebCore::toUnsharedInt32Array):
(WebCore::toUnsharedUint8Array):
(WebCore::toUnsharedUint8ClampedArray):
(WebCore::toUnsharedUint16Array):
(WebCore::toUnsharedUint32Array):
(WebCore::toUnsharedFloat32Array):
(WebCore::toUnsharedFloat64Array):
(WebCore::toArrayBufferView): Deleted.
(WebCore::toInt8Array): Deleted.
(WebCore::toInt16Array): Deleted.
(WebCore::toInt32Array): Deleted.
(WebCore::toUint8Array): Deleted.
(WebCore::toUint8ClampedArray): Deleted.
(WebCore::toUint16Array): Deleted.
(WebCore::toUint32Array): Deleted.
(WebCore::toFloat32Array): Deleted.
(WebCore::toFloat64Array): Deleted.
* bindings/js/JSDataCueCustom.cpp:
(WebCore::constructJSDataCue):
* bindings/js/JSDictionary.cpp:
(WebCore::JSDictionary::convertValue):
* bindings/js/JSFileCustom.cpp:
(WebCore::constructJSFile):
* bindings/js/JSMessagePortCustom.cpp:
(WebCore::extractTransferables):
* bindings/js/JSWebGLRenderingContextBaseCustom.cpp:
(WebCore::dataFunctionf):
(WebCore::dataFunctioni):
(WebCore::dataFunctionMatrix):
* bindings/js/JSXMLHttpRequestCustom.cpp:
(WebCore::JSXMLHttpRequest::send):
* bindings/js/SerializedScriptValue.cpp:
(WebCore::CloneSerializer::dumpArrayBufferView):
(WebCore::CloneSerializer::dumpIfTerminal):
(WebCore::CloneDeserializer::readArrayBufferView):
(WebCore::CloneDeserializer::readTerminal):
(WebCore::SerializedScriptValue::transferArrayBuffers):
* bindings/js/StructuredClone.cpp:
(WebCore::structuredCloneArrayBuffer):
(WebCore::structuredCloneArrayBufferView):
* bindings/scripts/CodeGeneratorJS.pm:
(JSValueToNative):
* css/FontFace.cpp:
(WebCore::FontFace::create):
* html/canvas/WebGL2RenderingContext.cpp:
(WebCore::WebGL2RenderingContext::bufferData):
(WebCore::WebGL2RenderingContext::bufferSubData):
* platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp:
(WebCore::MediaPlayerPrivateAVFoundation::extractKeyURIKeyIDAndCertificateFromInitData):
Source/WebKit/mac:
Support the RuntimeFlag.
* WebView/WebPreferencesPrivate.h:
Source/WebKit/win:
Support the RuntimeFlag.
* Interfaces/IWebPreferencesPrivate.idl:
Source/WebKit2:
Adds some small things we need for SharedArrayBuffer.
* UIProcess/API/C/WKPreferencesRefPrivate.h:
* UIProcess/API/Cocoa/WKPreferencesPrivate.h:
* WebProcess/InjectedBundle/InjectedBundle.cpp:
(WebKit::InjectedBundle::createWebDataFromUint8Array):
Source/WTF:
Adds some small things we need for SharedArrayBuffer.
* wtf/Atomics.h:
(WTF::Atomic::compareExchangeWeakRelaxed):
(WTF::Atomic::exchangeAdd):
(WTF::Atomic::exchangeAnd):
(WTF::Atomic::exchangeOr):
(WTF::Atomic::exchangeSub):
(WTF::Atomic::exchangeXor):
(WTF::atomicLoad):
(WTF::atomicStore):
(WTF::atomicCompareExchangeWeak):
(WTF::atomicCompareExchangeWeakRelaxed):
(WTF::atomicCompareExchangeStrong):
(WTF::atomicExchangeAdd):
(WTF::atomicExchangeAnd):
(WTF::atomicExchangeOr):
(WTF::atomicExchangeSub):
(WTF::atomicExchangeXor):
(WTF::atomicExchange):
(WTF::Atomic::exchangeAndAdd): Deleted.
(WTF::weakCompareAndSwap): Deleted.
We need to be able to do atomics operations on naked pointers. We also need to be able to do
all of the things that std::atomic does. This adds those things and renames
weakCompareAndSwap to atomicCompareExchangeWeakRelaxed so that we're using consistent
terminology.
* wtf/Bitmap.h:
(WTF::WordType>::concurrentTestAndSet): Renamed weakCompareAndSwap.
(WTF::WordType>::concurrentTestAndClear): Renamed weakCompareAndSwap.
* wtf/FastBitVector.h:
(WTF::FastBitVector::atomicSetAndCheck): Renamed weakCompareAndSwap.
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkCount):
* wtf/ParkingLot.h:
Added unparkCount(), which lets you unpark some bounded number of threads and returns the
number of threads unparked. This is just a modest extension of unparkAll(). unparkAll() now
just calls unparkCount(ptr, UINT_MAX).
Tools:
Use the right kind of typed array API.
* DumpRenderTree/TestRunner.cpp:
(setAudioResultCallback):
LayoutTests:
Adding tests. This is a work in progress.
* workers/sab: Added.
* workers/sab/simple-worker-1.js: Added.
(onmessage):
* workers/sab/simple-worker-2.js: Added.
(onmessage):
* workers/sab/simple.html: Added.
Canonical link: https://commits.webkit.org/181984@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208209 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-01 03:10:00 +00:00
|
|
|
|
|
|
|
WTF_EXPORT_PRIVATE static unsigned unparkCount(const void* address, unsigned count);
|
WTF::Lock should not suffer from the thundering herd
https://bugs.webkit.org/show_bug.cgi?id=147947
Reviewed by Geoffrey Garen.
Source/WTF:
This changes Lock::unlockSlow() to use unparkOne() instead of unparkAll(). The problem with
doing this is that it's not obvious after calling unparkOne() if there are any other threads
that are still parked on the lock's queue. If we assume that there are and leave the
hasParkedBit set, then future calls to unlock() will take the slow path. We don't want that
if there aren't actually any threads parked. On the other hand, if we assume that there
aren't any threads parked and clear the hasParkedBit, then if there actually were some
threads parked, then they may never be awoken since future calls to unlock() won't take slow
path and so won't call unparkOne(). In other words, we need a way to be very precise about
when we clear the hasParkedBit and we need to do it in a race-free way: it can't be the case
that we clear the bit just as some thread gets parked on the queue.
A similar problem arises in futexes, and one of the solutions is to have a thread that
acquires a lock after parking sets the hasParkedBit. This is what Rusty Russel's usersem
does. It's a subtle algorithm. Also, it means that if a thread barges in before the unparked
thread runs, then that barging thread will not know that there are threads parked. This
could increase the severity of barging.
Since ParkingLot is a user-level API, we don't have to worry about the kernel-user security
issues and so we can expose callbacks while ParkingLot is holding its internal locks. This
change does exactly that for unparkOne(). The new variant of unparkOne() will call a user
function while the queue from which we are unparking is locked. The callback is told basic
stats about the queue: did we unpark a thread this time, and could there be more threads to
unpark in the future. The callback runs while it's impossible for the queue state to change,
since the ParkingLot's internal locks for the queue is held. This means that
Lock::unlockSlow() can either clear, or leave, the hasParkedBit while releasing the lock
inside the callback from unparkOne(). This takes care of the thundering herd problem while
also reducing the greed that arises from barging threads.
This required some careful reworking of the ParkingLot algorithm. The first thing I noticed
was that the ThreadData::shouldPark flag was useless, since it's set exactly when
ThreadData::address is non-null. Then I had to make sure that dequeue() could lazily create
both hashtables and buckets, since the "callback is called while queue is locked" invariant
requires that we didn't exit early due to the hashtable or bucket not being present. Note
that all of this is done in such a way that the old unparkOne() and unparkAll() don't have
to create any buckets, though they now may create the hashtable. We don't care as much about
the hashtable being created by unpark since it's just such an unlikely scenario and it would
only happen once.
This change reduces the kernel CPU usage of WTF::Lock for the long critical section test by
about 8x and makes it always perform as well as WTF::WordLock and WTF::Mutex for that
benchmark.
* benchmarks/LockSpeedTest.cpp:
* wtf/Lock.cpp:
(WTF::LockBase::unlockSlow):
* wtf/Lock.h:
(WTF::LockBase::isLocked):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
* wtf/WordLock.h:
(WTF::WordLock::isLocked):
(WTF::WordLock::isFullyReset):
Tools:
Add testing that checks that locks return to a pristine state after contention is over.
* TestWebKitAPI/Tests/WTF/Lock.cpp:
(TestWebKitAPI::LockInspector::isFullyReset):
(TestWebKitAPI::runLockTest):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/166072@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188374 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 03:51:25 +00:00
|
|
|
|
2015-08-11 19:51:35 +00:00
|
|
|
// Unparks every thread from the queue associated with the given address, which cannot be null.
|
|
|
|
WTF_EXPORT_PRIVATE static void unparkAll(const void* address);
|
|
|
|
|
|
|
|
// Locks the parking lot and walks all of the parked threads and the addresses they are waiting
|
|
|
|
// on. Threads that are on the same queue are guaranteed to be walked from first to last, but the
|
|
|
|
// queues may be randomly interleaved. For example, if the queue for address A1 has T1 and T2 and
|
|
|
|
// the queue for address A2 has T3 and T4, then you might see iteration orders like:
|
|
|
|
//
|
|
|
|
// A1,T1 A1,T2 A2,T3 A2,T4
|
|
|
|
// A2,T3 A2,T4 A1,T1 A1,T2
|
|
|
|
// A1,T1 A2,T3 A1,T2 A2,T4
|
|
|
|
// A1,T1 A2,T3 A2,T4 A1,T2
|
|
|
|
//
|
Replace all remaining uses of WTF::Mutex with WTF::Lock
https://bugs.webkit.org/show_bug.cgi?id=148089
Reviewed by Geoffrey Garen.
Source/WebCore:
No new tests because no new behavior.
* Modules/webaudio/AsyncAudioDecoder.cpp:
(WebCore::AsyncAudioDecoder::AsyncAudioDecoder):
(WebCore::AsyncAudioDecoder::runLoop):
* Modules/webaudio/AsyncAudioDecoder.h:
* Modules/webaudio/AudioContext.h:
* Modules/webaudio/MediaStreamAudioSource.cpp:
(WebCore::MediaStreamAudioSource::addAudioConsumer):
(WebCore::MediaStreamAudioSource::removeAudioConsumer):
(WebCore::MediaStreamAudioSource::setAudioFormat):
(WebCore::MediaStreamAudioSource::consumeAudio):
* Modules/webaudio/MediaStreamAudioSource.h:
* Modules/webdatabase/Database.cpp:
(WebCore::Database::close):
(WebCore::Database::runTransaction):
(WebCore::Database::inProgressTransactionCompleted):
(WebCore::Database::hasPendingTransaction):
* Modules/webdatabase/Database.h:
* Modules/webdatabase/DatabaseTask.cpp:
(WebCore::DatabaseTaskSynchronizer::taskCompleted):
* Modules/webdatabase/DatabaseTask.h:
* Modules/webdatabase/DatabaseThread.cpp:
(WebCore::DatabaseThread::start):
(WebCore::DatabaseThread::databaseThread):
* Modules/webdatabase/DatabaseThread.h:
* Modules/webdatabase/DatabaseTracker.cpp:
(WebCore::DatabaseTracker::setDatabaseDirectoryPath):
(WebCore::DatabaseTracker::canEstablishDatabase):
(WebCore::DatabaseTracker::retryCanEstablishDatabase):
(WebCore::DatabaseTracker::hasEntryForOrigin):
(WebCore::DatabaseTracker::getMaxSizeForDatabase):
(WebCore::DatabaseTracker::closeAllDatabases):
(WebCore::DatabaseTracker::fullPathForDatabase):
(WebCore::DatabaseTracker::origins):
(WebCore::DatabaseTracker::databaseNamesForOrigin):
(WebCore::DatabaseTracker::detailsForNameAndOrigin):
(WebCore::DatabaseTracker::setDatabaseDetails):
(WebCore::DatabaseTracker::doneCreatingDatabase):
(WebCore::DatabaseTracker::addOpenDatabase):
(WebCore::DatabaseTracker::removeOpenDatabase):
(WebCore::DatabaseTracker::getOpenDatabases):
(WebCore::DatabaseTracker::originLockFor):
(WebCore::DatabaseTracker::quotaForOrigin):
(WebCore::DatabaseTracker::setQuota):
(WebCore::DatabaseTracker::deleteOrigin):
(WebCore::DatabaseTracker::deleteDatabase):
(WebCore::DatabaseTracker::deleteDatabaseFile):
(WebCore::DatabaseTracker::removeDeletedOpenedDatabases):
(WebCore::DatabaseTracker::deleteDatabaseFileIfEmpty):
(WebCore::DatabaseTracker::openDatabaseMutex):
(WebCore::DatabaseTracker::setClient):
(WebCore::notificationMutex):
(WebCore::DatabaseTracker::scheduleNotifyDatabaseChanged):
(WebCore::DatabaseTracker::notifyDatabasesChanged):
* Modules/webdatabase/DatabaseTracker.h:
* Modules/webdatabase/OriginLock.h:
* Modules/webdatabase/SQLCallbackWrapper.h:
(WebCore::SQLCallbackWrapper::clear):
(WebCore::SQLCallbackWrapper::unwrap):
(WebCore::SQLCallbackWrapper::hasCallback):
* Modules/webdatabase/SQLTransactionBackend.cpp:
(WebCore::SQLTransactionBackend::doCleanup):
(WebCore::SQLTransactionBackend::enqueueStatementBackend):
(WebCore::SQLTransactionBackend::getNextStatement):
* Modules/webdatabase/SQLTransactionBackend.h:
* bindings/js/WorkerScriptController.cpp:
(WebCore::WorkerScriptController::scheduleExecutionTermination):
(WebCore::WorkerScriptController::isExecutionTerminating):
* bindings/js/WorkerScriptController.h:
* dom/default/PlatformMessagePortChannel.cpp:
(WebCore::MessagePortChannel::postMessageToRemote):
(WebCore::MessagePortChannel::tryGetMessageFromRemote):
(WebCore::MessagePortChannel::isConnectedTo):
(WebCore::MessagePortChannel::hasPendingActivity):
(WebCore::MessagePortChannel::locallyEntangledPort):
(WebCore::PlatformMessagePortChannel::setRemotePort):
(WebCore::PlatformMessagePortChannel::entangledChannel):
(WebCore::PlatformMessagePortChannel::closeInternal):
* dom/default/PlatformMessagePortChannel.h:
* loader/icon/IconDatabase.cpp:
(WebCore::IconDatabase::removeAllIcons):
(WebCore::IconDatabase::synchronousIconForPageURL):
(WebCore::IconDatabase::synchronousNativeIconForPageURL):
(WebCore::IconDatabase::synchronousIconURLForPageURL):
(WebCore::IconDatabase::retainIconForPageURL):
(WebCore::IconDatabase::performRetainIconForPageURL):
(WebCore::IconDatabase::releaseIconForPageURL):
(WebCore::IconDatabase::performReleaseIconForPageURL):
(WebCore::IconDatabase::setIconDataForIconURL):
(WebCore::IconDatabase::setIconURLForPageURL):
(WebCore::IconDatabase::synchronousLoadDecisionForIconURL):
(WebCore::IconDatabase::synchronousIconDataKnownForIconURL):
(WebCore::IconDatabase::pageURLMappingCount):
(WebCore::IconDatabase::retainedPageURLCount):
(WebCore::IconDatabase::iconRecordCount):
(WebCore::IconDatabase::iconRecordCountWithData):
(WebCore::IconDatabase::wakeSyncThread):
(WebCore::IconDatabase::scheduleOrDeferSyncTimer):
(WebCore::IconDatabase::isOpenBesidesMainThreadCallbacks):
(WebCore::IconDatabase::databasePath):
(WebCore::IconDatabase::getOrCreatePageURLRecord):
(WebCore::IconDatabase::iconDatabaseSyncThread):
(WebCore::IconDatabase::performOpenInitialization):
(WebCore::IconDatabase::performURLImport):
(WebCore::IconDatabase::syncThreadMainLoop):
(WebCore::IconDatabase::performPendingRetainAndReleaseOperations):
(WebCore::IconDatabase::readFromDatabase):
(WebCore::IconDatabase::writeToDatabase):
(WebCore::IconDatabase::pruneUnretainedIcons):
(WebCore::IconDatabase::cleanupSyncThread):
* loader/icon/IconDatabase.h:
* page/scrolling/ScrollingTree.cpp:
(WebCore::ScrollingTree::shouldHandleWheelEventSynchronously):
(WebCore::ScrollingTree::commitNewTreeState):
(WebCore::ScrollingTree::setMainFramePinState):
(WebCore::ScrollingTree::mainFrameScrollPosition):
(WebCore::ScrollingTree::setMainFrameScrollPosition):
(WebCore::ScrollingTree::isPointInNonFastScrollableRegion):
(WebCore::ScrollingTree::isRubberBandInProgress):
(WebCore::ScrollingTree::setMainFrameIsRubberBanding):
(WebCore::ScrollingTree::isScrollSnapInProgress):
(WebCore::ScrollingTree::setMainFrameIsScrollSnapping):
(WebCore::ScrollingTree::setCanRubberBandState):
(WebCore::ScrollingTree::rubberBandsAtLeft):
(WebCore::ScrollingTree::rubberBandsAtRight):
(WebCore::ScrollingTree::rubberBandsAtBottom):
(WebCore::ScrollingTree::rubberBandsAtTop):
(WebCore::ScrollingTree::setScrollPinningBehavior):
(WebCore::ScrollingTree::scrollPinningBehavior):
(WebCore::ScrollingTree::willWheelEventStartSwipeGesture):
(WebCore::ScrollingTree::latchedNode):
(WebCore::ScrollingTree::setLatchedNode):
(WebCore::ScrollingTree::clearLatchedNode):
* page/scrolling/ScrollingTree.h:
* platform/MemoryPressureHandler.h:
* platform/audio/HRTFDatabaseLoader.cpp:
(WebCore::HRTFDatabaseLoader::loadAsynchronously):
(WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion):
* platform/audio/HRTFDatabaseLoader.h:
* platform/cocoa/MemoryPressureHandlerCocoa.mm:
(WebCore::MemoryPressureHandler::setReceivedMemoryPressure):
(WebCore::MemoryPressureHandler::clearMemoryPressure):
(WebCore::MemoryPressureHandler::shouldWaitForMemoryClearMessage):
(WebCore::MemoryPressureHandler::respondToMemoryPressureIfNeeded):
* platform/graphics/DisplayRefreshMonitor.cpp:
(WebCore::DisplayRefreshMonitor::displayDidRefresh):
* platform/graphics/DisplayRefreshMonitor.h:
(WebCore::DisplayRefreshMonitor::setMonotonicAnimationStartTime):
(WebCore::DisplayRefreshMonitor::mutex):
* platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp:
(WebCore::MediaPlayerPrivateAVFoundation::setDelayCallbacks):
(WebCore::MediaPlayerPrivateAVFoundation::clearMainThreadPendingFlag):
(WebCore::MediaPlayerPrivateAVFoundation::dispatchNotification):
* platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h:
* platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp:
(WebCore::AVFWrapper::callbackContext):
(WebCore::AVFWrapper::~AVFWrapper):
(WebCore::AVFWrapper::mapLock):
(WebCore::AVFWrapper::addToMap):
(WebCore::AVFWrapper::removeFromMap):
(WebCore::AVFWrapper::periodicTimeObserverCallback):
(WebCore::AVFWrapper::processNotification):
(WebCore::AVFWrapper::loadPlayableCompletionCallback):
(WebCore::AVFWrapper::loadMetadataCompletionCallback):
(WebCore::AVFWrapper::seekCompletedCallback):
(WebCore::AVFWrapper::processCue):
(WebCore::AVFWrapper::legibleOutputCallback):
(WebCore::AVFWrapper::processShouldWaitForLoadingOfResource):
(WebCore::AVFWrapper::resourceLoaderShouldWaitForLoadingOfRequestedResource):
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp:
(WebCore::InbandTextTrackPrivateGStreamer::handleSample):
(WebCore::InbandTextTrackPrivateGStreamer::notifyTrackOfSample):
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h:
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:
(WebCore::TrackPrivateBaseGStreamer::tagsChanged):
(WebCore::TrackPrivateBaseGStreamer::notifyTrackOfTagsChanged):
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h:
* platform/graphics/mac/DisplayRefreshMonitorMac.cpp:
(WebCore::DisplayRefreshMonitorMac::requestRefreshCallback):
(WebCore::DisplayRefreshMonitorMac::displayLinkFired):
* platform/graphics/win/MediaPlayerPrivateMediaFoundation.cpp:
(WebCore::MediaPlayerPrivateMediaFoundation::addListener):
(WebCore::MediaPlayerPrivateMediaFoundation::removeListener):
(WebCore::MediaPlayerPrivateMediaFoundation::notifyDeleted):
(WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::Invoke):
(WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::onMediaPlayerDeleted):
* platform/graphics/win/MediaPlayerPrivateMediaFoundation.h:
* platform/ios/LegacyTileCache.h:
* platform/ios/LegacyTileCache.mm:
(WebCore::LegacyTileCache::setTilesOpaque):
(WebCore::LegacyTileCache::doLayoutTiles):
(WebCore::LegacyTileCache::setCurrentScale):
(WebCore::LegacyTileCache::commitScaleChange):
(WebCore::LegacyTileCache::layoutTilesNow):
(WebCore::LegacyTileCache::layoutTilesNowForRect):
(WebCore::LegacyTileCache::removeAllNonVisibleTiles):
(WebCore::LegacyTileCache::removeAllTiles):
(WebCore::LegacyTileCache::removeForegroundTiles):
(WebCore::LegacyTileCache::setContentReplacementImage):
(WebCore::LegacyTileCache::contentReplacementImage):
(WebCore::LegacyTileCache::tileCreationTimerFired):
(WebCore::LegacyTileCache::setNeedsDisplayInRect):
(WebCore::LegacyTileCache::updateTilingMode):
(WebCore::LegacyTileCache::setTilingMode):
(WebCore::LegacyTileCache::doPendingRepaints):
(WebCore::LegacyTileCache::flushSavedDisplayRects):
(WebCore::LegacyTileCache::prepareToDraw):
* platform/ios/LegacyTileLayerPool.h:
* platform/ios/LegacyTileLayerPool.mm:
(WebCore::LegacyTileLayerPool::addLayer):
(WebCore::LegacyTileLayerPool::takeLayerWithSize):
(WebCore::LegacyTileLayerPool::setCapacity):
(WebCore::LegacyTileLayerPool::prune):
(WebCore::LegacyTileLayerPool::drain):
* platform/network/curl/CurlDownload.cpp:
(WebCore::CurlDownloadManager::add):
(WebCore::CurlDownloadManager::remove):
(WebCore::CurlDownloadManager::getActiveDownloadCount):
(WebCore::CurlDownloadManager::getPendingDownloadCount):
(WebCore::CurlDownloadManager::stopThreadIfIdle):
(WebCore::CurlDownloadManager::updateHandleList):
(WebCore::CurlDownload::~CurlDownload):
(WebCore::CurlDownload::init):
(WebCore::CurlDownload::getTempPath):
(WebCore::CurlDownload::getUrl):
(WebCore::CurlDownload::getResponse):
(WebCore::CurlDownload::closeFile):
(WebCore::CurlDownload::didReceiveHeader):
(WebCore::CurlDownload::didReceiveData):
(WebCore::CurlDownload::didFail):
* platform/network/curl/CurlDownload.h:
* platform/network/curl/ResourceHandleManager.cpp:
(WebCore::cookieJarPath):
(WebCore::sharedResourceMutex):
(WebCore::curl_lock_callback):
(WebCore::curl_unlock_callback):
* platform/network/ios/QuickLook.mm:
(WebCore::QLDirectoryAttributes):
(qlPreviewConverterDictionaryMutex):
(WebCore::addQLPreviewConverterWithFileForURL):
(WebCore::qlPreviewConverterUTIForURL):
(WebCore::removeQLPreviewConverterForURL):
(WebCore::safeQLURLForDocumentURLAndResourceURL):
* platform/sql/SQLiteDatabase.cpp:
(WebCore::SQLiteDatabase::close):
(WebCore::SQLiteDatabase::maximumSize):
(WebCore::SQLiteDatabase::setMaximumSize):
(WebCore::SQLiteDatabase::pageSize):
(WebCore::SQLiteDatabase::freeSpaceSize):
(WebCore::SQLiteDatabase::totalSize):
(WebCore::SQLiteDatabase::runIncrementalVacuumCommand):
(WebCore::SQLiteDatabase::setAuthorizer):
* platform/sql/SQLiteDatabase.h:
(WebCore::SQLiteDatabase::databaseMutex):
* platform/sql/SQLiteStatement.cpp:
(WebCore::SQLiteStatement::prepare):
(WebCore::SQLiteStatement::step):
* workers/WorkerThread.cpp:
(WebCore::WorkerThread::start):
(WebCore::WorkerThread::workerThread):
(WebCore::WorkerThread::stop):
* workers/WorkerThread.h:
Source/WebKit:
* Storage/StorageAreaSync.cpp:
(WebCore::StorageAreaSync::syncTimerFired):
(WebCore::StorageAreaSync::markImported):
(WebCore::StorageAreaSync::blockUntilImportComplete):
(WebCore::StorageAreaSync::performSync):
* Storage/StorageAreaSync.h:
* Storage/StorageTracker.cpp:
(WebCore::StorageTracker::setDatabaseDirectoryPath):
(WebCore::StorageTracker::finishedImportingOriginIdentifiers):
(WebCore::StorageTracker::syncImportOriginIdentifiers):
(WebCore::StorageTracker::syncFileSystemAndTrackerDatabase):
(WebCore::StorageTracker::setOriginDetails):
(WebCore::StorageTracker::syncSetOriginDetails):
(WebCore::StorageTracker::origins):
(WebCore::StorageTracker::deleteAllOrigins):
(WebCore::StorageTracker::syncDeleteAllOrigins):
(WebCore::StorageTracker::deleteOrigin):
(WebCore::StorageTracker::syncDeleteOrigin):
(WebCore::StorageTracker::canDeleteOrigin):
(WebCore::StorageTracker::cancelDeletingOrigin):
(WebCore::StorageTracker::diskUsageForOrigin):
* Storage/StorageTracker.h:
Source/WebKit/ios:
* WebCoreSupport/WebFixedPositionContent.mm:
(WebFixedPositionContentDataLock):
(-[WebFixedPositionContent scrollOrZoomChanged:]):
(-[WebFixedPositionContent overflowScrollPositionForLayer:changedTo:]):
(-[WebFixedPositionContent setViewportConstrainedLayers:stickyContainerMap:]):
(-[WebFixedPositionContent hasFixedOrStickyPositionLayers]):
(-[WebFixedPositionContent minimumOffsetFromFixedPositionLayersToAnchorEdge:ofRect:inLayer:]):
Source/WebKit/mac:
* Storage/WebDatabaseManager.mm:
(transactionBackgroundTaskIdentifierLock):
(+[WebDatabaseManager startBackgroundTask]):
(+[WebDatabaseManager endBackgroundTask]):
* WebView/WebView.mm:
(-[WebView _synchronizeCustomFixedPositionLayoutRect]):
(-[WebView _setCustomFixedPositionLayoutRectInWebThread:synchronize:]):
(-[WebView _setCustomFixedPositionLayoutRect:]):
(-[WebView _fetchCustomFixedPositionLayoutRect:]):
* WebView/WebViewData.h:
Source/WebKit/win:
* Plugins/PluginMainThreadScheduler.cpp:
(WebCore::PluginMainThreadScheduler::scheduleCall):
(WebCore::PluginMainThreadScheduler::registerPlugin):
(WebCore::PluginMainThreadScheduler::unregisterPlugin):
(WebCore::PluginMainThreadScheduler::dispatchCallsForPlugin):
* Plugins/PluginMainThreadScheduler.h:
* WebIconDatabase.cpp:
(WebIconDatabase::didRemoveAllIcons):
(WebIconDatabase::didImportIconURLForPageURL):
(WebIconDatabase::deliverNotifications):
* WebIconDatabase.h:
* WebLocalizableStrings.cpp:
(mainBundleLocStrings):
(frameworkLocStringsMutex):
(findCachedString):
(cacheString):
Source/WebKit2:
* DatabaseProcess/DatabaseProcess.cpp:
(WebKit::DatabaseProcess::postDatabaseTask):
(WebKit::DatabaseProcess::performNextDatabaseTask):
* DatabaseProcess/DatabaseProcess.h:
* DatabaseProcess/IndexedDB/UniqueIDBDatabase.cpp:
(WebKit::UniqueIDBDatabase::shutdown):
(WebKit::UniqueIDBDatabase::postMainThreadTask):
(WebKit::UniqueIDBDatabase::performNextMainThreadTask):
(WebKit::UniqueIDBDatabase::postDatabaseTask):
(WebKit::UniqueIDBDatabase::performNextDatabaseTask):
* DatabaseProcess/IndexedDB/UniqueIDBDatabase.h:
* Platform/IPC/Connection.cpp:
(IPC::Connection::sendSyncMessage):
(IPC::Connection::sendSyncMessageFromSecondaryThread):
(IPC::Connection::waitForSyncReply):
(IPC::Connection::processIncomingSyncReply):
(IPC::Connection::connectionDidClose):
* Platform/IPC/Connection.h:
* Shared/CoordinatedGraphics/CoordinatedGraphicsScene.cpp:
(WebKit::CoordinatedGraphicsScene::appendUpdate):
* Shared/CoordinatedGraphics/CoordinatedGraphicsScene.h:
* Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.cpp:
(WebKit::ThreadedCompositor::createCompositingThread):
(WebKit::ThreadedCompositor::runCompositingThread):
(WebKit::ThreadedCompositor::terminateCompositingThread):
* Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.h:
* Shared/Network/CustomProtocols/Cocoa/CustomProtocolManagerCocoa.mm:
(WebKit::CustomProtocolManager::addCustomProtocol):
(WebKit::CustomProtocolManager::removeCustomProtocol):
(WebKit::CustomProtocolManager::registerScheme):
(WebKit::CustomProtocolManager::unregisterScheme):
(WebKit::CustomProtocolManager::supportsScheme):
(WebKit::CustomProtocolManager::protocolForID):
* Shared/Network/CustomProtocols/CustomProtocolManager.h:
* Shared/linux/SeccompFilters/SeccompBroker.cpp:
* WebProcess/Plugins/PluginProcessConnectionManager.cpp:
(WebKit::PluginProcessConnectionManager::getPluginProcessConnection):
(WebKit::PluginProcessConnectionManager::removePluginProcessConnection):
(WebKit::PluginProcessConnectionManager::pluginProcessCrashed):
* WebProcess/Plugins/PluginProcessConnectionManager.h:
* WebProcess/WebPage/EventDispatcher.cpp:
(WebKit::EventDispatcher::addScrollingTreeForPage):
(WebKit::EventDispatcher::removeScrollingTreeForPage):
(WebKit::EventDispatcher::wheelEvent):
* WebProcess/WebPage/EventDispatcher.h:
* WebProcess/soup/WebKitSoupRequestInputStream.cpp:
(webkitSoupRequestInputStreamReadAsync):
(webkitSoupRequestInputStreamAddData):
Source/WTF:
This also beefs up and rationalizes the Condition API, so that it can deal with units of time
other than just steady_clock. This makes it easier to port ThreadCondition and
std::condition_variable code over to Condition. This patch does not take a position on what
kind of time is best; from reading a lot of the uses in WebCore, it seems like our use of
double to measure seconds is often nicer than the many different classes in std::chrono.
Also added a Condition speed test, to make sure that all of this is a good idea. And indeed it
is. The 1-producer/1-consumer scenario with a 100-element queue runs 36x faster using
Lock/Condition than Mutex/ThreadCondition when you use the notifyOne()-per-enqueue style. It
runs 58x faster with Lock/Condition when you use the notifyAll()-at-boundary style. Note that
I have a bug open for making the notifyOne()-per-enqueue style even faster:
https://bugs.webkit.org/show_bug.cgi?id=148090. Also, the 10-consumer/10-producer scenario with
a 100-element queue runs 20x faster with Lock/Condition for notifyOne()-per-enqueue and 30x
faster with notifyAll()-at-boundary. The only way to tweak the test to get
Mutex/ThreadCondition to win is to have one producer, one consumer, a 1-element queue, and use
the notifyOne()-per-enqueue style. In that case, one of the two threads is going to be waiting
most of the time and the test basically measures wake-up latency and nothing else. Because
Condition::wait() does a little bit more work than ThreadCondition::wait(),
Mutex/ThreadCondition end up running 3% faster in this test case. But if you vary any of the
parameters of the test, Mutex/ThreadCondition ends up losing - all it takes is more threads or
a queue size of 5 or more. To my knowledge, we never do producer/consumer with a queue bounded
to one element precisely because that approach is the least efficient regardless of locking
algorithm. For example, neither WTF::MessageQueue nor DFG::Worklist have any bounds on their
queue size. So, it seems that replacing all uses of system mutexes and condition variables with
our own thing is a great idea.
* benchmarks/LockSpeedTest.cpp:
* benchmarks/ConditionSpeedTest.cpp: Added.
* wtf/Condition.h:
(WTF::Condition::Condition):
(WTF::Condition::waitUntil):
(WTF::Condition::waitFor):
(WTF::Condition::wait):
(WTF::Condition::waitUntilWallClockSeconds):
(WTF::Condition::waitUntilMonotonicClockSeconds):
(WTF::Condition::notifyOne):
(WTF::Condition::notifyAll):
(WTF::Condition::waitForSecondsImpl):
(WTF::Condition::waitForImpl):
(WTF::Condition::absoluteFromRelative):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime):
(WTF::MessageQueue<DataType>::append):
(WTF::MessageQueue<DataType>::appendAndKill):
(WTF::MessageQueue<DataType>::appendAndCheckEmpty):
(WTF::MessageQueue<DataType>::prepend):
(WTF::MessageQueue<DataType>::removeIf):
(WTF::MessageQueue<DataType>::isEmpty):
(WTF::MessageQueue<DataType>::kill):
(WTF::MessageQueue<DataType>::killed):
* wtf/ParallelJobsGeneric.cpp:
(WTF::ParallelEnvironment::ThreadPrivate::execute):
(WTF::ParallelEnvironment::ThreadPrivate::waitForFinish):
(WTF::ParallelEnvironment::ThreadPrivate::workerThread):
* wtf/ParallelJobsGeneric.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionally):
* wtf/ParkingLot.h:
(WTF::ParkingLot::compareAndPark):
* wtf/ThreadingPthreads.cpp:
(WTF::initializeThreading):
* wtf/ThreadingWin.cpp:
(WTF::initializeThreading):
* wtf/dtoa.cpp:
(WTF::pow5mult):
* wtf/dtoa.h:
Tools:
* DumpRenderTree/JavaScriptThreading.cpp:
(javaScriptThreadsMutex):
(runJavaScriptThread):
(startJavaScriptThreads):
(stopJavaScriptThreads):
* TestWebKitAPI/Tests/WTF/Condition.cpp: Fixed a bug in the test that I found from turning the test into a benchmark.
* TestWebKitAPI/Tests/WTF/WorkQueue.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/glib/WorkQueueGLib.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/166252@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188594 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-18 19:31:28 +00:00
|
|
|
// As well as many other possible interleavings that all have T1 before T2 and T3 before T4 but are
|
2015-08-11 19:51:35 +00:00
|
|
|
// otherwise unconstrained. This method is useful primarily for debugging. It's also used by unit
|
|
|
|
// tests.
|
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
https://bugs.webkit.org/show_bug.cgi?id=160125
Reviewed by Geoffrey Garen and Keith Miller.
JSTests:
Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.
* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.
Source/bmalloc:
I needed to tryMemalign, so I added such a thing.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):
Source/JavaScriptCore:
In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.
Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.
At a high level the mechanical changes are:
- We use AuxiliaryBarrier instead of CopyBarrier.
- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
CheckedBoolean stuff, since it's so much more trouble than it's worth.
- The JITs have to emit inlined marked space allocations instead of inline copy space
allocations.
- Everyone has to get used to zeroing their butterflies after allocation instead of relying
on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
space doesn't.
That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:
- MarkedSpace always causes internal fragmentation for array allocations because the vector
length we choose when we resize usually leads to a cell size that doesn't correspond to any
size class. I got around this by making array allocations usually round up vectorLength to
the maximum allowed by the size class that we would have allocated in. Also,
ensureLengthSlow() and friends first make sure that the requested length can't just be
fulfilled with the current allocation size. This safeguard means that not every array
allocation has to do size class queries. For example, the fast path of new Array(length)
never does any size class queries, under the assumption that (1) the speed gained from
avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
size class query, is too small to offset the speed lost by doing the query on every
allocation and (2) new Array(length) is a pretty good hint that resizing is not very
likely.
- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
changes MarkedSpace size classes to use a linear progression for very small sizes followed
by a geometric progression that naturally transitions to a hyperbolic progression. We want
hyperbolic sizes when we get close to blockSize: for example the largest size we want is
payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
next size down should be payloadSize / 3 rounded down, and so on. After the last precise
size (80 bytes), we proceed using a geometric progression, but round up each size to
minimize slop at the end of the block. This naturally causes the geometric progression to
turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
- Large allocations caused massive internal fragmentation, since the smallest large
allocation had to use exactly blockSize, and the largest small allocation used
blockSize / 2. The next size up - the first large allocation size to require two blocks -
also had 50% internal fragmentation. This is because we required large allocations to be
blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
out great since (1) typed arrays won't use large allocations anyway since they have their
own malloc fallback and (2) large array butterflies already have a 8 byte header, which
means that the 8 byte base misalignment aligns the large array payload on a 16 byte
boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
rare as possible; for example, ExecState::vm() skips the check because we know that callees
must be small allocations. It's also possible to use template tricks to do one check for
cell container kind, and then invoke a function specialized for MarkedBlock or a function
specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
that get used from functions that are template-specialized like this. That's mostly to
speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
cutoff for large allocations is runtime-configurable, so long as you don't choose something
so small that callees end up large. I found that 400 bytes is roughly optimal. This means
that the MarkedBlock size classes end up being:
16, 32, 48, 64, 80, 112, 160, 224, 320
The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.
- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
MarkedBlocks because the list that contains the interesting meta-data is allocated on the
side, for better locality during a sequential walk. It's no longer necessary to scan
MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
bits because we now use versioned mark bits: to clear then, just increment the 64-bit
heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
together, these optimizations allowed me to reduce block size to 16KB without losing much
performance. There is some small perf loss on JetStream/splay, but not enough to hurt
JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
progression on membuster. That's not possible yet, since there is still enough per-block
overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
- Even after all of that, copying butterflies was still faster because it allowed us to skip
sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
larger than O(live), especially in an eden collection. Copying satisfies this premise while
mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
that we walk the block to build a free list. This means walking dead space. The new
allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
is completely empty, which is the common case because of the generational hypothesis: the
number of objects that survive an eden collection is a tiny fraction of the number of
objects that had been allocated, and this fraction is so small that there are typically
fewer than one survivors per MarkedBlock. This change was enough to make this change a net
win over tip-of-tree.
- FTL now shares the same allocation fast paths as everything else, which is great, because
bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
control flow, since it won't be able to improve the machine code we write ourselves. GC
fast paths are best written in assembly. So, I've empowered B3 to have even better support
for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
ability to constant-fold some allocator logic (in case we first assume that we're doing a
variable-length allocation but then realize that the length is known). I think it makes
sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
since this makes lowering easier (you can constant fold during lowering more easily) and it
reduces the amount of malloc traffic. In the future, we could teach B3 how to better
constant-fold this code. That would require allowing loads to be constant-folded, which is
doable but hella tricky.
- It used to be that if a logical object allocation required two physical allocations (first
the butterfly and then the cell), then the JIT would emit the code in such a way that a
failure in the second fast path would cause us to forget the successful first physical
allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
register to storing either the butterfly or null, because the butterfly register is anyway
going to be free inside the first allocation. The only overhead here is zeroing the
butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
path, which can then either allocate a butterfly or not. So now we never waste a successful
allocation. This patch implements such a solution both in DFG (where it's easy to do this
since we control registers already) and in FTL (where it's annoying, because mutable
"butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
one version of this patch, this wastage would resonate with some Kraken benchmark: the
benchmark would always allocate N small things followed by one bigger thing. The problem
was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
the JSObject size class, which both the small and big thing shared for their cell, could
hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
it allocated the big thing. So, it would end up having to allocate the big thing's large
butterfly twice, every single time! Ouch!
- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
of the allocator to initialize the object to whatever state they need. It used to be that
we would just use a dumb loop. I initially changed this so that we would end up in memset
for large allocations, but this didn't actually help performance that much. I got a much
better result by playing with different memsets written in assembly. First I wrote one
using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
"rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
of stores (if the size is known), but those paths in the JIT won't trigger for
NewArrayWithSize() if the size is large, since that takes us to the
operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
optimizations here are all in JSArray::create() - that's the hot place for large arrays
that need to be filled with holes.
All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.
We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.
This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):
Source/WebCore:
No new tests because no new WebCore behavior.
Just rewiring #includes.
* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:
Source/WebKit2:
Just rewiring some #includes.
* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:
Source/WTF:
I needed tryFastAlignedMalloc() so I added it.
* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.
Tools:
* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/179778@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
|
|
|
template<typename Func>
|
|
|
|
static void forEach(const Func& func)
|
|
|
|
{
|
[WTF] Introduce Thread class and use RefPtr<Thread> and align Windows Threading implementation semantics to Pthread one
https://bugs.webkit.org/show_bug.cgi?id=170502
Reviewed by Mark Lam.
Source/JavaScriptCore:
* API/tests/CompareAndSwapTest.cpp:
(testCompareAndSwap):
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/air/testair.cpp:
* b3/testb3.cpp:
(JSC::B3::run):
* bytecode/SuperSampler.cpp:
(JSC::initializeSuperSampler):
* dfg/DFGWorklist.cpp:
* disassembler/Disassembler.cpp:
* heap/Heap.cpp:
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::notifyIsSafeToCollect):
* heap/Heap.h:
* heap/MachineStackMarker.cpp:
(JSC::MachineThreads::~MachineThreads):
(JSC::MachineThreads::addCurrentThread):
(JSC::MachineThreads::removeThread):
(JSC::MachineThreads::removeThreadIfFound):
(JSC::MachineThreads::MachineThread::MachineThread):
(JSC::MachineThreads::MachineThread::getRegisters):
(JSC::MachineThreads::MachineThread::Registers::stackPointer):
(JSC::MachineThreads::MachineThread::Registers::framePointer):
(JSC::MachineThreads::MachineThread::Registers::instructionPointer):
(JSC::MachineThreads::MachineThread::Registers::llintPC):
(JSC::MachineThreads::MachineThread::captureStack):
(JSC::MachineThreads::tryCopyOtherThreadStack):
(JSC::MachineThreads::tryCopyOtherThreadStacks):
(pthreadSignalHandlerSuspendResume): Deleted.
(JSC::threadData): Deleted.
(JSC::MachineThreads::Thread::Thread): Deleted.
(JSC::MachineThreads::Thread::createForCurrentThread): Deleted.
(JSC::MachineThreads::Thread::operator==): Deleted.
(JSC::MachineThreads::machineThreadForCurrentThread): Deleted.
(JSC::MachineThreads::ThreadData::ThreadData): Deleted.
(JSC::MachineThreads::ThreadData::~ThreadData): Deleted.
(JSC::MachineThreads::ThreadData::suspend): Deleted.
(JSC::MachineThreads::ThreadData::resume): Deleted.
(JSC::MachineThreads::ThreadData::getRegisters): Deleted.
(JSC::MachineThreads::ThreadData::Registers::stackPointer): Deleted.
(JSC::MachineThreads::ThreadData::Registers::framePointer): Deleted.
(JSC::MachineThreads::ThreadData::Registers::instructionPointer): Deleted.
(JSC::MachineThreads::ThreadData::Registers::llintPC): Deleted.
(JSC::MachineThreads::ThreadData::freeRegisters): Deleted.
(JSC::MachineThreads::ThreadData::captureStack): Deleted.
* heap/MachineStackMarker.h:
(JSC::MachineThreads::MachineThread::suspend):
(JSC::MachineThreads::MachineThread::resume):
(JSC::MachineThreads::MachineThread::threadID):
(JSC::MachineThreads::MachineThread::stackBase):
(JSC::MachineThreads::MachineThread::stackEnd):
(JSC::MachineThreads::threadsListHead):
(JSC::MachineThreads::Thread::operator!=): Deleted.
(JSC::MachineThreads::Thread::suspend): Deleted.
(JSC::MachineThreads::Thread::resume): Deleted.
(JSC::MachineThreads::Thread::getRegisters): Deleted.
(JSC::MachineThreads::Thread::freeRegisters): Deleted.
(JSC::MachineThreads::Thread::captureStack): Deleted.
(JSC::MachineThreads::Thread::platformThread): Deleted.
(JSC::MachineThreads::Thread::stackBase): Deleted.
(JSC::MachineThreads::Thread::stackEnd): Deleted.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
(JSC::ICStats::~ICStats):
* jit/ICStats.h:
* jsc.cpp:
(functionDollarAgentStart):
(startTimeoutThreadIfNeeded):
* runtime/JSLock.cpp:
(JSC::JSLock::lock):
* runtime/JSLock.h:
(JSC::JSLock::ownerThread):
(JSC::JSLock::currentThreadIsHoldingLock):
* runtime/SamplingProfiler.cpp:
(JSC::FrameWalker::isValidFramePointer):
(JSC::SamplingProfiler::SamplingProfiler):
(JSC::SamplingProfiler::createThreadIfNecessary):
(JSC::SamplingProfiler::takeSample):
* runtime/SamplingProfiler.h:
* runtime/VM.h:
(JSC::VM::ownerThread):
* runtime/VMTraps.cpp:
(JSC::findActiveVMAndStackBounds):
(JSC::VMTraps::SignalSender::send):
(JSC::VMTraps::fireTrap):
Source/WebCore:
Mechanical change. Use Thread:: APIs.
* Modules/indexeddb/server/IDBServer.cpp:
(WebCore::IDBServer::IDBServer::IDBServer):
* Modules/indexeddb/server/IDBServer.h:
* Modules/webaudio/AsyncAudioDecoder.cpp:
(WebCore::AsyncAudioDecoder::AsyncAudioDecoder):
(WebCore::AsyncAudioDecoder::~AsyncAudioDecoder):
(WebCore::AsyncAudioDecoder::runLoop):
* Modules/webaudio/AsyncAudioDecoder.h:
* Modules/webaudio/OfflineAudioDestinationNode.cpp:
(WebCore::OfflineAudioDestinationNode::OfflineAudioDestinationNode):
(WebCore::OfflineAudioDestinationNode::uninitialize):
(WebCore::OfflineAudioDestinationNode::startRendering):
* Modules/webaudio/OfflineAudioDestinationNode.h:
* Modules/webdatabase/Database.cpp:
(WebCore::Database::securityOrigin):
* Modules/webdatabase/DatabaseThread.cpp:
(WebCore::DatabaseThread::start):
(WebCore::DatabaseThread::databaseThread):
(WebCore::DatabaseThread::recordDatabaseOpen):
(WebCore::DatabaseThread::recordDatabaseClosed):
* Modules/webdatabase/DatabaseThread.h:
(WebCore::DatabaseThread::getThreadID):
* bindings/js/GCController.cpp:
(WebCore::GCController::garbageCollectOnAlternateThreadForDebugging):
* fileapi/AsyncFileStream.cpp:
(WebCore::callOnFileThread):
* loader/icon/IconDatabase.cpp:
(WebCore::IconDatabase::open):
(WebCore::IconDatabase::close):
* loader/icon/IconDatabase.h:
* page/ResourceUsageThread.cpp:
(WebCore::ResourceUsageThread::createThreadIfNeeded):
* page/ResourceUsageThread.h:
* page/scrolling/ScrollingThread.cpp:
(WebCore::ScrollingThread::ScrollingThread):
(WebCore::ScrollingThread::isCurrentThread):
(WebCore::ScrollingThread::createThreadIfNeeded):
(WebCore::ScrollingThread::threadCallback):
* page/scrolling/ScrollingThread.h:
* platform/audio/HRTFDatabaseLoader.cpp:
(WebCore::HRTFDatabaseLoader::HRTFDatabaseLoader):
(WebCore::HRTFDatabaseLoader::loadAsynchronously):
(WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion):
* platform/audio/HRTFDatabaseLoader.h:
* platform/audio/ReverbConvolver.cpp:
(WebCore::ReverbConvolver::ReverbConvolver):
(WebCore::ReverbConvolver::~ReverbConvolver):
* platform/audio/ReverbConvolver.h:
* platform/audio/gstreamer/AudioFileReaderGStreamer.cpp:
(WebCore::createBusFromAudioFile):
(WebCore::createBusFromInMemoryAudioFile):
* platform/graphics/gstreamer/WebKitWebSourceGStreamer.cpp:
(ResourceHandleStreamingClient::ResourceHandleStreamingClient):
(ResourceHandleStreamingClient::~ResourceHandleStreamingClient):
* platform/network/cf/LoaderRunLoopCF.cpp:
(WebCore::loaderRunLoop):
* platform/network/curl/CurlDownload.cpp:
(WebCore::CurlDownloadManager::startThreadIfNeeded):
(WebCore::CurlDownloadManager::stopThread):
* platform/network/curl/CurlDownload.h:
* platform/network/curl/SocketStreamHandleImpl.h:
* platform/network/curl/SocketStreamHandleImplCurl.cpp:
(WebCore::SocketStreamHandleImpl::startThread):
(WebCore::SocketStreamHandleImpl::stopThread):
* workers/WorkerThread.cpp:
(WebCore::WorkerThread::WorkerThread):
(WebCore::WorkerThread::start):
(WebCore::WorkerThread::workerThread):
* workers/WorkerThread.h:
(WebCore::WorkerThread::threadID):
Source/WebKit:
Mechanical change. Use Thread:: APIs.
* Storage/StorageThread.cpp:
(WebCore::StorageThread::StorageThread):
(WebCore::StorageThread::~StorageThread):
(WebCore::StorageThread::start):
(WebCore::StorageThread::dispatch):
(WebCore::StorageThread::terminate):
* Storage/StorageThread.h:
Source/WebKit2:
Mechanical change. Use Thread:: APIs.
* NetworkProcess/NetworkProcess.cpp:
(WebKit::NetworkProcess::initializeNetworkProcess):
* NetworkProcess/cache/NetworkCacheIOChannelSoup.cpp:
(WebKit::NetworkCache::IOChannel::readSyncInThread):
* Platform/IPC/Connection.cpp:
(IPC::Connection::processIncomingMessage):
* Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.h:
(WebKit::XPCServiceInitializer):
* UIProcess/linux/MemoryPressureMonitor.cpp:
(WebKit::MemoryPressureMonitor::MemoryPressureMonitor):
* WebProcess/WebProcess.cpp:
(WebKit::WebProcess::initializeWebProcess):
Source/WTF:
This patch is refactoring of WTF Threading mechanism to merge JavaScriptCore's threading extension to WTF Threading.
Previously, JavaScriptCore requires richer threading features (such as suspending and resuming threads), and they
are implemented for PlatformThread in JavaScriptCore. But these features should be implemented in WTF Threading side
instead of maintaining JSC's own threading features too. This patch removes these features from JSC and move it to
WTF Threading.
However, current WTF Threading has one problem: Windows version of WTF Threading has different semantics from Pthreads
one. In Windows WTF Threading, we cannot perform any operation after the target thread is detached: WTF Threading stop
tracking the state of the thread once the thread is detached. But this is not the same to Pthreads one. In Pthreads,
pthread_detach just means that the resource of the thread will be destroyed automatically. While some operations like
pthread_join will be rejected, some operations like pthread_kill will be accepted.
The problem is that detached thread can be suspended and resumed in JSC. For example, in jsc.cpp, we start the worker
thread and detach it immediately. In worker thread, we will create VM and thus concurrent GC will suspend and resume
the detached thread. However, in Windows WTF Threading, we have no reference to the detached thread. Thus we cannot
perform suspend and resume operations onto the detached thread.
To solve the problem, we change Windows Threading mechanism drastically to align it to the Pthread semantics. In the
new Threading, we have RefPtr<Thread> class. It holds a handle to a platform thread. We can perform threading operations
with this class. For example, Thread::suspend is offered. And we use destructor of the thread local variable to release
the resources held by RefPtr<Thread>. In Windows, Thread::detach does nothing because the resource will be destroyed
automatically by RefPtr<Thread>.
To do so, we introduce ThreadHolder for Windows. This is similar to the previous ThreadIdentifierData for Pthreads.
It holds RefPtr<Thread> in the thread local storage (technically, it is Fiber Local Storage in Windows). Thread::current()
will return this reference.
The problematic situation is that the order of the deallocation of the thread local storage is not defined. So we should
not touch thread local storage in the destructor of the thread local storage. To avoid such edge cases, we have
currentThread() / Thread::currentID() APIs. They are safe to be called even in the destructor of the other thread local
storage. And in Windows, in the FLS destructor, we will create the thread_local variable to defer the destruction of
the ThreadHolder. We ensure that this destructor is called after the other FLS destructors are called in Windows 10.
This patch is performance neutral.
* WTF.xcodeproj/project.pbxproj:
* benchmarks/ConditionSpeedTest.cpp:
* benchmarks/LockFairnessTest.cpp:
* benchmarks/LockSpeedTest.cpp:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/MainThread.h:
* wtf/MemoryPressureHandler.h:
* wtf/ParallelJobsGeneric.cpp:
(WTF::ParallelEnvironment::ThreadPrivate::tryLockFor):
(WTF::ParallelEnvironment::ThreadPrivate::workerThread):
* wtf/ParallelJobsGeneric.h:
(WTF::ParallelEnvironment::ThreadPrivate::ThreadPrivate): Deleted.
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
* wtf/ParkingLot.h:
(WTF::ParkingLot::forEach):
* wtf/PlatformRegisters.h: Renamed from Source/JavaScriptCore/runtime/PlatformThread.h.
* wtf/RefPtr.h:
(WTF::RefPtr::RefPtr):
* wtf/ThreadFunctionInvocation.h:
(WTF::ThreadFunctionInvocation::ThreadFunctionInvocation):
* wtf/ThreadHolder.cpp: Added.
(WTF::ThreadHolder::~ThreadHolder):
(WTF::ThreadHolder::initialize):
* wtf/ThreadHolder.h: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.h.
(WTF::ThreadHolder::thread):
(WTF::ThreadHolder::ThreadHolder):
* wtf/ThreadHolderPthreads.cpp: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.cpp.
(WTF::ThreadHolder::initializeOnce):
(WTF::ThreadHolder::current):
(WTF::ThreadHolder::destruct):
* wtf/ThreadHolderWin.cpp: Added.
(WTF::threadMapMutex):
(WTF::threadMap):
(WTF::ThreadHolder::initializeOnce):
(WTF::ThreadHolder::current):
(WTF::ThreadHolder::destruct):
* wtf/ThreadSpecific.h:
* wtf/Threading.cpp:
(WTF::Thread::normalizeThreadName):
(WTF::threadEntryPoint):
(WTF::Thread::create):
(WTF::Thread::setCurrentThreadIsUserInteractive):
(WTF::Thread::setCurrentThreadIsUserInitiated):
(WTF::Thread::setGlobalMaxQOSClass):
(WTF::Thread::adjustedQOSClass):
(WTF::Thread::dump):
(WTF::initializeThreading):
(WTF::normalizeThreadName): Deleted.
(WTF::createThread): Deleted.
(WTF::setCurrentThreadIsUserInteractive): Deleted.
(WTF::setCurrentThreadIsUserInitiated): Deleted.
(WTF::setGlobalMaxQOSClass): Deleted.
(WTF::adjustedQOSClass): Deleted.
* wtf/Threading.h:
(WTF::Thread::id):
(WTF::Thread::operator==):
(WTF::Thread::operator!=):
(WTF::Thread::joinableState):
(WTF::Thread::didBecomeDetached):
(WTF::Thread::didJoin):
(WTF::Thread::hasExited):
(WTF::currentThread):
* wtf/ThreadingPthreads.cpp:
(WTF::Thread::Thread):
(WTF::Thread::~Thread):
(WTF::Thread::signalHandlerSuspendResume):
(WTF::Thread::initializePlatformThreading):
(WTF::initializeCurrentThreadEvenIfNonWTFCreated):
(WTF::wtfThreadEntryPoint):
(WTF::Thread::createInternal):
(WTF::Thread::initializeCurrentThreadInternal):
(WTF::Thread::changePriority):
(WTF::Thread::waitForCompletion):
(WTF::Thread::detach):
(WTF::Thread::current):
(WTF::Thread::currentID):
(WTF::Thread::signal):
(WTF::Thread::resume):
(WTF::Thread::getRegisters):
(WTF::Thread::didExit):
(WTF::Thread::establish):
(WTF::PthreadState::PthreadState): Deleted.
(WTF::PthreadState::joinableState): Deleted.
(WTF::PthreadState::pthreadHandle): Deleted.
(WTF::PthreadState::didBecomeDetached): Deleted.
(WTF::PthreadState::didExit): Deleted.
(WTF::PthreadState::didJoin): Deleted.
(WTF::PthreadState::hasExited): Deleted.
(WTF::threadMapMutex): Deleted.
(WTF::initializeThreading): Deleted.
(WTF::threadMap): Deleted.
(WTF::identifierByPthreadHandle): Deleted.
(WTF::establishIdentifierForPthreadHandle): Deleted.
(WTF::pthreadHandleForIdentifierWithLockAlreadyHeld): Deleted.
(WTF::createThreadInternal): Deleted.
(WTF::initializeCurrentThreadInternal): Deleted.
(WTF::changeThreadPriority): Deleted.
(WTF::waitForThreadCompletion): Deleted.
(WTF::detachThread): Deleted.
(WTF::threadDidExit): Deleted.
(WTF::currentThread): Deleted.
(WTF::signalThread): Deleted.
* wtf/ThreadingWin.cpp:
(WTF::Thread::Thread):
(WTF::Thread::~Thread):
(WTF::Thread::initializeCurrentThreadInternal):
(WTF::Thread::initializePlatformThreading):
(WTF::wtfThreadEntryPoint):
(WTF::Thread::createInternal):
(WTF::Thread::changePriority):
(WTF::Thread::waitForCompletion):
(WTF::Thread::detach):
(WTF::Thread::resume):
(WTF::Thread::getRegisters):
(WTF::Thread::current):
(WTF::Thread::currentID):
(WTF::Thread::didExit):
(WTF::Thread::establish):
(WTF::initializeCurrentThreadInternal): Deleted.
(WTF::threadMapMutex): Deleted.
(WTF::initializeThreading): Deleted.
(WTF::threadMap): Deleted.
(WTF::storeThreadHandleByIdentifier): Deleted.
(WTF::threadHandleForIdentifier): Deleted.
(WTF::clearThreadHandleForIdentifier): Deleted.
(WTF::createThreadInternal): Deleted.
(WTF::changeThreadPriority): Deleted.
(WTF::waitForThreadCompletion): Deleted.
(WTF::detachThread): Deleted.
(WTF::currentThread): Deleted.
* wtf/WorkQueue.cpp:
(WTF::WorkQueue::concurrentApply):
* wtf/WorkQueue.h:
* wtf/cocoa/WorkQueueCocoa.cpp:
(WTF::dispatchQOSClass):
* wtf/generic/WorkQueueGeneric.cpp:
(WorkQueue::platformInitialize):
(WorkQueue::platformInvalidate):
* wtf/linux/MemoryPressureHandlerLinux.cpp:
(WTF::MemoryPressureHandler::EventFDPoller::EventFDPoller):
(WTF::MemoryPressureHandler::EventFDPoller::~EventFDPoller):
* wtf/win/MainThreadWin.cpp:
(WTF::initializeMainThreadPlatform):
Tools:
Mechanical change. Use Thread:: APIs.
* DumpRenderTree/JavaScriptThreading.cpp:
(runJavaScriptThread):
(startJavaScriptThreads):
(stopJavaScriptThreads):
* DumpRenderTree/mac/DumpRenderTree.mm:
(testThreadIdentifierMap):
* TestWebKitAPI/Tests/WTF/Condition.cpp:
* TestWebKitAPI/Tests/WTF/Lock.cpp:
(TestWebKitAPI::runLockTest):
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/187690@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@215265 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-04-12 12:08:29 +00:00
|
|
|
forEachImpl(scopedLambdaRef<void(Thread&, const void*)>(func));
|
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
https://bugs.webkit.org/show_bug.cgi?id=160125
Reviewed by Geoffrey Garen and Keith Miller.
JSTests:
Most of the things I did properly covered by existing tests, but I found some simple cases of
unshifting that had sketchy coverage.
* stress/array-storage-array-unshift.js: Added.
* stress/contiguous-array-unshift.js: Added.
* stress/double-array-unshift.js: Added.
* stress/int32-array-unshift.js: Added.
Source/bmalloc:
I needed to tryMemalign, so I added such a thing.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::allocate):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
* bmalloc/Allocator.h:
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
* bmalloc/bmalloc.h:
(bmalloc::api::tryMemalign):
Source/JavaScriptCore:
In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
to get rid of copied space. This change moves copied space's biggest client over to marked
space.
Moving butterflies to marked space means having them use the new Auxiliary HeapCell
allocation path. This is a fairly mechanical change, but it caused performance regressions
everywhere, so this change also fixes MarkedSpace's performance issues.
At a high level the mechanical changes are:
- We use AuxiliaryBarrier instead of CopyBarrier.
- We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
CheckedBoolean stuff, since it's so much more trouble than it's worth.
- The JITs have to emit inlined marked space allocations instead of inline copy space
allocations.
- Everyone has to get used to zeroing their butterflies after allocation instead of relying
on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
space doesn't.
That's about 1/3 of this change. But this led to performance problems, which I fixed with
optimizations that amounted to a major MarkedSpace rewrite:
- MarkedSpace always causes internal fragmentation for array allocations because the vector
length we choose when we resize usually leads to a cell size that doesn't correspond to any
size class. I got around this by making array allocations usually round up vectorLength to
the maximum allowed by the size class that we would have allocated in. Also,
ensureLengthSlow() and friends first make sure that the requested length can't just be
fulfilled with the current allocation size. This safeguard means that not every array
allocation has to do size class queries. For example, the fast path of new Array(length)
never does any size class queries, under the assumption that (1) the speed gained from
avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the
size class query, is too small to offset the speed lost by doing the query on every
allocation and (2) new Array(length) is a pretty good hint that resizing is not very
likely.
- Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
changes MarkedSpace size classes to use a linear progression for very small sizes followed
by a geometric progression that naturally transitions to a hyperbolic progression. We want
hyperbolic sizes when we get close to blockSize: for example the largest size we want is
payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
next size down should be payloadSize / 3 rounded down, and so on. After the last precise
size (80 bytes), we proceed using a geometric progression, but round up each size to
minimize slop at the end of the block. This naturally causes the geometric progression to
turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
it can be controlled with runtime options. I found that a base of 1.4 works pretty well.
- Large allocations caused massive internal fragmentation, since the smallest large
allocation had to use exactly blockSize, and the largest small allocation used
blockSize / 2. The next size up - the first large allocation size to require two blocks -
also had 50% internal fragmentation. This is because we required large allocations to be
blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of
that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be
owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that
a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set.
Basically, large allocations are deliberately misaligned by 8 bytes. This actually works
out great since (1) typed arrays won't use large allocations anyway since they have their
own malloc fallback and (2) large array butterflies already have a 8 byte header, which
means that the 8 byte base misalignment aligns the large array payload on a 16 byte
boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as
rare as possible; for example, ExecState::vm() skips the check because we know that callees
must be small allocations. It's also possible to use template tricks to do one check for
cell container kind, and then invoke a function specialized for MarkedBlock or a function
specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods
that get used from functions that are template-specialized like this. That's mostly to
speed up the GC marking code. Most other code can use CellContainer API or HeapCell API
directly. That's another thing: HeapCell, the common base of JSCell and auxiliary
allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(),
HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size
cutoff for large allocations is runtime-configurable, so long as you don't choose something
so small that callees end up large. I found that 400 bytes is roughly optimal. This means
that the MarkedBlock size classes end up being:
16, 32, 48, 64, 80, 112, 160, 224, 320
The next size class would have been 432, but that's above the 400 byte cutoff. All of this
is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what
size classes you end up with by doing --dumpSizeClasses=true.
- Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot
of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a
lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all
MarkedBlocks because the list that contains the interesting meta-data is allocated on the
side, for better locality during a sequential walk. It's no longer necessary to scan
MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are
maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark
bits because we now use versioned mark bits: to clear then, just increment the 64-bit
heap version. It's no longer necessary to scan retired MarkedBlocks while allocating
because marking retires them on-the-fly. It's no longer necessary to sort all blocks in
the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put
together, these optimizations allowed me to reduce block size to 16KB without losing much
performance. There is some small perf loss on JetStream/splay, but not enough to hurt
JetStream overall. I tried reducing block sizes further, to 4KB, since that is a
progression on membuster. That's not possible yet, since there is still enough per-block
overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving
this further: https://bugs.webkit.org/show_bug.cgi?id=161581.
- Even after all of that, copying butterflies was still faster because it allowed us to skip
sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
larger than O(live), especially in an eden collection. Copying satisfies this premise while
mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
that we walk the block to build a free list. This means walking dead space. The new
allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock
is completely empty, which is the common case because of the generational hypothesis: the
number of objects that survive an eden collection is a tiny fraction of the number of
objects that had been allocated, and this fraction is so small that there are typically
fewer than one survivors per MarkedBlock. This change was enough to make this change a net
win over tip-of-tree.
- FTL now shares the same allocation fast paths as everything else, which is great, because
bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
control flow, since it won't be able to improve the machine code we write ourselves. GC
fast paths are best written in assembly. So, I've empowered B3 to have even better support
for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
ability to constant-fold some allocator logic (in case we first assume that we're doing a
variable-length allocation but then realize that the length is known). I think it makes
sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
since this makes lowering easier (you can constant fold during lowering more easily) and it
reduces the amount of malloc traffic. In the future, we could teach B3 how to better
constant-fold this code. That would require allowing loads to be constant-folded, which is
doable but hella tricky.
- It used to be that if a logical object allocation required two physical allocations (first
the butterfly and then the cell), then the JIT would emit the code in such a way that a
failure in the second fast path would cause us to forget the successful first physical
allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a
register to storing either the butterfly or null, because the butterfly register is anyway
going to be free inside the first allocation. The only overhead here is zeroing the
butterfly register. With that in place, we can just pass the butterfly-or-null to the slow
path, which can then either allocate a butterfly or not. So now we never waste a successful
allocation. This patch implements such a solution both in DFG (where it's easy to do this
since we control registers already) and in FTL (where it's annoying, because mutable
"butterfly-or-null" variables are hard to say in SSA; also I realized that we had code
duplicated the JSArray allocation utility, so I deduplicated it). This came up because in
one version of this patch, this wastage would resonate with some Kraken benchmark: the
benchmark would always allocate N small things followed by one bigger thing. The problem
was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that
the JSObject size class, which both the small and big thing shared for their cell, could
hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when
it allocated the big thing. So, it would end up having to allocate the big thing's large
butterfly twice, every single time! Ouch!
- It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed
amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client
of the allocator to initialize the object to whatever state they need. It used to be that
we would just use a dumb loop. I initially changed this so that we would end up in memset
for large allocations, but this didn't actually help performance that much. I got a much
better result by playing with different memsets written in assembly. First I wrote one
using non-temporal stores. That was a small speed-up over memset. Then I tried the classic
"rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array
allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or
memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and
double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab
of stores (if the size is known), but those paths in the JIT won't trigger for
NewArrayWithSize() if the size is large, since that takes us to the
operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the
optimizations here are all in JSArray::create() - that's the hot place for large arrays
that need to be filled with holes.
All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1%
regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is
because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a
best-case scenario for bump allocation. I think that we should fix bmalloc to make up the
difference, but take the hit for now because it's a crazy corner case. By comparison, the
alternative approach of using a copy barrier would have cost us 1-2%. That's the real
apples-to-apples comparison if your premise is that we should have a concurrent GC. After we
finish removing copied space, we will be barrier-ready for concurrent GC: we already have a
marking barrier and we simply won't need a copying barrier. This change gets us there for
the purposes of our benchmarks, since the remaining clients of copied space are not very
important. On the other hand, if we keep copying, then getting barrier-ready would mean
adding back the copy barrier, which costs more perf.
We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
typed arrays and a few other weird things over to Aux MarkedSpace.
This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
and CellContainer meant that I had to include those files from everywhere. Fortunately,
just including JSCInlines.h (instead of manually including the files that includes) is
usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
that we were already basically doing. In places where JSCInlines.h would be too much, I just
included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
meant having to manually include HeapInlines.h from the places that previously got it
implicitly via JSObject.h. But that led to more problems for some reason: I started getting
build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
made it impossible to include it from outside JSC. This was a lot of work, but it was
necessary to get the patch to build on all ports. It's also a net win. There were many places
in WebCore that were transitively including a *ton* of JSC headers just because of the
JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
(for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
* API/JSManagedValue.mm:
(-[JSManagedValue initWithValue:]):
* API/JSTypedArray.cpp:
* API/ObjCCallbackFunction.mm:
* API/tests/testapi.mm:
(testObjectiveCAPI):
(testWeakValue): Deleted.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* Scripts/builtins/builtins_generate_combined_implementation.py:
(BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
(BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
* Scripts/builtins/builtins_generate_separate_implementation.py:
(BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
* assembler/AbstractMacroAssembler.h:
(JSC::AbstractMacroAssembler::JumpList::link):
(JSC::AbstractMacroAssembler::JumpList::linkTo):
* assembler/MacroAssembler.h:
* assembler/MacroAssemblerARM64.h:
(JSC::MacroAssemblerARM64::add32):
* assembler/MacroAssemblerCodeRef.cpp: Added.
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr):
(JSC::MacroAssemblerCodePtr::dumpWithName):
(JSC::MacroAssemblerCodePtr::dump):
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef):
(JSC::MacroAssemblerCodeRef::dump):
* assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted.
(JSC::MacroAssemblerCodePtr::dumpWithName): Deleted.
(JSC::MacroAssemblerCodePtr::dump): Deleted.
(JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted.
(JSC::MacroAssemblerCodeRef::dump): Deleted.
* b3/B3BasicBlock.cpp:
(JSC::B3::BasicBlock::appendBoolConstant):
* b3/B3BasicBlock.h:
* b3/B3DuplicateTails.cpp:
* b3/B3StackmapGenerationParams.h:
* b3/testb3.cpp:
(JSC::B3::testPatchpointTerminalReturnValue):
(JSC::B3::run):
* bindings/ScriptValue.cpp:
* bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
* bytecode/BytecodeBasicBlock.cpp:
* bytecode/BytecodeLivenessAnalysis.cpp:
* bytecode/BytecodeUseDef.h:
* bytecode/CallLinkInfo.cpp:
(JSC::CallLinkInfo::callTypeFor):
* bytecode/CallLinkInfo.h:
(JSC::CallLinkInfo::callTypeFor): Deleted.
* bytecode/CallLinkStatus.cpp:
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::clearLLIntGetByIdCache):
(JSC::CodeBlock::predictedMachineCodeSize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::jitCodeMap): Deleted.
(JSC::clearLLIntGetByIdCache): Deleted.
* bytecode/ExecutionCounter.h:
* bytecode/Instruction.h:
* bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
(JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal):
* bytecode/ObjectAllocationProfile.h:
(JSC::ObjectAllocationProfile::isNull):
(JSC::ObjectAllocationProfile::initialize):
* bytecode/Opcode.h:
(JSC::padOpcodeName):
* bytecode/PolymorphicAccess.cpp:
(JSC::AccessCase::generateImpl):
(JSC::PolymorphicAccess::regenerate):
* bytecode/PolymorphicAccess.h:
* bytecode/PreciseJumpTargets.cpp:
* bytecode/StructureStubInfo.cpp:
* bytecode/StructureStubInfo.h:
* bytecode/UnlinkedCodeBlock.cpp:
(JSC::UnlinkedCodeBlock::vm): Deleted.
* bytecode/UnlinkedCodeBlock.h:
* bytecode/UnlinkedInstructionStream.cpp:
* bytecode/UnlinkedInstructionStream.h:
* dfg/DFGOperations.cpp:
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileMakeRope):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
(JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
(JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* ftl/FTLAbstractHeapRepository.h:
* ftl/FTLCompile.cpp:
* ftl/FTLJITFinalizer.cpp:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
(JSC::FTL::DFG::LowerDFGToB3::compileCreateRest):
(JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
(JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateCell):
(JSC::FTL::DFG::LowerDFGToB3::allocateObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
(JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted.
* ftl/FTLOutput.cpp:
(JSC::FTL::Output::constBool):
(JSC::FTL::Output::add):
(JSC::FTL::Output::shl):
(JSC::FTL::Output::aShr):
(JSC::FTL::Output::lShr):
(JSC::FTL::Output::zeroExt):
(JSC::FTL::Output::equal):
(JSC::FTL::Output::notEqual):
(JSC::FTL::Output::above):
(JSC::FTL::Output::aboveOrEqual):
(JSC::FTL::Output::below):
(JSC::FTL::Output::belowOrEqual):
(JSC::FTL::Output::greaterThan):
(JSC::FTL::Output::greaterThanOrEqual):
(JSC::FTL::Output::lessThan):
(JSC::FTL::Output::lessThanOrEqual):
(JSC::FTL::Output::select):
(JSC::FTL::Output::appendSuccessor):
(JSC::FTL::Output::addIncomingToPhi):
* ftl/FTLOutput.h:
* ftl/FTLValueFromBlock.h:
(JSC::FTL::ValueFromBlock::operator bool):
(JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted.
* ftl/FTLWeightedTarget.h:
(JSC::FTL::WeightedTarget::frequentedBlock):
* heap/CellContainer.h: Added.
(JSC::CellContainer::CellContainer):
(JSC::CellContainer::operator bool):
(JSC::CellContainer::isMarkedBlock):
(JSC::CellContainer::isLargeAllocation):
(JSC::CellContainer::markedBlock):
(JSC::CellContainer::largeAllocation):
* heap/CellContainerInlines.h: Added.
(JSC::CellContainer::isMarked):
(JSC::CellContainer::isMarkedOrNewlyAllocated):
(JSC::CellContainer::noteMarked):
(JSC::CellContainer::cellSize):
(JSC::CellContainer::weakSet):
(JSC::CellContainer::flipIfNecessary):
* heap/ConservativeRoots.cpp:
(JSC::ConservativeRoots::ConservativeRoots):
(JSC::ConservativeRoots::~ConservativeRoots):
(JSC::ConservativeRoots::grow):
(JSC::ConservativeRoots::genericAddPointer):
(JSC::ConservativeRoots::genericAddSpan):
* heap/ConservativeRoots.h:
(JSC::ConservativeRoots::roots):
* heap/CopyToken.h:
* heap/FreeList.cpp: Added.
(JSC::FreeList::dump):
* heap/FreeList.h: Added.
(JSC::FreeList::FreeList):
(JSC::FreeList::list):
(JSC::FreeList::bump):
(JSC::FreeList::operator==):
(JSC::FreeList::operator!=):
(JSC::FreeList::operator bool):
(JSC::FreeList::allocationWillFail):
(JSC::FreeList::allocationWillSucceed):
* heap/GCTypeMap.h: Added.
(JSC::GCTypeMap::operator[]):
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalizeUnconditionalFinalizers):
(JSC::Heap::markRoots):
(JSC::Heap::copyBackingStores):
(JSC::Heap::gatherStackRoots):
(JSC::Heap::gatherJSStackRoots):
(JSC::Heap::gatherScratchBufferRoots):
(JSC::Heap::clearLivenessData):
(JSC::Heap::visitSmallStrings):
(JSC::Heap::visitConservativeRoots):
(JSC::Heap::removeDeadCompilerWorklistEntries):
(JSC::Heap::gatherExtraHeapSnapshotData):
(JSC::Heap::removeDeadHeapSnapshotNodes):
(JSC::Heap::visitProtectedObjects):
(JSC::Heap::visitArgumentBuffers):
(JSC::Heap::visitException):
(JSC::Heap::visitStrongHandles):
(JSC::Heap::visitHandleStack):
(JSC::Heap::visitSamplingProfiler):
(JSC::Heap::traceCodeBlocksAndJITStubRoutines):
(JSC::Heap::converge):
(JSC::Heap::visitWeakHandles):
(JSC::Heap::updateObjectCounts):
(JSC::Heap::clearUnmarkedExecutables):
(JSC::Heap::deleteUnmarkedCompiledCode):
(JSC::Heap::collectAllGarbage):
(JSC::Heap::collect):
(JSC::Heap::collectWithoutAnySweep):
(JSC::Heap::collectImpl):
(JSC::Heap::suspendCompilerThreads):
(JSC::Heap::willStartCollection):
(JSC::Heap::flushOldStructureIDTables):
(JSC::Heap::flushWriteBarrierBuffer):
(JSC::Heap::stopAllocation):
(JSC::Heap::prepareForMarking):
(JSC::Heap::reapWeakHandles):
(JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
(JSC::Heap::sweepArrayBuffers):
(JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor):
(JSC::MarkedBlockSnapshotFunctor::operator()):
(JSC::Heap::snapshotMarkedSpace):
(JSC::Heap::deleteSourceProviderCaches):
(JSC::Heap::notifyIncrementalSweeper):
(JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
(JSC::Heap::resetAllocators):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::didFinishCollection):
(JSC::Heap::resumeCompilerThreads):
(JSC::Zombify::visit):
(JSC::Heap::forEachCodeBlockImpl):
* heap/Heap.h:
(JSC::Heap::allocatorForObjectWithoutDestructor):
(JSC::Heap::allocatorForObjectWithDestructor):
(JSC::Heap::allocatorForAuxiliaryData):
(JSC::Heap::jitStubRoutines):
(JSC::Heap::codeBlockSet):
(JSC::Heap::storageAllocator): Deleted.
* heap/HeapCell.h:
(JSC::HeapCell::isZapped): Deleted.
* heap/HeapCellInlines.h: Added.
(JSC::HeapCell::isLargeAllocation):
(JSC::HeapCell::cellContainer):
(JSC::HeapCell::markedBlock):
(JSC::HeapCell::largeAllocation):
(JSC::HeapCell::heap):
(JSC::HeapCell::vm):
(JSC::HeapCell::cellSize):
(JSC::HeapCell::allocatorAttributes):
(JSC::HeapCell::destructionMode):
(JSC::HeapCell::cellKind):
* heap/HeapInlines.h:
(JSC::Heap::heap):
(JSC::Heap::isLive):
(JSC::Heap::isMarked):
(JSC::Heap::testAndSetMarked):
(JSC::Heap::setMarked):
(JSC::Heap::cellSize):
(JSC::Heap::forEachCodeBlock):
(JSC::Heap::allocateObjectOfType):
(JSC::Heap::subspaceForObjectOfType):
(JSC::Heap::allocatorForObjectOfType):
(JSC::Heap::allocateAuxiliary):
(JSC::Heap::tryAllocateAuxiliary):
(JSC::Heap::tryReallocateAuxiliary):
(JSC::Heap::isPointerGCObject): Deleted.
(JSC::Heap::isValueGCObject): Deleted.
* heap/HeapOperation.cpp: Added.
(WTF::printInternal):
* heap/HeapOperation.h:
* heap/HeapUtil.h: Added.
(JSC::HeapUtil::findGCObjectPointersForMarking):
(JSC::HeapUtil::isPointerGCObjectJSCell):
(JSC::HeapUtil::isValueGCObject):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::sweepNextBlock):
* heap/IncrementalSweeper.h:
* heap/LargeAllocation.cpp: Added.
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::LargeAllocation):
(JSC::LargeAllocation::lastChanceToFinalize):
(JSC::LargeAllocation::shrink):
(JSC::LargeAllocation::visitWeakSet):
(JSC::LargeAllocation::reapWeakSet):
(JSC::LargeAllocation::flip):
(JSC::LargeAllocation::isEmpty):
(JSC::LargeAllocation::sweep):
(JSC::LargeAllocation::destroy):
(JSC::LargeAllocation::dump):
* heap/LargeAllocation.h: Added.
(JSC::LargeAllocation::fromCell):
(JSC::LargeAllocation::cell):
(JSC::LargeAllocation::isLargeAllocation):
(JSC::LargeAllocation::heap):
(JSC::LargeAllocation::vm):
(JSC::LargeAllocation::weakSet):
(JSC::LargeAllocation::clearNewlyAllocated):
(JSC::LargeAllocation::isNewlyAllocated):
(JSC::LargeAllocation::isMarked):
(JSC::LargeAllocation::isMarkedOrNewlyAllocated):
(JSC::LargeAllocation::isLive):
(JSC::LargeAllocation::hasValidCell):
(JSC::LargeAllocation::cellSize):
(JSC::LargeAllocation::aboveLowerBound):
(JSC::LargeAllocation::belowUpperBound):
(JSC::LargeAllocation::contains):
(JSC::LargeAllocation::attributes):
(JSC::LargeAllocation::flipIfNecessary):
(JSC::LargeAllocation::flipIfNecessaryConcurrently):
(JSC::LargeAllocation::testAndSetMarked):
(JSC::LargeAllocation::setMarked):
(JSC::LargeAllocation::clearMarked):
(JSC::LargeAllocation::noteMarked):
(JSC::LargeAllocation::headerSize):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::MarkedAllocator):
(JSC::MarkedAllocator::isPagedOut):
(JSC::MarkedAllocator::retire):
(JSC::MarkedAllocator::filterNextBlock):
(JSC::MarkedAllocator::setNextBlockToSweep):
(JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::allocateSlowCase):
(JSC::MarkedAllocator::tryAllocateSlowCase):
(JSC::MarkedAllocator::allocateSlowCaseImpl):
(JSC::blockHeaderSize):
(JSC::MarkedAllocator::blockSizeForBytes):
(JSC::MarkedAllocator::tryAllocateBlock):
(JSC::MarkedAllocator::addBlock):
(JSC::MarkedAllocator::removeBlock):
(JSC::MarkedAllocator::stopAllocating):
(JSC::MarkedAllocator::reset):
(JSC::MarkedAllocator::lastChanceToFinalize):
(JSC::MarkedAllocator::setFreeList):
(JSC::isListPagedOut): Deleted.
(JSC::MarkedAllocator::tryAllocateHelper): Deleted.
(JSC::MarkedAllocator::tryPopFreeList): Deleted.
(JSC::MarkedAllocator::tryAllocate): Deleted.
(JSC::MarkedAllocator::allocateBlock): Deleted.
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::takeLastActiveBlock):
(JSC::MarkedAllocator::offsetOfFreeList):
(JSC::MarkedAllocator::offsetOfCellSize):
(JSC::MarkedAllocator::tryAllocate):
(JSC::MarkedAllocator::allocate):
(JSC::MarkedAllocator::forEachBlock):
(JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
(JSC::MarkedAllocator::MarkedAllocator): Deleted.
(JSC::MarkedAllocator::init): Deleted.
(JSC::MarkedAllocator::stopAllocating): Deleted.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::MarkedBlock):
(JSC::MarkedBlock::Handle::specializedSweep):
(JSC::MarkedBlock::Handle::sweep):
(JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode):
(JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode):
(JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated):
(JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor):
(JSC::SetNewlyAllocatedFunctor::operator()):
(JSC::MarkedBlock::Handle::stopAllocating):
(JSC::MarkedBlock::Handle::lastChanceToFinalize):
(JSC::MarkedBlock::Handle::resumeAllocating):
(JSC::MarkedBlock::Handle::zap):
(JSC::MarkedBlock::Handle::forEachFreeCell):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessarySlow):
(JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow):
(JSC::MarkedBlock::clearMarks):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::needsFlip):
(JSC::MarkedBlock::Handle::needsFlip):
(JSC::MarkedBlock::Handle::willRemoveBlock):
(JSC::MarkedBlock::Handle::didConsumeFreeList):
(JSC::MarkedBlock::markCount):
(JSC::MarkedBlock::Handle::isEmpty):
(JSC::MarkedBlock::clearHasAnyMarked):
(JSC::MarkedBlock::noteMarkedSlow):
(WTF::printInternal):
(JSC::MarkedBlock::create): Deleted.
(JSC::MarkedBlock::destroy): Deleted.
(JSC::MarkedBlock::callDestructor): Deleted.
(JSC::MarkedBlock::specializedSweep): Deleted.
(JSC::MarkedBlock::sweep): Deleted.
(JSC::MarkedBlock::sweepHelper): Deleted.
(JSC::MarkedBlock::stopAllocating): Deleted.
(JSC::MarkedBlock::clearMarksWithCollectionType): Deleted.
(JSC::MarkedBlock::lastChanceToFinalize): Deleted.
(JSC::MarkedBlock::resumeAllocating): Deleted.
(JSC::MarkedBlock::didRetireBlock): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::VoidFunctor::returnValue):
(JSC::MarkedBlock::CountFunctor::CountFunctor):
(JSC::MarkedBlock::CountFunctor::count):
(JSC::MarkedBlock::CountFunctor::returnValue):
(JSC::MarkedBlock::Handle::hasAnyNewlyAllocated):
(JSC::MarkedBlock::Handle::isOnBlocksToSweep):
(JSC::MarkedBlock::Handle::setIsOnBlocksToSweep):
(JSC::MarkedBlock::Handle::state):
(JSC::MarkedBlock::needsDestruction):
(JSC::MarkedBlock::handle):
(JSC::MarkedBlock::Handle::block):
(JSC::MarkedBlock::firstAtom):
(JSC::MarkedBlock::atoms):
(JSC::MarkedBlock::isAtomAligned):
(JSC::MarkedBlock::Handle::cellAlign):
(JSC::MarkedBlock::blockFor):
(JSC::MarkedBlock::Handle::allocator):
(JSC::MarkedBlock::Handle::heap):
(JSC::MarkedBlock::Handle::vm):
(JSC::MarkedBlock::vm):
(JSC::MarkedBlock::Handle::weakSet):
(JSC::MarkedBlock::weakSet):
(JSC::MarkedBlock::Handle::shrink):
(JSC::MarkedBlock::Handle::visitWeakSet):
(JSC::MarkedBlock::Handle::reapWeakSet):
(JSC::MarkedBlock::Handle::cellSize):
(JSC::MarkedBlock::cellSize):
(JSC::MarkedBlock::Handle::attributes):
(JSC::MarkedBlock::attributes):
(JSC::MarkedBlock::Handle::needsDestruction):
(JSC::MarkedBlock::Handle::destruction):
(JSC::MarkedBlock::Handle::cellKind):
(JSC::MarkedBlock::Handle::markCount):
(JSC::MarkedBlock::Handle::size):
(JSC::MarkedBlock::atomNumber):
(JSC::MarkedBlock::flipIfNecessary):
(JSC::MarkedBlock::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipIfNecessary):
(JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently):
(JSC::MarkedBlock::Handle::flipForEdenCollection):
(JSC::MarkedBlock::assertFlipped):
(JSC::MarkedBlock::Handle::assertFlipped):
(JSC::MarkedBlock::isMarked):
(JSC::MarkedBlock::testAndSetMarked):
(JSC::MarkedBlock::Handle::isNewlyAllocated):
(JSC::MarkedBlock::Handle::setNewlyAllocated):
(JSC::MarkedBlock::Handle::clearNewlyAllocated):
(JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::isMarkedOrNewlyAllocated):
(JSC::MarkedBlock::Handle::isLive):
(JSC::MarkedBlock::isAtom):
(JSC::MarkedBlock::Handle::isLiveCell):
(JSC::MarkedBlock::Handle::forEachCell):
(JSC::MarkedBlock::Handle::forEachLiveCell):
(JSC::MarkedBlock::Handle::forEachDeadCell):
(JSC::MarkedBlock::Handle::needsSweeping):
(JSC::MarkedBlock::Handle::isAllocated):
(JSC::MarkedBlock::Handle::isMarked):
(JSC::MarkedBlock::Handle::isFreeListed):
(JSC::MarkedBlock::hasAnyMarked):
(JSC::MarkedBlock::noteMarked):
(WTF::MarkedBlockHash::hash):
(JSC::MarkedBlock::FreeList::FreeList): Deleted.
(JSC::MarkedBlock::allocator): Deleted.
(JSC::MarkedBlock::heap): Deleted.
(JSC::MarkedBlock::shrink): Deleted.
(JSC::MarkedBlock::visitWeakSet): Deleted.
(JSC::MarkedBlock::reapWeakSet): Deleted.
(JSC::MarkedBlock::willRemoveBlock): Deleted.
(JSC::MarkedBlock::didConsumeFreeList): Deleted.
(JSC::MarkedBlock::markCount): Deleted.
(JSC::MarkedBlock::isEmpty): Deleted.
(JSC::MarkedBlock::destruction): Deleted.
(JSC::MarkedBlock::cellKind): Deleted.
(JSC::MarkedBlock::size): Deleted.
(JSC::MarkedBlock::capacity): Deleted.
(JSC::MarkedBlock::setMarked): Deleted.
(JSC::MarkedBlock::clearMarked): Deleted.
(JSC::MarkedBlock::isNewlyAllocated): Deleted.
(JSC::MarkedBlock::setNewlyAllocated): Deleted.
(JSC::MarkedBlock::clearNewlyAllocated): Deleted.
(JSC::MarkedBlock::isLive): Deleted.
(JSC::MarkedBlock::isLiveCell): Deleted.
(JSC::MarkedBlock::forEachCell): Deleted.
(JSC::MarkedBlock::forEachLiveCell): Deleted.
(JSC::MarkedBlock::forEachDeadCell): Deleted.
(JSC::MarkedBlock::needsSweeping): Deleted.
(JSC::MarkedBlock::isAllocated): Deleted.
(JSC::MarkedBlock::isMarkedOrRetired): Deleted.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::initializeSizeClassForStepSize):
(JSC::MarkedSpace::MarkedSpace):
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::lastChanceToFinalize):
(JSC::MarkedSpace::allocate):
(JSC::MarkedSpace::tryAllocate):
(JSC::MarkedSpace::allocateLarge):
(JSC::MarkedSpace::tryAllocateLarge):
(JSC::MarkedSpace::sweep):
(JSC::MarkedSpace::sweepLargeAllocations):
(JSC::MarkedSpace::zombifySweep):
(JSC::MarkedSpace::resetAllocators):
(JSC::MarkedSpace::visitWeakSets):
(JSC::MarkedSpace::reapWeakSets):
(JSC::MarkedSpace::stopAllocating):
(JSC::MarkedSpace::prepareForMarking):
(JSC::MarkedSpace::resumeAllocating):
(JSC::MarkedSpace::isPagedOut):
(JSC::MarkedSpace::freeBlock):
(JSC::MarkedSpace::freeOrShrinkBlock):
(JSC::MarkedSpace::shrink):
(JSC::MarkedSpace::clearNewlyAllocated):
(JSC::VerifyMarked::operator()):
(JSC::MarkedSpace::flip):
(JSC::MarkedSpace::objectCount):
(JSC::MarkedSpace::size):
(JSC::MarkedSpace::capacity):
(JSC::MarkedSpace::addActiveWeakSet):
(JSC::MarkedSpace::didAddBlock):
(JSC::MarkedSpace::didAllocateInBlock):
(JSC::MarkedSpace::forEachAllocator): Deleted.
(JSC::VerifyMarkedOrRetired::operator()): Deleted.
(JSC::MarkedSpace::clearMarks): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::sizeClassToIndex):
(JSC::MarkedSpace::indexToSizeClass):
(JSC::MarkedSpace::version):
(JSC::MarkedSpace::blocksWithNewObjects):
(JSC::MarkedSpace::largeAllocations):
(JSC::MarkedSpace::largeAllocationsNurseryOffset):
(JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
(JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
(JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
(JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
(JSC::MarkedSpace::forEachLiveCell):
(JSC::MarkedSpace::forEachDeadCell):
(JSC::MarkedSpace::allocatorFor):
(JSC::MarkedSpace::destructorAllocatorFor):
(JSC::MarkedSpace::auxiliaryAllocatorFor):
(JSC::MarkedSpace::allocateWithoutDestructor):
(JSC::MarkedSpace::allocateWithDestructor):
(JSC::MarkedSpace::allocateAuxiliary):
(JSC::MarkedSpace::tryAllocateAuxiliary):
(JSC::MarkedSpace::forEachBlock):
(JSC::MarkedSpace::forEachAllocator):
(JSC::MarkedSpace::optimalSizeFor):
(JSC::MarkedSpace::didAddBlock): Deleted.
(JSC::MarkedSpace::didAllocateInBlock): Deleted.
(JSC::MarkedSpace::objectCount): Deleted.
(JSC::MarkedSpace::size): Deleted.
(JSC::MarkedSpace::capacity): Deleted.
* heap/SlotVisitor.cpp:
(JSC::SlotVisitor::SlotVisitor):
(JSC::SlotVisitor::didStartMarking):
(JSC::SlotVisitor::reset):
(JSC::SlotVisitor::append):
(JSC::SlotVisitor::appendJSCellOrAuxiliary):
(JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
(JSC::SlotVisitor::appendToMarkStack):
(JSC::SlotVisitor::markAuxiliary):
(JSC::SlotVisitor::noteLiveAuxiliaryCell):
(JSC::SlotVisitor::visitChildren):
* heap/SlotVisitor.h:
* heap/WeakBlock.cpp:
(JSC::WeakBlock::create):
(JSC::WeakBlock::WeakBlock):
(JSC::WeakBlock::visit):
(JSC::WeakBlock::reap):
* heap/WeakBlock.h:
(JSC::WeakBlock::disconnectContainer):
(JSC::WeakBlock::disconnectMarkedBlock): Deleted.
* heap/WeakSet.cpp:
(JSC::WeakSet::~WeakSet):
(JSC::WeakSet::sweep):
(JSC::WeakSet::shrink):
(JSC::WeakSet::addAllocator):
* heap/WeakSet.h:
(JSC::WeakSet::container):
(JSC::WeakSet::setContainer):
(JSC::WeakSet::WeakSet):
(JSC::WeakSet::visit):
(JSC::WeakSet::shrink): Deleted.
* heap/WeakSetInlines.h:
(JSC::WeakSet::allocate):
* inspector/InjectedScriptManager.cpp:
* inspector/JSGlobalObjectInspectorController.cpp:
* inspector/JSJavaScriptCallFrame.cpp:
* inspector/ScriptDebugServer.cpp:
* inspector/agents/InspectorDebuggerAgent.cpp:
* interpreter/CachedCall.h:
(JSC::CachedCall::CachedCall):
* interpreter/Interpreter.cpp:
(JSC::loadVarargs):
(JSC::StackFrame::sourceID): Deleted.
(JSC::StackFrame::sourceURL): Deleted.
(JSC::StackFrame::functionName): Deleted.
(JSC::StackFrame::computeLineAndColumn): Deleted.
(JSC::StackFrame::toString): Deleted.
* interpreter/Interpreter.h:
(JSC::StackFrame::isNative): Deleted.
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
(JSC::AssemblyHelpers::emitAllocate):
(JSC::AssemblyHelpers::emitAllocateJSCell):
(JSC::AssemblyHelpers::emitAllocateJSObject):
(JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
(JSC::AssemblyHelpers::emitAllocateVariableSized):
* jit/GCAwareJITStubRoutine.cpp:
(JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine):
* jit/JIT.cpp:
(JSC::JIT::compileCTINativeCall):
(JSC::JIT::link):
* jit/JIT.h:
(JSC::JIT::compileCTINativeCall): Deleted.
* jit/JITExceptions.cpp:
(JSC::genericUnwind):
* jit/JITExceptions.h:
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emitSlow_op_create_this):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitWriteBarrier):
* jit/JITThunks.cpp:
* jit/JITThunks.h:
* jsc.cpp:
(functionDescribeArray):
(main):
* llint/LLIntData.cpp:
(JSC::LLInt::Data::performAssertions):
* llint/LLIntExceptions.cpp:
* llint/LLIntThunks.cpp:
* llint/LLIntThunks.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter.cpp:
* llint/LowLevelInterpreter32_64.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ModuleAnalyzer.cpp:
* parser/NodeConstructors.h:
* parser/Nodes.h:
* profiler/ProfilerBytecode.cpp:
* profiler/ProfilerBytecode.h:
* profiler/ProfilerBytecodeSequence.cpp:
* runtime/ArrayConventions.h:
(JSC::indexingHeaderForArrayStorage):
(JSC::baseIndexingHeaderForArrayStorage):
(JSC::indexingHeaderForArray): Deleted.
(JSC::baseIndexingHeaderForArray): Deleted.
* runtime/ArrayPrototype.cpp:
(JSC::arrayProtoFuncSplice):
(JSC::concatAppendOne):
(JSC::arrayProtoPrivateFuncConcatMemcpy):
* runtime/ArrayStorage.h:
(JSC::ArrayStorage::vectorLength):
(JSC::ArrayStorage::totalSizeFor):
(JSC::ArrayStorage::totalSize):
(JSC::ArrayStorage::availableVectorLength):
(JSC::ArrayStorage::optimalVectorLength):
(JSC::ArrayStorage::sizeFor): Deleted.
* runtime/AuxiliaryBarrier.h: Added.
(JSC::AuxiliaryBarrier::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier::clear):
(JSC::AuxiliaryBarrier::get):
(JSC::AuxiliaryBarrier::slot):
(JSC::AuxiliaryBarrier::operator bool):
(JSC::AuxiliaryBarrier::setWithoutBarrier):
* runtime/AuxiliaryBarrierInlines.h: Added.
(JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
(JSC::AuxiliaryBarrier<T>::set):
* runtime/Butterfly.h:
* runtime/ButterflyInlines.h:
(JSC::Butterfly::availableContiguousVectorLength):
(JSC::Butterfly::optimalContiguousVectorLength):
(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::growArrayRight):
* runtime/ClonedArguments.cpp:
(JSC::ClonedArguments::createEmpty):
* runtime/CommonSlowPathsExceptions.cpp:
* runtime/CommonSlowPathsExceptions.h:
* runtime/DataView.cpp:
* runtime/DirectArguments.h:
* runtime/ECMAScriptSpecInternalFunctions.cpp:
* runtime/Error.cpp:
* runtime/Error.h:
* runtime/ErrorInstance.cpp:
* runtime/ErrorInstance.h:
* runtime/Exception.cpp:
* runtime/Exception.h:
* runtime/GeneratorFrame.cpp:
* runtime/GeneratorPrototype.cpp:
* runtime/InternalFunction.cpp:
(JSC::InternalFunction::InternalFunction):
* runtime/IntlCollator.cpp:
* runtime/IntlCollatorConstructor.cpp:
* runtime/IntlCollatorPrototype.cpp:
* runtime/IntlDateTimeFormat.cpp:
* runtime/IntlDateTimeFormatConstructor.cpp:
* runtime/IntlDateTimeFormatPrototype.cpp:
* runtime/IntlNumberFormat.cpp:
* runtime/IntlNumberFormatConstructor.cpp:
* runtime/IntlNumberFormatPrototype.cpp:
* runtime/IntlObject.cpp:
* runtime/IteratorPrototype.cpp:
* runtime/JSArray.cpp:
(JSC::JSArray::tryCreateUninitialized):
(JSC::JSArray::setLengthWritable):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLengthWithArrayStorage):
(JSC::JSArray::appendMemcpy):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithArrayStorage):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):
* runtime/JSArray.h:
(JSC::createContiguousArrayButterfly):
(JSC::createArrayButterfly):
(JSC::JSArray::create):
(JSC::JSArray::tryCreateUninitialized): Deleted.
* runtime/JSArrayBufferView.h:
* runtime/JSCInlines.h:
* runtime/JSCJSValue.cpp:
(JSC::JSValue::dumpInContextAssumingStructure):
* runtime/JSCallee.cpp:
(JSC::JSCallee::JSCallee):
* runtime/JSCell.cpp:
(JSC::JSCell::estimatedSize):
* runtime/JSCell.h:
(JSC::JSCell::cellStateOffset): Deleted.
* runtime/JSCellInlines.h:
(JSC::ExecState::vm):
(JSC::JSCell::classInfo):
(JSC::JSCell::callDestructor):
(JSC::JSCell::vm): Deleted.
* runtime/JSFunction.cpp:
(JSC::JSFunction::create):
(JSC::JSFunction::allocateAndInitializeRareData):
(JSC::JSFunction::initializeRareData):
(JSC::JSFunction::getOwnPropertySlot):
(JSC::JSFunction::put):
(JSC::JSFunction::deleteProperty):
(JSC::JSFunction::defineOwnProperty):
(JSC::JSFunction::setFunctionName):
(JSC::JSFunction::reifyLength):
(JSC::JSFunction::reifyName):
(JSC::JSFunction::reifyLazyPropertyIfNeeded):
(JSC::JSFunction::reifyBoundNameIfNeeded):
* runtime/JSFunction.h:
* runtime/JSFunctionInlines.h:
(JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
(JSC::JSFunction::JSFunction):
* runtime/JSGenericTypedArrayViewInlines.h:
(JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
* runtime/JSInternalPromise.cpp:
* runtime/JSInternalPromiseConstructor.cpp:
* runtime/JSInternalPromiseDeferred.cpp:
* runtime/JSInternalPromisePrototype.cpp:
* runtime/JSJob.cpp:
* runtime/JSMapIterator.cpp:
* runtime/JSModuleNamespaceObject.cpp:
* runtime/JSModuleRecord.cpp:
* runtime/JSObject.cpp:
(JSC::JSObject::visitButterfly):
(JSC::JSObject::notifyPresenceOfIndexedAccessors):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createInitialUndecided):
(JSC::JSObject::createInitialInt32):
(JSC::JSObject::createInitialDouble):
(JSC::JSObject::createInitialContiguous):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::createInitialArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::putByIndexBeyondVectorLength):
(JSC::JSObject::putDirectIndexBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::increaseVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::growOutOfLineStorage):
(JSC::JSObject::copyButterfly): Deleted.
(JSC::JSObject::copyBackingStore): Deleted.
* runtime/JSObject.h:
(JSC::JSObject::globalObject):
(JSC::JSObject::putDirectInternal):
(JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
* runtime/JSObjectInlines.h:
* runtime/JSPromise.cpp:
* runtime/JSPromiseConstructor.cpp:
* runtime/JSPromiseDeferred.cpp:
* runtime/JSPromisePrototype.cpp:
* runtime/JSPropertyNameIterator.cpp:
* runtime/JSScope.cpp:
(JSC::JSScope::resolve):
* runtime/JSScope.h:
(JSC::JSScope::globalObject):
(JSC::JSScope::vm): Deleted.
* runtime/JSSetIterator.cpp:
* runtime/JSStringIterator.cpp:
* runtime/JSTemplateRegistryKey.cpp:
* runtime/JSTypedArrayViewConstructor.cpp:
* runtime/JSTypedArrayViewPrototype.cpp:
* runtime/JSWeakMap.cpp:
* runtime/JSWeakSet.cpp:
* runtime/MapConstructor.cpp:
* runtime/MapIteratorPrototype.cpp:
* runtime/MapPrototype.cpp:
* runtime/NativeErrorConstructor.cpp:
* runtime/NativeStdFunctionCell.cpp:
* runtime/Operations.h:
(JSC::scribbleFreeCells):
(JSC::scribble):
* runtime/Options.h:
* runtime/PropertyTable.cpp:
* runtime/ProxyConstructor.cpp:
* runtime/ProxyObject.cpp:
* runtime/ProxyRevoke.cpp:
* runtime/RegExp.cpp:
(JSC::RegExp::match):
(JSC::RegExp::matchConcurrently):
(JSC::RegExp::matchCompareWithInterpreter):
* runtime/RegExp.h:
* runtime/RegExpConstructor.h:
* runtime/RegExpInlines.h:
(JSC::RegExp::matchInline):
* runtime/RegExpMatchesArray.h:
(JSC::tryCreateUninitializedRegExpMatchesArray):
(JSC::createRegExpMatchesArray):
* runtime/RegExpPrototype.cpp:
(JSC::genericSplit):
* runtime/RuntimeType.cpp:
* runtime/SamplingProfiler.cpp:
(JSC::SamplingProfiler::processUnverifiedStackTraces):
* runtime/SetConstructor.cpp:
* runtime/SetIteratorPrototype.cpp:
* runtime/SetPrototype.cpp:
* runtime/StackFrame.cpp: Added.
(JSC::StackFrame::sourceID):
(JSC::StackFrame::sourceURL):
(JSC::StackFrame::functionName):
(JSC::StackFrame::computeLineAndColumn):
(JSC::StackFrame::toString):
* runtime/StackFrame.h: Added.
(JSC::StackFrame::isNative):
* runtime/StringConstructor.cpp:
* runtime/StringIteratorPrototype.cpp:
* runtime/StructureInlines.h:
(JSC::Structure::propertyTable):
* runtime/TemplateRegistry.cpp:
* runtime/TestRunnerUtils.cpp:
(JSC::finalizeStatsAtEndOfTesting):
* runtime/TestRunnerUtils.h:
* runtime/TypeProfilerLog.cpp:
* runtime/TypeSet.cpp:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::ensureStackCapacityForCLoop):
(JSC::VM::isSafeToRecurseSoftCLoop):
* runtime/VM.h:
* runtime/VMEntryScope.h:
* runtime/VMInlines.h:
(JSC::VM::ensureStackCapacityFor):
(JSC::VM::isSafeToRecurseSoft):
* runtime/WeakMapConstructor.cpp:
* runtime/WeakMapData.cpp:
* runtime/WeakMapPrototype.cpp:
* runtime/WeakSetConstructor.cpp:
* runtime/WeakSetPrototype.cpp:
* testRegExp.cpp:
(testOneRegExp):
* tools/JSDollarVM.cpp:
* tools/JSDollarVMPrototype.cpp:
(JSC::JSDollarVMPrototype::isInObjectSpace):
Source/WebCore:
No new tests because no new WebCore behavior.
Just rewiring #includes.
* ForwardingHeaders/heap/HeapInlines.h: Added.
* ForwardingHeaders/interpreter/Interpreter.h: Removed.
* ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
* Modules/indexeddb/IDBCursorWithValue.cpp:
* Modules/indexeddb/client/TransactionOperation.cpp:
* Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
* Modules/indexeddb/server/UniqueIDBDatabase.cpp:
* bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
* bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
* bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
* bindings/js/JSClientRectCustom.cpp:
* bindings/js/JSDOMBinding.cpp:
* bindings/js/JSDOMBinding.h:
* bindings/js/JSDeviceMotionEventCustom.cpp:
* bindings/js/JSDeviceOrientationEventCustom.cpp:
* bindings/js/JSErrorEventCustom.cpp:
* bindings/js/JSIDBCursorWithValueCustom.cpp:
* bindings/js/JSIDBIndexCustom.cpp:
* bindings/js/JSPopStateEventCustom.cpp:
* bindings/js/JSWebGL2RenderingContextCustom.cpp:
* bindings/js/JSWorkerGlobalScopeCustom.cpp:
* bindings/js/WorkerScriptController.cpp:
* contentextensions/ContentExtensionParser.cpp:
* dom/ErrorEvent.cpp:
* html/HTMLCanvasElement.cpp:
* html/MediaDocument.cpp:
* inspector/CommandLineAPIModule.cpp:
* loader/EmptyClients.cpp:
* page/CaptionUserPreferences.cpp:
* page/Frame.cpp:
* page/PageGroup.cpp:
* page/UserContentController.cpp:
* platform/mock/mediasource/MockBox.cpp:
* testing/GCObservation.cpp:
Source/WebKit2:
Just rewiring some #includes.
* UIProcess/ViewGestureController.cpp:
* UIProcess/WebPageProxy.cpp:
* UIProcess/WebProcessPool.cpp:
* UIProcess/WebProcessProxy.cpp:
* WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
* WebProcess/Plugins/Netscape/JSNPObject.cpp:
Source/WTF:
I needed tryFastAlignedMalloc() so I added it.
* wtf/FastMalloc.cpp:
(WTF::tryFastAlignedMalloc):
* wtf/FastMalloc.h:
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
(WTF::ParkingLot::forEach): Deleted.
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::forEach):
* wtf/ScopedLambda.h:
(WTF::scopedLambdaRef):
* wtf/SentinelLinkedList.h:
(WTF::SentinelLinkedList::forEach):
(WTF::RawNode>::takeFrom):
* wtf/SimpleStats.h:
(WTF::SimpleStats::operator bool):
(WTF::SimpleStats::operator!): Deleted.
Tools:
* DumpRenderTree/TestRunner.cpp:
* DumpRenderTree/mac/DumpRenderTree.mm:
(DumpRenderTreeMain):
* Scripts/run-jsc-stress-tests:
* TestWebKitAPI/Tests/WTF/Vector.cpp:
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/179778@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
|
|
|
}
|
2016-04-20 04:25:02 +00:00
|
|
|
|
|
|
|
private:
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
WTF_EXPORT_PRIVATE static ParkResult parkConditionallyImpl(
|
2016-04-20 04:25:02 +00:00
|
|
|
const void* address,
|
2016-05-26 21:58:42 +00:00
|
|
|
const ScopedLambda<bool()>& validation,
|
|
|
|
const ScopedLambda<void()>& beforeSleep,
|
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows
https://bugs.webkit.org/show_bug.cgi?id=152045
Reviewed by Andy Estes.
Source/JavaScriptCore:
Probably the nicest example of why this patch is a good idea is the change in
AtomicsObject.cpp.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
* runtime/AtomicsObject.cpp:
(JSC::atomicsFuncWait):
Source/WebCore:
No new layout tests because no new behavior. The new WTF time classes have some unit tests
in TestWebKitAPI.
* fileapi/ThreadableBlobRegistry.cpp:
(WebCore::ThreadableBlobRegistry::blobSize):
* platform/MainThreadSharedTimer.h:
* platform/SharedTimer.h:
* platform/ThreadTimers.cpp:
(WebCore::ThreadTimers::updateSharedTimer):
* platform/cf/MainThreadSharedTimerCF.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/efl/MainThreadSharedTimerEfl.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/glib/MainThreadSharedTimerGLib.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* platform/win/MainThreadSharedTimerWin.cpp:
(WebCore::MainThreadSharedTimer::setFireInterval):
* workers/WorkerRunLoop.cpp:
(WebCore::WorkerRunLoop::runInMode):
Source/WebKit2:
* Platform/IPC/Connection.cpp:
(IPC::Connection::SyncMessageState::wait):
(IPC::Connection::sendMessage):
(IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting):
(IPC::Connection::waitForMessage):
(IPC::Connection::sendSyncMessage):
(IPC::Connection::waitForSyncReply):
* Platform/IPC/Connection.h:
(IPC::Connection::sendSync):
(IPC::Connection::waitForAndDispatchImmediately):
* Platform/IPC/MessageSender.h:
(IPC::MessageSender::sendSync):
* UIProcess/ChildProcessProxy.h:
(WebKit::ChildProcessProxy::sendSync):
* UIProcess/Network/NetworkProcessProxy.cpp:
(WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/Storage/StorageManager.cpp:
(WebKit::StorageManager::applicationWillTerminate):
* UIProcess/WebProcessProxy.cpp:
(WebKit::WebProcessProxy::sendProcessWillSuspendImminently):
* UIProcess/WebResourceLoadStatisticsStore.cpp:
(WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate):
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h:
* UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm:
(-[WKOneShotDisplayLinkHandler displayLinkFired:]):
(WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree):
(WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay):
(WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm:
(WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState):
* UIProcess/mac/WKImmediateActionController.mm:
(-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]):
* UIProcess/mac/WebPageProxyMac.mm:
(WebKit::WebPageProxy::stringSelectionForPasteboard):
(WebKit::WebPageProxy::dataSelectionForPasteboard):
(WebKit::WebPageProxy::readSelectionFromPasteboard):
(WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent):
(WebKit::WebPageProxy::acceptsFirstMouse):
* WebProcess/WebCoreSupport/WebChromeClient.cpp:
(WebKit::WebChromeClient::runBeforeUnloadConfirmPanel):
(WebKit::WebChromeClient::runJavaScriptAlert):
(WebKit::WebChromeClient::runJavaScriptConfirm):
(WebKit::WebChromeClient::runJavaScriptPrompt):
(WebKit::WebChromeClient::print):
(WebKit::WebChromeClient::exceededDatabaseQuota):
(WebKit::WebChromeClient::reachedApplicationCacheOriginQuota):
* WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp:
(WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse):
* WebProcess/WebPage/WebPage.cpp:
(WebKit::WebPage::postSynchronousMessageForTesting):
Source/WTF:
We used to use 'double' for all time measurements. Sometimes it was milliseconds,
sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time
since some epoch. When we spoke of time since epoch, we either used a monotonic clock or
a wall clock. The type - always 'double' - never told us what kind of time we had, even
though there were roughly six of them (sec interval, ms interval, sec since epoch on wall,
ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic).
At some point, we thought that it would be a good idea to replace these doubles with
std::chrono. But since replacing some things with std::chrono, we found it to be terribly
inconvenient:
- Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say
std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally
duration_cast would not even be a thing.
- No overflow protection. std::chrono uses integers by default and using anything else is
clumsy. But the integer math is done without regard for the rough edges of integer math,
so any cast between std::chrono types risks overflow. Any comparison risks overflow
because it may do conversions silently. We have even found bugs where some C++
implementations had more overflows than others, which ends up being a special kind of
hell. In many cases, the overflow also has nasal demons.
It's an error to represent time using integers. It would have been excusable back when
floating point math was not guaranteed to be supported on all platforms, but that would
have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit
for floating point:
- Floating point preserves precision under multiplication in all but extreme cases, so
using floating point for time means that unit conversions are almost completely
lossless. This means that we don't have to think very hard about what units to use. In
this patch, we use seconds almost everywhere. We only convert at boundaries, like an API
boundary that wants something other than seconds.
- Floating point makes it easy to reason about infinity, which is something that time code
wants to do a lot. Example: when would you like to timeout? Infinity please! This is the
most elegant way of having an API support both a timeout variant and a no-timeout
variant.
- Floating point does well-understood things when math goes wrong, and these things are
pretty well optimized to match what a mathematician would do when computing with real
numbers represented using scientific notation with a finite number of significant
digits. This means that time math under floating point looks like normal math. On the
other hand, std::chrono time math looks like garbage because you have to always check
for multiple possible UB corners whenever you touch large integers. Integers that
represent time are very likely to be large and you don't have to do much to overflow
them. At this time, based on the number of bugs we have already seen due to chrono
overflows, I am not certain that we even understand what are all of the corner cases
that we should even check for.
This patch introduces a new set of timekeeping classes that are all based on double, and
all internally use seconds. These classes support algebraic typing. The classes are:
- Seconds: this is for measuring a duration.
- WallTime: time since epoch according to a wall clock (aka real time clock).
- MonotonicTime: time since epoch according to a monotonic clock.
- ClockType: enum that says either Wall or Monotonic.
- TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a
wall time or a monotonic time.
All of these classes behave like C++ values and are cheap to copy around since they are
very nearly POD. This supports comprehensive conversions between the various time types.
Most of this is by way of algebra. Here are just some of the rules we recognize:
WallTime = WallTime + Seconds
Seconds = WallTime - WallTime
MonotonicTime = MonotonicTime + Seconds
etc...
We support negative, infinite, and NaN times because math.
We support conversions between MonotonicTime and WallTime, like:
WallTime wt = mt.approximateWallTime()
This is called this "approximate" because the only way to do it is to get the current time
on both clocks and convert relative to that.
Many of our APIs would be happy using whatever notion of time the user wanted to use. For
those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You
can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This
means that if you use a WallTime with Condition::waitUntil, then Condition's internal
logic for when it should wake up makes its decision based on the current WallTime - but if
you use MonotonicTime then waitUntil will make its decision based on current
MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono
did not have the concept of a dynamic clock type.
This patch does not include conversions between std::chrono and these new time classes,
because past experience shows that we're quite bad at getting conversions between
std::chrono and anything else right. Also, I didn't need such conversion code because this
patch only converts code that transitively touches ParkingLot and Condition. It was easy
to get all of that code onto the new time classes.
* WTF.xcodeproj/project.pbxproj:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/ClockType.cpp: Added.
(WTF::printInternal):
* wtf/ClockType.h: Added.
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::waitFor):
(WTF::ConditionBase::wait):
(WTF::ConditionBase::waitUntilWallClockSeconds): Deleted.
(WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted.
(WTF::ConditionBase::waitForSeconds): Deleted.
(WTF::ConditionBase::waitForSecondsImpl): Deleted.
(WTF::ConditionBase::waitForImpl): Deleted.
(WTF::ConditionBase::absoluteFromRelative): Deleted.
* wtf/CrossThreadQueue.h:
(WTF::CrossThreadQueue<DataType>::waitForMessage):
* wtf/CurrentTime.cpp:
(WTF::sleep):
* wtf/MessageQueue.h:
(WTF::MessageQueue::infiniteTime): Deleted.
* wtf/MonotonicTime.cpp: Added.
(WTF::MonotonicTime::now):
(WTF::MonotonicTime::approximateWallTime):
(WTF::MonotonicTime::dump):
(WTF::MonotonicTime::sleep):
* wtf/MonotonicTime.h: Added.
(WTF::MonotonicTime::MonotonicTime):
(WTF::MonotonicTime::fromRawDouble):
(WTF::MonotonicTime::infinity):
(WTF::MonotonicTime::secondsSinceEpoch):
(WTF::MonotonicTime::approximateMonotonicTime):
(WTF::MonotonicTime::operator bool):
(WTF::MonotonicTime::operator+):
(WTF::MonotonicTime::operator-):
(WTF::MonotonicTime::operator+=):
(WTF::MonotonicTime::operator-=):
(WTF::MonotonicTime::operator==):
(WTF::MonotonicTime::operator!=):
(WTF::MonotonicTime::operator<):
(WTF::MonotonicTime::operator>):
(WTF::MonotonicTime::operator<=):
(WTF::MonotonicTime::operator>=):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkCount):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
* wtf/Seconds.cpp: Added.
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::dump):
(WTF::Seconds::sleep):
* wtf/Seconds.h: Added.
(WTF::Seconds::Seconds):
(WTF::Seconds::value):
(WTF::Seconds::seconds):
(WTF::Seconds::milliseconds):
(WTF::Seconds::microseconds):
(WTF::Seconds::nanoseconds):
(WTF::Seconds::fromMilliseconds):
(WTF::Seconds::fromMicroseconds):
(WTF::Seconds::fromNanoseconds):
(WTF::Seconds::infinity):
(WTF::Seconds::operator bool):
(WTF::Seconds::operator+):
(WTF::Seconds::operator-):
(WTF::Seconds::operator*):
(WTF::Seconds::operator/):
(WTF::Seconds::operator+=):
(WTF::Seconds::operator-=):
(WTF::Seconds::operator*=):
(WTF::Seconds::operator/=):
(WTF::Seconds::operator==):
(WTF::Seconds::operator!=):
(WTF::Seconds::operator<):
(WTF::Seconds::operator>):
(WTF::Seconds::operator<=):
(WTF::Seconds::operator>=):
* wtf/TimeWithDynamicClockType.cpp: Added.
(WTF::TimeWithDynamicClockType::now):
(WTF::TimeWithDynamicClockType::nowWithSameClock):
(WTF::TimeWithDynamicClockType::wallTime):
(WTF::TimeWithDynamicClockType::monotonicTime):
(WTF::TimeWithDynamicClockType::approximateWallTime):
(WTF::TimeWithDynamicClockType::approximateMonotonicTime):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator<):
(WTF::TimeWithDynamicClockType::operator>):
(WTF::TimeWithDynamicClockType::operator<=):
(WTF::TimeWithDynamicClockType::operator>=):
(WTF::TimeWithDynamicClockType::dump):
(WTF::TimeWithDynamicClockType::sleep):
* wtf/TimeWithDynamicClockType.h: Added.
(WTF::TimeWithDynamicClockType::TimeWithDynamicClockType):
(WTF::TimeWithDynamicClockType::fromRawDouble):
(WTF::TimeWithDynamicClockType::secondsSinceEpoch):
(WTF::TimeWithDynamicClockType::clockType):
(WTF::TimeWithDynamicClockType::withSameClockAndRawDouble):
(WTF::TimeWithDynamicClockType::operator bool):
(WTF::TimeWithDynamicClockType::operator+):
(WTF::TimeWithDynamicClockType::operator-):
(WTF::TimeWithDynamicClockType::operator+=):
(WTF::TimeWithDynamicClockType::operator-=):
(WTF::TimeWithDynamicClockType::operator==):
(WTF::TimeWithDynamicClockType::operator!=):
* wtf/WallTime.cpp: Added.
(WTF::WallTime::now):
(WTF::WallTime::approximateMonotonicTime):
(WTF::WallTime::dump):
(WTF::WallTime::sleep):
* wtf/WallTime.h: Added.
(WTF::WallTime::WallTime):
(WTF::WallTime::fromRawDouble):
(WTF::WallTime::infinity):
(WTF::WallTime::secondsSinceEpoch):
(WTF::WallTime::approximateWallTime):
(WTF::WallTime::operator bool):
(WTF::WallTime::operator+):
(WTF::WallTime::operator-):
(WTF::WallTime::operator+=):
(WTF::WallTime::operator-=):
(WTF::WallTime::operator==):
(WTF::WallTime::operator!=):
(WTF::WallTime::operator<):
(WTF::WallTime::operator>):
(WTF::WallTime::operator<=):
(WTF::WallTime::operator>=):
* wtf/threads/BinarySemaphore.cpp:
(WTF::BinarySemaphore::wait):
* wtf/threads/BinarySemaphore.h:
Tools:
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/Condition.cpp:
(TestWebKitAPI::TEST):
* TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp:
(TestWebKitAPI::ToUpperConverter::stopProducing):
(TestWebKitAPI::ToUpperConverter::stopConsuming):
* TestWebKitAPI/Tests/WTF/Time.cpp: Added.
(WTF::operator<<):
(TestWebKitAPI::TEST):
Canonical link: https://commits.webkit.org/182152@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
|
|
|
const TimeWithDynamicClockType& timeout);
|
2016-04-20 04:25:02 +00:00
|
|
|
|
|
|
|
WTF_EXPORT_PRIVATE static void unparkOneImpl(
|
WTF::Lock should be fair eventually
https://bugs.webkit.org/show_bug.cgi?id=159384
Reviewed by Geoffrey Garen.
Source/WTF:
In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of
locks makes them fast. That post presented lock fairness as a trade-off between two
extremes:
- Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a
thread on the queue. If there was a thread on the queue, the lock is released and that
thread is made runnable. That thread may then grab the lock, or some other thread may grab
the lock first (it may barge). Usually, the barging thread is the thread that released the
lock in the first place. This maximizes throughput but hurts fairness. There is no good
theoretical bound on how unfair the lock may become, but empirical data suggests that it's
fair enough for the cases we previously measured.
- FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock()
if there is a thread waiting. If there is a thread waiting, unlock() will make that thread
runnable and inform it that it now holds the lock. This ensures perfect round-robin
fairness and allows us to reason theoretically about how long it may take for a thread to
grab the lock. For example, if we know that only N threads are running and each one may
contend on a critical section, and each one may hold the lock for at most S seconds, then
the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly
in most cases. This is because for the common case of short critical sections, they force
a context switch after each critical section if the lock is contended.
This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging.
Thanks to this new algorithm, you can now have both of these things at the same time.
This change makes WTF::Lock eventually fair. We can almost (more on the caveats below)
guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical
sections that are longer than 1ms are always fair. For shorter critical sections, the amount
of time that any thread waits is 1ms times the number of threads. There are some caveats
that arise from our use of randomness, but even then, in the limit as the critical section
length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our
experiments show that the lock becomes exactly as fair as a FIFO lock for any critical
section that is 1ms or longer.
The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock
fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use
fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair.
ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each
other via a token. unparkOne() can pass a token, which parkConditionally() will return. This
change also makes parkConditionally() a lot more precise about when it was unparked due to a
call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is
guaranteed to report that it was unparked rather than timing out, and that thread is
guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as
a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By
default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing
fair unlocking. In that case, unlock() will not release the lock, and lock() will know that
it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies
on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock
can make a decision about unlock strategy and inject a token while it has complete knowledge
over the state of the queue. As such, it's not immediately obvious how to implement this
algorithm on top of futexes. You really need ParkingLot!
WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(),
which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a
per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback
sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms
at random. When a dequeue happens and there are threads that actually get dequeued, we check
if the time since the last unfair unlock (the last time timeToBeFair was set to true) is
more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout.
This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to
happen at least once per millisecond. It will happen at 2 KHz on average. If there are
collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the
average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid
resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the
timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment
of everyone else. Randomness ensures that we aren't too fair to any one thread.
Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous
improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the
old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to
allow only one thread to hold the lock during a whole second in which each thread is holding
the lock for 1ms at a time. This is because in a barging lock, releasing a lock after
holding it for 1ms and then reacquiring it immediately virtually ensures that none of the
other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock
handles this case like a champ: each thread gets equal turns.
Here's some data. If we launch 10 threads and have each of them run for 1 second while
repeatedly holding a critical section for 1ms, then here's how many times each thread gets
to hold the lock using the old WTF::Lock algorithm:
799, 6, 1, 1, 1, 1, 1, 1, 1, 1
One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock
becomes totally fair:
80, 79, 79, 79, 79, 79, 79, 80, 80, 79
I don't know of anyone creating such an automatically-fair adaptive lock before, so I think
that this is a pretty awesome advancement to the state of the art!
This change is good for three reasons:
- We do have long critical sections in WebKit and we don't want to have to worry about
starvation. This reduces the likelihood that we will see starvation due to our lock
strategy.
- I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or
lockFairly() or some moral equivalent for the scavenger thread.
- If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability
to unlock and relock without barging.
* benchmarks/LockFairnessTest.cpp:
(main):
* benchmarks/ToyLocks.h:
* wtf/Condition.h:
(WTF::ConditionBase::waitUntil):
(WTF::ConditionBase::notifyOne):
* wtf/Lock.cpp:
(WTF::LockBase::lockSlow):
(WTF::LockBase::unlockSlow):
(WTF::LockBase::unlockFairlySlow):
(WTF::LockBase::unlockSlowImpl):
* wtf/Lock.h:
(WTF::LockBase::try_lock):
(WTF::LockBase::unlock):
(WTF::LockBase::unlockFairly):
(WTF::LockBase::isHeld):
(WTF::LockBase::isFullyReset):
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::parkConditionallyImpl):
(WTF::ParkingLot::unparkOne):
(WTF::ParkingLot::unparkOneImpl):
(WTF::ParkingLot::unparkAll):
* wtf/ParkingLot.h:
(WTF::ParkingLot::parkConditionally):
(WTF::ParkingLot::compareAndPark):
(WTF::ParkingLot::unparkOne):
Tools:
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/178039@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
|
|
|
const void* address, const ScopedLambda<intptr_t(UnparkResult)>& callback);
|
2016-04-20 04:25:02 +00:00
|
|
|
|
[WTF] Introduce Thread class and use RefPtr<Thread> and align Windows Threading implementation semantics to Pthread one
https://bugs.webkit.org/show_bug.cgi?id=170502
Reviewed by Mark Lam.
Source/JavaScriptCore:
* API/tests/CompareAndSwapTest.cpp:
(testCompareAndSwap):
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/air/testair.cpp:
* b3/testb3.cpp:
(JSC::B3::run):
* bytecode/SuperSampler.cpp:
(JSC::initializeSuperSampler):
* dfg/DFGWorklist.cpp:
* disassembler/Disassembler.cpp:
* heap/Heap.cpp:
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::notifyIsSafeToCollect):
* heap/Heap.h:
* heap/MachineStackMarker.cpp:
(JSC::MachineThreads::~MachineThreads):
(JSC::MachineThreads::addCurrentThread):
(JSC::MachineThreads::removeThread):
(JSC::MachineThreads::removeThreadIfFound):
(JSC::MachineThreads::MachineThread::MachineThread):
(JSC::MachineThreads::MachineThread::getRegisters):
(JSC::MachineThreads::MachineThread::Registers::stackPointer):
(JSC::MachineThreads::MachineThread::Registers::framePointer):
(JSC::MachineThreads::MachineThread::Registers::instructionPointer):
(JSC::MachineThreads::MachineThread::Registers::llintPC):
(JSC::MachineThreads::MachineThread::captureStack):
(JSC::MachineThreads::tryCopyOtherThreadStack):
(JSC::MachineThreads::tryCopyOtherThreadStacks):
(pthreadSignalHandlerSuspendResume): Deleted.
(JSC::threadData): Deleted.
(JSC::MachineThreads::Thread::Thread): Deleted.
(JSC::MachineThreads::Thread::createForCurrentThread): Deleted.
(JSC::MachineThreads::Thread::operator==): Deleted.
(JSC::MachineThreads::machineThreadForCurrentThread): Deleted.
(JSC::MachineThreads::ThreadData::ThreadData): Deleted.
(JSC::MachineThreads::ThreadData::~ThreadData): Deleted.
(JSC::MachineThreads::ThreadData::suspend): Deleted.
(JSC::MachineThreads::ThreadData::resume): Deleted.
(JSC::MachineThreads::ThreadData::getRegisters): Deleted.
(JSC::MachineThreads::ThreadData::Registers::stackPointer): Deleted.
(JSC::MachineThreads::ThreadData::Registers::framePointer): Deleted.
(JSC::MachineThreads::ThreadData::Registers::instructionPointer): Deleted.
(JSC::MachineThreads::ThreadData::Registers::llintPC): Deleted.
(JSC::MachineThreads::ThreadData::freeRegisters): Deleted.
(JSC::MachineThreads::ThreadData::captureStack): Deleted.
* heap/MachineStackMarker.h:
(JSC::MachineThreads::MachineThread::suspend):
(JSC::MachineThreads::MachineThread::resume):
(JSC::MachineThreads::MachineThread::threadID):
(JSC::MachineThreads::MachineThread::stackBase):
(JSC::MachineThreads::MachineThread::stackEnd):
(JSC::MachineThreads::threadsListHead):
(JSC::MachineThreads::Thread::operator!=): Deleted.
(JSC::MachineThreads::Thread::suspend): Deleted.
(JSC::MachineThreads::Thread::resume): Deleted.
(JSC::MachineThreads::Thread::getRegisters): Deleted.
(JSC::MachineThreads::Thread::freeRegisters): Deleted.
(JSC::MachineThreads::Thread::captureStack): Deleted.
(JSC::MachineThreads::Thread::platformThread): Deleted.
(JSC::MachineThreads::Thread::stackBase): Deleted.
(JSC::MachineThreads::Thread::stackEnd): Deleted.
* jit/ICStats.cpp:
(JSC::ICStats::ICStats):
(JSC::ICStats::~ICStats):
* jit/ICStats.h:
* jsc.cpp:
(functionDollarAgentStart):
(startTimeoutThreadIfNeeded):
* runtime/JSLock.cpp:
(JSC::JSLock::lock):
* runtime/JSLock.h:
(JSC::JSLock::ownerThread):
(JSC::JSLock::currentThreadIsHoldingLock):
* runtime/SamplingProfiler.cpp:
(JSC::FrameWalker::isValidFramePointer):
(JSC::SamplingProfiler::SamplingProfiler):
(JSC::SamplingProfiler::createThreadIfNecessary):
(JSC::SamplingProfiler::takeSample):
* runtime/SamplingProfiler.h:
* runtime/VM.h:
(JSC::VM::ownerThread):
* runtime/VMTraps.cpp:
(JSC::findActiveVMAndStackBounds):
(JSC::VMTraps::SignalSender::send):
(JSC::VMTraps::fireTrap):
Source/WebCore:
Mechanical change. Use Thread:: APIs.
* Modules/indexeddb/server/IDBServer.cpp:
(WebCore::IDBServer::IDBServer::IDBServer):
* Modules/indexeddb/server/IDBServer.h:
* Modules/webaudio/AsyncAudioDecoder.cpp:
(WebCore::AsyncAudioDecoder::AsyncAudioDecoder):
(WebCore::AsyncAudioDecoder::~AsyncAudioDecoder):
(WebCore::AsyncAudioDecoder::runLoop):
* Modules/webaudio/AsyncAudioDecoder.h:
* Modules/webaudio/OfflineAudioDestinationNode.cpp:
(WebCore::OfflineAudioDestinationNode::OfflineAudioDestinationNode):
(WebCore::OfflineAudioDestinationNode::uninitialize):
(WebCore::OfflineAudioDestinationNode::startRendering):
* Modules/webaudio/OfflineAudioDestinationNode.h:
* Modules/webdatabase/Database.cpp:
(WebCore::Database::securityOrigin):
* Modules/webdatabase/DatabaseThread.cpp:
(WebCore::DatabaseThread::start):
(WebCore::DatabaseThread::databaseThread):
(WebCore::DatabaseThread::recordDatabaseOpen):
(WebCore::DatabaseThread::recordDatabaseClosed):
* Modules/webdatabase/DatabaseThread.h:
(WebCore::DatabaseThread::getThreadID):
* bindings/js/GCController.cpp:
(WebCore::GCController::garbageCollectOnAlternateThreadForDebugging):
* fileapi/AsyncFileStream.cpp:
(WebCore::callOnFileThread):
* loader/icon/IconDatabase.cpp:
(WebCore::IconDatabase::open):
(WebCore::IconDatabase::close):
* loader/icon/IconDatabase.h:
* page/ResourceUsageThread.cpp:
(WebCore::ResourceUsageThread::createThreadIfNeeded):
* page/ResourceUsageThread.h:
* page/scrolling/ScrollingThread.cpp:
(WebCore::ScrollingThread::ScrollingThread):
(WebCore::ScrollingThread::isCurrentThread):
(WebCore::ScrollingThread::createThreadIfNeeded):
(WebCore::ScrollingThread::threadCallback):
* page/scrolling/ScrollingThread.h:
* platform/audio/HRTFDatabaseLoader.cpp:
(WebCore::HRTFDatabaseLoader::HRTFDatabaseLoader):
(WebCore::HRTFDatabaseLoader::loadAsynchronously):
(WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion):
* platform/audio/HRTFDatabaseLoader.h:
* platform/audio/ReverbConvolver.cpp:
(WebCore::ReverbConvolver::ReverbConvolver):
(WebCore::ReverbConvolver::~ReverbConvolver):
* platform/audio/ReverbConvolver.h:
* platform/audio/gstreamer/AudioFileReaderGStreamer.cpp:
(WebCore::createBusFromAudioFile):
(WebCore::createBusFromInMemoryAudioFile):
* platform/graphics/gstreamer/WebKitWebSourceGStreamer.cpp:
(ResourceHandleStreamingClient::ResourceHandleStreamingClient):
(ResourceHandleStreamingClient::~ResourceHandleStreamingClient):
* platform/network/cf/LoaderRunLoopCF.cpp:
(WebCore::loaderRunLoop):
* platform/network/curl/CurlDownload.cpp:
(WebCore::CurlDownloadManager::startThreadIfNeeded):
(WebCore::CurlDownloadManager::stopThread):
* platform/network/curl/CurlDownload.h:
* platform/network/curl/SocketStreamHandleImpl.h:
* platform/network/curl/SocketStreamHandleImplCurl.cpp:
(WebCore::SocketStreamHandleImpl::startThread):
(WebCore::SocketStreamHandleImpl::stopThread):
* workers/WorkerThread.cpp:
(WebCore::WorkerThread::WorkerThread):
(WebCore::WorkerThread::start):
(WebCore::WorkerThread::workerThread):
* workers/WorkerThread.h:
(WebCore::WorkerThread::threadID):
Source/WebKit:
Mechanical change. Use Thread:: APIs.
* Storage/StorageThread.cpp:
(WebCore::StorageThread::StorageThread):
(WebCore::StorageThread::~StorageThread):
(WebCore::StorageThread::start):
(WebCore::StorageThread::dispatch):
(WebCore::StorageThread::terminate):
* Storage/StorageThread.h:
Source/WebKit2:
Mechanical change. Use Thread:: APIs.
* NetworkProcess/NetworkProcess.cpp:
(WebKit::NetworkProcess::initializeNetworkProcess):
* NetworkProcess/cache/NetworkCacheIOChannelSoup.cpp:
(WebKit::NetworkCache::IOChannel::readSyncInThread):
* Platform/IPC/Connection.cpp:
(IPC::Connection::processIncomingMessage):
* Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.h:
(WebKit::XPCServiceInitializer):
* UIProcess/linux/MemoryPressureMonitor.cpp:
(WebKit::MemoryPressureMonitor::MemoryPressureMonitor):
* WebProcess/WebProcess.cpp:
(WebKit::WebProcess::initializeWebProcess):
Source/WTF:
This patch is refactoring of WTF Threading mechanism to merge JavaScriptCore's threading extension to WTF Threading.
Previously, JavaScriptCore requires richer threading features (such as suspending and resuming threads), and they
are implemented for PlatformThread in JavaScriptCore. But these features should be implemented in WTF Threading side
instead of maintaining JSC's own threading features too. This patch removes these features from JSC and move it to
WTF Threading.
However, current WTF Threading has one problem: Windows version of WTF Threading has different semantics from Pthreads
one. In Windows WTF Threading, we cannot perform any operation after the target thread is detached: WTF Threading stop
tracking the state of the thread once the thread is detached. But this is not the same to Pthreads one. In Pthreads,
pthread_detach just means that the resource of the thread will be destroyed automatically. While some operations like
pthread_join will be rejected, some operations like pthread_kill will be accepted.
The problem is that detached thread can be suspended and resumed in JSC. For example, in jsc.cpp, we start the worker
thread and detach it immediately. In worker thread, we will create VM and thus concurrent GC will suspend and resume
the detached thread. However, in Windows WTF Threading, we have no reference to the detached thread. Thus we cannot
perform suspend and resume operations onto the detached thread.
To solve the problem, we change Windows Threading mechanism drastically to align it to the Pthread semantics. In the
new Threading, we have RefPtr<Thread> class. It holds a handle to a platform thread. We can perform threading operations
with this class. For example, Thread::suspend is offered. And we use destructor of the thread local variable to release
the resources held by RefPtr<Thread>. In Windows, Thread::detach does nothing because the resource will be destroyed
automatically by RefPtr<Thread>.
To do so, we introduce ThreadHolder for Windows. This is similar to the previous ThreadIdentifierData for Pthreads.
It holds RefPtr<Thread> in the thread local storage (technically, it is Fiber Local Storage in Windows). Thread::current()
will return this reference.
The problematic situation is that the order of the deallocation of the thread local storage is not defined. So we should
not touch thread local storage in the destructor of the thread local storage. To avoid such edge cases, we have
currentThread() / Thread::currentID() APIs. They are safe to be called even in the destructor of the other thread local
storage. And in Windows, in the FLS destructor, we will create the thread_local variable to defer the destruction of
the ThreadHolder. We ensure that this destructor is called after the other FLS destructors are called in Windows 10.
This patch is performance neutral.
* WTF.xcodeproj/project.pbxproj:
* benchmarks/ConditionSpeedTest.cpp:
* benchmarks/LockFairnessTest.cpp:
* benchmarks/LockSpeedTest.cpp:
* wtf/AutomaticThread.cpp:
(WTF::AutomaticThread::start):
* wtf/CMakeLists.txt:
* wtf/MainThread.h:
* wtf/MemoryPressureHandler.h:
* wtf/ParallelJobsGeneric.cpp:
(WTF::ParallelEnvironment::ThreadPrivate::tryLockFor):
(WTF::ParallelEnvironment::ThreadPrivate::workerThread):
* wtf/ParallelJobsGeneric.h:
(WTF::ParallelEnvironment::ThreadPrivate::ThreadPrivate): Deleted.
* wtf/ParkingLot.cpp:
(WTF::ParkingLot::forEachImpl):
* wtf/ParkingLot.h:
(WTF::ParkingLot::forEach):
* wtf/PlatformRegisters.h: Renamed from Source/JavaScriptCore/runtime/PlatformThread.h.
* wtf/RefPtr.h:
(WTF::RefPtr::RefPtr):
* wtf/ThreadFunctionInvocation.h:
(WTF::ThreadFunctionInvocation::ThreadFunctionInvocation):
* wtf/ThreadHolder.cpp: Added.
(WTF::ThreadHolder::~ThreadHolder):
(WTF::ThreadHolder::initialize):
* wtf/ThreadHolder.h: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.h.
(WTF::ThreadHolder::thread):
(WTF::ThreadHolder::ThreadHolder):
* wtf/ThreadHolderPthreads.cpp: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.cpp.
(WTF::ThreadHolder::initializeOnce):
(WTF::ThreadHolder::current):
(WTF::ThreadHolder::destruct):
* wtf/ThreadHolderWin.cpp: Added.
(WTF::threadMapMutex):
(WTF::threadMap):
(WTF::ThreadHolder::initializeOnce):
(WTF::ThreadHolder::current):
(WTF::ThreadHolder::destruct):
* wtf/ThreadSpecific.h:
* wtf/Threading.cpp:
(WTF::Thread::normalizeThreadName):
(WTF::threadEntryPoint):
(WTF::Thread::create):
(WTF::Thread::setCurrentThreadIsUserInteractive):
(WTF::Thread::setCurrentThreadIsUserInitiated):
(WTF::Thread::setGlobalMaxQOSClass):
(WTF::Thread::adjustedQOSClass):
(WTF::Thread::dump):
(WTF::initializeThreading):
(WTF::normalizeThreadName): Deleted.
(WTF::createThread): Deleted.
(WTF::setCurrentThreadIsUserInteractive): Deleted.
(WTF::setCurrentThreadIsUserInitiated): Deleted.
(WTF::setGlobalMaxQOSClass): Deleted.
(WTF::adjustedQOSClass): Deleted.
* wtf/Threading.h:
(WTF::Thread::id):
(WTF::Thread::operator==):
(WTF::Thread::operator!=):
(WTF::Thread::joinableState):
(WTF::Thread::didBecomeDetached):
(WTF::Thread::didJoin):
(WTF::Thread::hasExited):
(WTF::currentThread):
* wtf/ThreadingPthreads.cpp:
(WTF::Thread::Thread):
(WTF::Thread::~Thread):
(WTF::Thread::signalHandlerSuspendResume):
(WTF::Thread::initializePlatformThreading):
(WTF::initializeCurrentThreadEvenIfNonWTFCreated):
(WTF::wtfThreadEntryPoint):
(WTF::Thread::createInternal):
(WTF::Thread::initializeCurrentThreadInternal):
(WTF::Thread::changePriority):
(WTF::Thread::waitForCompletion):
(WTF::Thread::detach):
(WTF::Thread::current):
(WTF::Thread::currentID):
(WTF::Thread::signal):
(WTF::Thread::resume):
(WTF::Thread::getRegisters):
(WTF::Thread::didExit):
(WTF::Thread::establish):
(WTF::PthreadState::PthreadState): Deleted.
(WTF::PthreadState::joinableState): Deleted.
(WTF::PthreadState::pthreadHandle): Deleted.
(WTF::PthreadState::didBecomeDetached): Deleted.
(WTF::PthreadState::didExit): Deleted.
(WTF::PthreadState::didJoin): Deleted.
(WTF::PthreadState::hasExited): Deleted.
(WTF::threadMapMutex): Deleted.
(WTF::initializeThreading): Deleted.
(WTF::threadMap): Deleted.
(WTF::identifierByPthreadHandle): Deleted.
(WTF::establishIdentifierForPthreadHandle): Deleted.
(WTF::pthreadHandleForIdentifierWithLockAlreadyHeld): Deleted.
(WTF::createThreadInternal): Deleted.
(WTF::initializeCurrentThreadInternal): Deleted.
(WTF::changeThreadPriority): Deleted.
(WTF::waitForThreadCompletion): Deleted.
(WTF::detachThread): Deleted.
(WTF::threadDidExit): Deleted.
(WTF::currentThread): Deleted.
(WTF::signalThread): Deleted.
* wtf/ThreadingWin.cpp:
(WTF::Thread::Thread):
(WTF::Thread::~Thread):
(WTF::Thread::initializeCurrentThreadInternal):
(WTF::Thread::initializePlatformThreading):
(WTF::wtfThreadEntryPoint):
(WTF::Thread::createInternal):
(WTF::Thread::changePriority):
(WTF::Thread::waitForCompletion):
(WTF::Thread::detach):
(WTF::Thread::resume):
(WTF::Thread::getRegisters):
(WTF::Thread::current):
(WTF::Thread::currentID):
(WTF::Thread::didExit):
(WTF::Thread::establish):
(WTF::initializeCurrentThreadInternal): Deleted.
(WTF::threadMapMutex): Deleted.
(WTF::initializeThreading): Deleted.
(WTF::threadMap): Deleted.
(WTF::storeThreadHandleByIdentifier): Deleted.
(WTF::threadHandleForIdentifier): Deleted.
(WTF::clearThreadHandleForIdentifier): Deleted.
(WTF::createThreadInternal): Deleted.
(WTF::changeThreadPriority): Deleted.
(WTF::waitForThreadCompletion): Deleted.
(WTF::detachThread): Deleted.
(WTF::currentThread): Deleted.
* wtf/WorkQueue.cpp:
(WTF::WorkQueue::concurrentApply):
* wtf/WorkQueue.h:
* wtf/cocoa/WorkQueueCocoa.cpp:
(WTF::dispatchQOSClass):
* wtf/generic/WorkQueueGeneric.cpp:
(WorkQueue::platformInitialize):
(WorkQueue::platformInvalidate):
* wtf/linux/MemoryPressureHandlerLinux.cpp:
(WTF::MemoryPressureHandler::EventFDPoller::EventFDPoller):
(WTF::MemoryPressureHandler::EventFDPoller::~EventFDPoller):
* wtf/win/MainThreadWin.cpp:
(WTF::initializeMainThreadPlatform):
Tools:
Mechanical change. Use Thread:: APIs.
* DumpRenderTree/JavaScriptThreading.cpp:
(runJavaScriptThread):
(startJavaScriptThreads):
(stopJavaScriptThreads):
* DumpRenderTree/mac/DumpRenderTree.mm:
(testThreadIdentifierMap):
* TestWebKitAPI/Tests/WTF/Condition.cpp:
* TestWebKitAPI/Tests/WTF/Lock.cpp:
(TestWebKitAPI::runLockTest):
* TestWebKitAPI/Tests/WTF/ParkingLot.cpp:
Canonical link: https://commits.webkit.org/187690@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@215265 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-04-12 12:08:29 +00:00
|
|
|
WTF_EXPORT_PRIVATE static void forEachImpl(const ScopedLambda<void(Thread&, const void*)>&);
|
2015-08-11 19:51:35 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
} // namespace WTF
|
|
|
|
|
|
|
|
using WTF::ParkingLot;
|