haikuwebkit/Source/WTF/wtf/ParkingLot.h

181 lines
8.6 KiB
C
Raw Permalink Normal View History

WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
/*
* Copyright (C) 2015-2016 Apple Inc. All rights reserved.
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
Use pragma once in WTF https://bugs.webkit.org/show_bug.cgi?id=190527 Reviewed by Chris Dumez. Source/WTF: We also need to consistently include wtf headers from within wtf so we can build wtf without symbol redefinition errors from including the copy in Source and the copy in the build directory. * wtf/ASCIICType.h: * wtf/Assertions.cpp: * wtf/Assertions.h: * wtf/Atomics.h: * wtf/AutomaticThread.cpp: * wtf/AutomaticThread.h: * wtf/BackwardsGraph.h: * wtf/Bag.h: * wtf/BagToHashMap.h: * wtf/BitVector.cpp: * wtf/BitVector.h: * wtf/Bitmap.h: * wtf/BloomFilter.h: * wtf/Box.h: * wtf/BubbleSort.h: * wtf/BumpPointerAllocator.h: * wtf/ByteOrder.h: * wtf/CPUTime.cpp: * wtf/CallbackAggregator.h: * wtf/CheckedArithmetic.h: * wtf/CheckedBoolean.h: * wtf/ClockType.cpp: * wtf/ClockType.h: * wtf/CommaPrinter.h: * wtf/CompilationThread.cpp: * wtf/CompilationThread.h: * wtf/Compiler.h: * wtf/ConcurrentPtrHashSet.cpp: * wtf/ConcurrentVector.h: * wtf/Condition.h: * wtf/CountingLock.cpp: * wtf/CrossThreadTaskHandler.cpp: * wtf/CryptographicUtilities.cpp: * wtf/CryptographicUtilities.h: * wtf/CryptographicallyRandomNumber.cpp: * wtf/CryptographicallyRandomNumber.h: * wtf/CurrentTime.cpp: * wtf/DataLog.cpp: * wtf/DataLog.h: * wtf/DateMath.cpp: * wtf/DateMath.h: * wtf/DecimalNumber.cpp: * wtf/DecimalNumber.h: * wtf/Deque.h: * wtf/DisallowCType.h: * wtf/Dominators.h: * wtf/DoublyLinkedList.h: * wtf/FastBitVector.cpp: * wtf/FastMalloc.cpp: * wtf/FastMalloc.h: * wtf/FeatureDefines.h: * wtf/FilePrintStream.cpp: * wtf/FilePrintStream.h: * wtf/FlipBytes.h: * wtf/FunctionDispatcher.cpp: * wtf/FunctionDispatcher.h: * wtf/GetPtr.h: * wtf/Gigacage.cpp: * wtf/GlobalVersion.cpp: * wtf/GraphNodeWorklist.h: * wtf/GregorianDateTime.cpp: * wtf/GregorianDateTime.h: * wtf/HashFunctions.h: * wtf/HashMap.h: * wtf/HashMethod.h: * wtf/HashSet.h: * wtf/HashTable.cpp: * wtf/HashTraits.h: * wtf/Indenter.h: * wtf/IndexSparseSet.h: * wtf/InlineASM.h: * wtf/Insertion.h: * wtf/IteratorAdaptors.h: * wtf/IteratorRange.h: * wtf/JSONValues.cpp: * wtf/JSValueMalloc.cpp: * wtf/LEBDecoder.h: * wtf/Language.cpp: * wtf/ListDump.h: * wtf/Lock.cpp: * wtf/Lock.h: * wtf/LockAlgorithm.h: * wtf/LockedPrintStream.cpp: * wtf/Locker.h: * wtf/MD5.cpp: * wtf/MD5.h: * wtf/MainThread.cpp: * wtf/MainThread.h: * wtf/MallocPtr.h: * wtf/MathExtras.h: * wtf/MediaTime.cpp: * wtf/MediaTime.h: * wtf/MemoryPressureHandler.cpp: * wtf/MessageQueue.h: * wtf/MetaAllocator.cpp: * wtf/MetaAllocator.h: * wtf/MetaAllocatorHandle.h: * wtf/MonotonicTime.cpp: * wtf/MonotonicTime.h: * wtf/NakedPtr.h: * wtf/NoLock.h: * wtf/NoTailCalls.h: * wtf/Noncopyable.h: * wtf/NumberOfCores.cpp: * wtf/NumberOfCores.h: * wtf/OSAllocator.h: * wtf/OSAllocatorPosix.cpp: * wtf/OSRandomSource.cpp: * wtf/OSRandomSource.h: * wtf/ObjcRuntimeExtras.h: * wtf/OrderMaker.h: * wtf/PackedIntVector.h: * wtf/PageAllocation.h: * wtf/PageBlock.cpp: * wtf/PageBlock.h: * wtf/PageReservation.h: * wtf/ParallelHelperPool.cpp: * wtf/ParallelHelperPool.h: * wtf/ParallelJobs.h: * wtf/ParallelJobsLibdispatch.h: * wtf/ParallelVectorIterator.h: * wtf/ParkingLot.cpp: * wtf/ParkingLot.h: * wtf/Platform.h: * wtf/PointerComparison.h: * wtf/Poisoned.cpp: * wtf/PrintStream.cpp: * wtf/PrintStream.h: * wtf/ProcessID.h: * wtf/ProcessPrivilege.cpp: * wtf/RAMSize.cpp: * wtf/RAMSize.h: * wtf/RandomDevice.cpp: * wtf/RandomNumber.cpp: * wtf/RandomNumber.h: * wtf/RandomNumberSeed.h: * wtf/RangeSet.h: * wtf/RawPointer.h: * wtf/ReadWriteLock.cpp: * wtf/RedBlackTree.h: * wtf/Ref.h: * wtf/RefCountedArray.h: * wtf/RefCountedLeakCounter.cpp: * wtf/RefCountedLeakCounter.h: * wtf/RefCounter.h: * wtf/RefPtr.h: * wtf/RetainPtr.h: * wtf/RunLoop.cpp: * wtf/RunLoop.h: * wtf/RunLoopTimer.h: * wtf/RunLoopTimerCF.cpp: * wtf/SHA1.cpp: * wtf/SHA1.h: * wtf/SaturatedArithmetic.h: (saturatedSubtraction): * wtf/SchedulePair.h: * wtf/SchedulePairCF.cpp: * wtf/SchedulePairMac.mm: * wtf/ScopedLambda.h: * wtf/Seconds.cpp: * wtf/Seconds.h: * wtf/SegmentedVector.h: * wtf/SentinelLinkedList.h: * wtf/SharedTask.h: * wtf/SimpleStats.h: * wtf/SingleRootGraph.h: * wtf/SinglyLinkedList.h: * wtf/SixCharacterHash.cpp: * wtf/SixCharacterHash.h: * wtf/SmallPtrSet.h: * wtf/Spectrum.h: * wtf/StackBounds.cpp: * wtf/StackBounds.h: * wtf/StackStats.cpp: * wtf/StackStats.h: * wtf/StackTrace.cpp: * wtf/StdLibExtras.h: * wtf/StreamBuffer.h: * wtf/StringHashDumpContext.h: * wtf/StringPrintStream.cpp: * wtf/StringPrintStream.h: * wtf/ThreadGroup.cpp: * wtf/ThreadMessage.cpp: * wtf/ThreadSpecific.h: * wtf/Threading.cpp: * wtf/Threading.h: * wtf/ThreadingPrimitives.h: * wtf/ThreadingPthreads.cpp: * wtf/TimeWithDynamicClockType.cpp: * wtf/TimeWithDynamicClockType.h: * wtf/TimingScope.cpp: * wtf/TinyLRUCache.h: * wtf/TinyPtrSet.h: * wtf/TriState.h: * wtf/TypeCasts.h: * wtf/UUID.cpp: * wtf/UnionFind.h: * wtf/VMTags.h: * wtf/ValueCheck.h: * wtf/Vector.h: * wtf/VectorTraits.h: * wtf/WallTime.cpp: * wtf/WallTime.h: * wtf/WeakPtr.h: * wtf/WeakRandom.h: * wtf/WordLock.cpp: * wtf/WordLock.h: * wtf/WorkQueue.cpp: * wtf/WorkQueue.h: * wtf/WorkerPool.cpp: * wtf/cf/LanguageCF.cpp: * wtf/cf/RunLoopCF.cpp: * wtf/cocoa/Entitlements.mm: * wtf/cocoa/MachSendRight.cpp: * wtf/cocoa/MainThreadCocoa.mm: * wtf/cocoa/MemoryFootprintCocoa.cpp: * wtf/cocoa/WorkQueueCocoa.cpp: * wtf/dtoa.cpp: * wtf/dtoa.h: * wtf/ios/WebCoreThread.cpp: * wtf/ios/WebCoreThread.h: * wtf/mac/AppKitCompatibilityDeclarations.h: * wtf/mac/DeprecatedSymbolsUsedBySafari.mm: * wtf/mbmalloc.cpp: * wtf/persistence/PersistentCoders.cpp: * wtf/persistence/PersistentDecoder.cpp: * wtf/persistence/PersistentEncoder.cpp: * wtf/spi/cf/CFBundleSPI.h: * wtf/spi/darwin/CommonCryptoSPI.h: * wtf/text/ASCIIFastPath.h: * wtf/text/ASCIILiteral.cpp: * wtf/text/AtomicString.cpp: * wtf/text/AtomicString.h: * wtf/text/AtomicStringHash.h: * wtf/text/AtomicStringImpl.cpp: * wtf/text/AtomicStringImpl.h: * wtf/text/AtomicStringTable.cpp: * wtf/text/AtomicStringTable.h: * wtf/text/Base64.cpp: * wtf/text/CString.cpp: * wtf/text/CString.h: * wtf/text/ConversionMode.h: * wtf/text/ExternalStringImpl.cpp: * wtf/text/IntegerToStringConversion.h: * wtf/text/LChar.h: * wtf/text/LineEnding.cpp: * wtf/text/StringBuffer.h: * wtf/text/StringBuilder.cpp: * wtf/text/StringBuilder.h: * wtf/text/StringBuilderJSON.cpp: * wtf/text/StringCommon.h: * wtf/text/StringConcatenate.h: * wtf/text/StringHash.h: * wtf/text/StringImpl.cpp: * wtf/text/StringImpl.h: * wtf/text/StringOperators.h: * wtf/text/StringView.cpp: * wtf/text/StringView.h: * wtf/text/SymbolImpl.cpp: * wtf/text/SymbolRegistry.cpp: * wtf/text/SymbolRegistry.h: * wtf/text/TextBreakIterator.cpp: * wtf/text/TextBreakIterator.h: * wtf/text/TextBreakIteratorInternalICU.h: * wtf/text/TextPosition.h: * wtf/text/TextStream.cpp: * wtf/text/UniquedStringImpl.h: * wtf/text/WTFString.cpp: * wtf/text/WTFString.h: * wtf/text/cocoa/StringCocoa.mm: * wtf/text/cocoa/StringViewCocoa.mm: * wtf/text/cocoa/TextBreakIteratorInternalICUCocoa.cpp: * wtf/text/icu/UTextProvider.cpp: * wtf/text/icu/UTextProvider.h: * wtf/text/icu/UTextProviderLatin1.cpp: * wtf/text/icu/UTextProviderLatin1.h: * wtf/text/icu/UTextProviderUTF16.cpp: * wtf/text/icu/UTextProviderUTF16.h: * wtf/threads/BinarySemaphore.cpp: * wtf/threads/BinarySemaphore.h: * wtf/threads/Signals.cpp: * wtf/unicode/CharacterNames.h: * wtf/unicode/Collator.h: * wtf/unicode/CollatorDefault.cpp: * wtf/unicode/UTF8.cpp: * wtf/unicode/UTF8.h: Tools: Put WorkQueue in namespace DRT so it does not conflict with WTF::WorkQueue. * DumpRenderTree/TestRunner.cpp: (TestRunner::queueLoadHTMLString): (TestRunner::queueLoadAlternateHTMLString): (TestRunner::queueBackNavigation): (TestRunner::queueForwardNavigation): (TestRunner::queueLoadingScript): (TestRunner::queueNonLoadingScript): (TestRunner::queueReload): * DumpRenderTree/WorkQueue.cpp: (WorkQueue::singleton): Deleted. (WorkQueue::WorkQueue): Deleted. (WorkQueue::queue): Deleted. (WorkQueue::dequeue): Deleted. (WorkQueue::count): Deleted. (WorkQueue::clear): Deleted. (WorkQueue::processWork): Deleted. * DumpRenderTree/WorkQueue.h: (WorkQueue::setFrozen): Deleted. * DumpRenderTree/WorkQueueItem.h: * DumpRenderTree/mac/DumpRenderTree.mm: (runTest): * DumpRenderTree/mac/FrameLoadDelegate.mm: (-[FrameLoadDelegate processWork:]): (-[FrameLoadDelegate webView:locationChangeDone:forDataSource:]): * DumpRenderTree/mac/TestRunnerMac.mm: (TestRunner::notifyDone): (TestRunner::forceImmediateCompletion): (TestRunner::queueLoad): * DumpRenderTree/win/DumpRenderTree.cpp: (runTest): * DumpRenderTree/win/FrameLoadDelegate.cpp: (FrameLoadDelegate::processWork): (FrameLoadDelegate::locationChangeDone): * DumpRenderTree/win/TestRunnerWin.cpp: (TestRunner::notifyDone): (TestRunner::forceImmediateCompletion): (TestRunner::queueLoad): Canonical link: https://commits.webkit.org/205473@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@237099 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-10-15 14:24:49 +00:00
#pragma once
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
#include <wtf/Atomics.h>
ScopedLambda should have a lifetime story that makes sense to the compiler https://bugs.webkit.org/show_bug.cgi?id=158118 Reviewed by Mark Lam. Source/WTF: Prior to this change, there were two lifetime bugs in ScopedLambda: - scopedLambda(Functor&&) would bind Functor to const lambda&, so the resulting ScopedLambdaFunctor would hold a reference to the original lambda. This would have surprising behavior; for example it meant that this code was wrong: auto l = scopedLambda<things>([&] ...); The solution is to have explicit copy/move versions of scopedLambda() rather than rely on perfect forwarding. - ScopedLambdaFunctor did not override its copy or move operations, so if the compiler did not RVO scopedLambda(), it would return a ScopedLambdaFunctor whose m_arg points to a dead temporary ScopedLambdaFunctor instance. The solution is to have explicit copy/move constructors and operators, which preserve the invariant that ScopedLambda::m_arg points to this. One nice side-effect of all of these constructors and operators being explicit is that we can rely on WTFMove's excellent assertions, which helped catch the first issue. This reverts ParkingLot to use ScopedLambda again. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): * wtf/ScopedLambda.h: (WTF::scopedLambda): Tools: Added a test case. This test crashes before the fix and now it passes. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/ScopedLambda.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/176236@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@201433 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-05-26 21:58:42 +00:00
#include <wtf/ScopedLambda.h>
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows https://bugs.webkit.org/show_bug.cgi?id=152045 Reviewed by Andy Estes. Source/JavaScriptCore: Probably the nicest example of why this patch is a good idea is the change in AtomicsObject.cpp. * jit/ICStats.cpp: (JSC::ICStats::ICStats): * runtime/AtomicsObject.cpp: (JSC::atomicsFuncWait): Source/WebCore: No new layout tests because no new behavior. The new WTF time classes have some unit tests in TestWebKitAPI. * fileapi/ThreadableBlobRegistry.cpp: (WebCore::ThreadableBlobRegistry::blobSize): * platform/MainThreadSharedTimer.h: * platform/SharedTimer.h: * platform/ThreadTimers.cpp: (WebCore::ThreadTimers::updateSharedTimer): * platform/cf/MainThreadSharedTimerCF.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/efl/MainThreadSharedTimerEfl.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/glib/MainThreadSharedTimerGLib.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/win/MainThreadSharedTimerWin.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * workers/WorkerRunLoop.cpp: (WebCore::WorkerRunLoop::runInMode): Source/WebKit2: * Platform/IPC/Connection.cpp: (IPC::Connection::SyncMessageState::wait): (IPC::Connection::sendMessage): (IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting): (IPC::Connection::waitForMessage): (IPC::Connection::sendSyncMessage): (IPC::Connection::waitForSyncReply): * Platform/IPC/Connection.h: (IPC::Connection::sendSync): (IPC::Connection::waitForAndDispatchImmediately): * Platform/IPC/MessageSender.h: (IPC::MessageSender::sendSync): * UIProcess/ChildProcessProxy.h: (WebKit::ChildProcessProxy::sendSync): * UIProcess/Network/NetworkProcessProxy.cpp: (WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently): * UIProcess/Storage/StorageManager.cpp: (WebKit::StorageManager::applicationWillTerminate): * UIProcess/WebProcessProxy.cpp: (WebKit::WebProcessProxy::sendProcessWillSuspendImminently): * UIProcess/WebResourceLoadStatisticsStore.cpp: (WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate): * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h: * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm: (-[WKOneShotDisplayLinkHandler displayLinkFired:]): (WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree): (WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay): (WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm: (WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/WKImmediateActionController.mm: (-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]): * UIProcess/mac/WebPageProxyMac.mm: (WebKit::WebPageProxy::stringSelectionForPasteboard): (WebKit::WebPageProxy::dataSelectionForPasteboard): (WebKit::WebPageProxy::readSelectionFromPasteboard): (WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent): (WebKit::WebPageProxy::acceptsFirstMouse): * WebProcess/WebCoreSupport/WebChromeClient.cpp: (WebKit::WebChromeClient::runBeforeUnloadConfirmPanel): (WebKit::WebChromeClient::runJavaScriptAlert): (WebKit::WebChromeClient::runJavaScriptConfirm): (WebKit::WebChromeClient::runJavaScriptPrompt): (WebKit::WebChromeClient::print): (WebKit::WebChromeClient::exceededDatabaseQuota): (WebKit::WebChromeClient::reachedApplicationCacheOriginQuota): * WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp: (WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse): * WebProcess/WebPage/WebPage.cpp: (WebKit::WebPage::postSynchronousMessageForTesting): Source/WTF: We used to use 'double' for all time measurements. Sometimes it was milliseconds, sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time since some epoch. When we spoke of time since epoch, we either used a monotonic clock or a wall clock. The type - always 'double' - never told us what kind of time we had, even though there were roughly six of them (sec interval, ms interval, sec since epoch on wall, ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic). At some point, we thought that it would be a good idea to replace these doubles with std::chrono. But since replacing some things with std::chrono, we found it to be terribly inconvenient: - Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally duration_cast would not even be a thing. - No overflow protection. std::chrono uses integers by default and using anything else is clumsy. But the integer math is done without regard for the rough edges of integer math, so any cast between std::chrono types risks overflow. Any comparison risks overflow because it may do conversions silently. We have even found bugs where some C++ implementations had more overflows than others, which ends up being a special kind of hell. In many cases, the overflow also has nasal demons. It's an error to represent time using integers. It would have been excusable back when floating point math was not guaranteed to be supported on all platforms, but that would have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit for floating point: - Floating point preserves precision under multiplication in all but extreme cases, so using floating point for time means that unit conversions are almost completely lossless. This means that we don't have to think very hard about what units to use. In this patch, we use seconds almost everywhere. We only convert at boundaries, like an API boundary that wants something other than seconds. - Floating point makes it easy to reason about infinity, which is something that time code wants to do a lot. Example: when would you like to timeout? Infinity please! This is the most elegant way of having an API support both a timeout variant and a no-timeout variant. - Floating point does well-understood things when math goes wrong, and these things are pretty well optimized to match what a mathematician would do when computing with real numbers represented using scientific notation with a finite number of significant digits. This means that time math under floating point looks like normal math. On the other hand, std::chrono time math looks like garbage because you have to always check for multiple possible UB corners whenever you touch large integers. Integers that represent time are very likely to be large and you don't have to do much to overflow them. At this time, based on the number of bugs we have already seen due to chrono overflows, I am not certain that we even understand what are all of the corner cases that we should even check for. This patch introduces a new set of timekeeping classes that are all based on double, and all internally use seconds. These classes support algebraic typing. The classes are: - Seconds: this is for measuring a duration. - WallTime: time since epoch according to a wall clock (aka real time clock). - MonotonicTime: time since epoch according to a monotonic clock. - ClockType: enum that says either Wall or Monotonic. - TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a wall time or a monotonic time. All of these classes behave like C++ values and are cheap to copy around since they are very nearly POD. This supports comprehensive conversions between the various time types. Most of this is by way of algebra. Here are just some of the rules we recognize: WallTime = WallTime + Seconds Seconds = WallTime - WallTime MonotonicTime = MonotonicTime + Seconds etc... We support negative, infinite, and NaN times because math. We support conversions between MonotonicTime and WallTime, like: WallTime wt = mt.approximateWallTime() This is called this "approximate" because the only way to do it is to get the current time on both clocks and convert relative to that. Many of our APIs would be happy using whatever notion of time the user wanted to use. For those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This means that if you use a WallTime with Condition::waitUntil, then Condition's internal logic for when it should wake up makes its decision based on the current WallTime - but if you use MonotonicTime then waitUntil will make its decision based on current MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono did not have the concept of a dynamic clock type. This patch does not include conversions between std::chrono and these new time classes, because past experience shows that we're quite bad at getting conversions between std::chrono and anything else right. Also, I didn't need such conversion code because this patch only converts code that transitively touches ParkingLot and Condition. It was easy to get all of that code onto the new time classes. * WTF.xcodeproj/project.pbxproj: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/ClockType.cpp: Added. (WTF::printInternal): * wtf/ClockType.h: Added. * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::waitFor): (WTF::ConditionBase::wait): (WTF::ConditionBase::waitUntilWallClockSeconds): Deleted. (WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted. (WTF::ConditionBase::waitForSeconds): Deleted. (WTF::ConditionBase::waitForSecondsImpl): Deleted. (WTF::ConditionBase::waitForImpl): Deleted. (WTF::ConditionBase::absoluteFromRelative): Deleted. * wtf/CrossThreadQueue.h: (WTF::CrossThreadQueue<DataType>::waitForMessage): * wtf/CurrentTime.cpp: (WTF::sleep): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): Deleted. * wtf/MonotonicTime.cpp: Added. (WTF::MonotonicTime::now): (WTF::MonotonicTime::approximateWallTime): (WTF::MonotonicTime::dump): (WTF::MonotonicTime::sleep): * wtf/MonotonicTime.h: Added. (WTF::MonotonicTime::MonotonicTime): (WTF::MonotonicTime::fromRawDouble): (WTF::MonotonicTime::infinity): (WTF::MonotonicTime::secondsSinceEpoch): (WTF::MonotonicTime::approximateMonotonicTime): (WTF::MonotonicTime::operator bool): (WTF::MonotonicTime::operator+): (WTF::MonotonicTime::operator-): (WTF::MonotonicTime::operator+=): (WTF::MonotonicTime::operator-=): (WTF::MonotonicTime::operator==): (WTF::MonotonicTime::operator!=): (WTF::MonotonicTime::operator<): (WTF::MonotonicTime::operator>): (WTF::MonotonicTime::operator<=): (WTF::MonotonicTime::operator>=): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkCount): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): * wtf/Seconds.cpp: Added. (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::dump): (WTF::Seconds::sleep): * wtf/Seconds.h: Added. (WTF::Seconds::Seconds): (WTF::Seconds::value): (WTF::Seconds::seconds): (WTF::Seconds::milliseconds): (WTF::Seconds::microseconds): (WTF::Seconds::nanoseconds): (WTF::Seconds::fromMilliseconds): (WTF::Seconds::fromMicroseconds): (WTF::Seconds::fromNanoseconds): (WTF::Seconds::infinity): (WTF::Seconds::operator bool): (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::operator*): (WTF::Seconds::operator/): (WTF::Seconds::operator+=): (WTF::Seconds::operator-=): (WTF::Seconds::operator*=): (WTF::Seconds::operator/=): (WTF::Seconds::operator==): (WTF::Seconds::operator!=): (WTF::Seconds::operator<): (WTF::Seconds::operator>): (WTF::Seconds::operator<=): (WTF::Seconds::operator>=): * wtf/TimeWithDynamicClockType.cpp: Added. (WTF::TimeWithDynamicClockType::now): (WTF::TimeWithDynamicClockType::nowWithSameClock): (WTF::TimeWithDynamicClockType::wallTime): (WTF::TimeWithDynamicClockType::monotonicTime): (WTF::TimeWithDynamicClockType::approximateWallTime): (WTF::TimeWithDynamicClockType::approximateMonotonicTime): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator<): (WTF::TimeWithDynamicClockType::operator>): (WTF::TimeWithDynamicClockType::operator<=): (WTF::TimeWithDynamicClockType::operator>=): (WTF::TimeWithDynamicClockType::dump): (WTF::TimeWithDynamicClockType::sleep): * wtf/TimeWithDynamicClockType.h: Added. (WTF::TimeWithDynamicClockType::TimeWithDynamicClockType): (WTF::TimeWithDynamicClockType::fromRawDouble): (WTF::TimeWithDynamicClockType::secondsSinceEpoch): (WTF::TimeWithDynamicClockType::clockType): (WTF::TimeWithDynamicClockType::withSameClockAndRawDouble): (WTF::TimeWithDynamicClockType::operator bool): (WTF::TimeWithDynamicClockType::operator+): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator+=): (WTF::TimeWithDynamicClockType::operator-=): (WTF::TimeWithDynamicClockType::operator==): (WTF::TimeWithDynamicClockType::operator!=): * wtf/WallTime.cpp: Added. (WTF::WallTime::now): (WTF::WallTime::approximateMonotonicTime): (WTF::WallTime::dump): (WTF::WallTime::sleep): * wtf/WallTime.h: Added. (WTF::WallTime::WallTime): (WTF::WallTime::fromRawDouble): (WTF::WallTime::infinity): (WTF::WallTime::secondsSinceEpoch): (WTF::WallTime::approximateWallTime): (WTF::WallTime::operator bool): (WTF::WallTime::operator+): (WTF::WallTime::operator-): (WTF::WallTime::operator+=): (WTF::WallTime::operator-=): (WTF::WallTime::operator==): (WTF::WallTime::operator!=): (WTF::WallTime::operator<): (WTF::WallTime::operator>): (WTF::WallTime::operator<=): (WTF::WallTime::operator>=): * wtf/threads/BinarySemaphore.cpp: (WTF::BinarySemaphore::wait): * wtf/threads/BinarySemaphore.h: Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp: (TestWebKitAPI::ToUpperConverter::stopProducing): (TestWebKitAPI::ToUpperConverter::stopConsuming): * TestWebKitAPI/Tests/WTF/Time.cpp: Added. (WTF::operator<<): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/182152@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
#include <wtf/TimeWithDynamicClockType.h>
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
namespace WTF {
[WTF] Implement WTF::ThreadGroup https://bugs.webkit.org/show_bug.cgi?id=174081 Reviewed by Mark Lam. Source/JavaScriptCore: Large part of MachineThreads are now removed and replaced with WTF::ThreadGroup. And SamplingProfiler and others interact with WTF::Thread directly. * API/tests/ExecutionTimeLimitTest.cpp: * heap/MachineStackMarker.cpp: (JSC::MachineThreads::MachineThreads): (JSC::captureStack): (JSC::MachineThreads::tryCopyOtherThreadStack): (JSC::MachineThreads::tryCopyOtherThreadStacks): (JSC::MachineThreads::gatherConservativeRoots): (JSC::ActiveMachineThreadsManager::Locker::Locker): Deleted. (JSC::ActiveMachineThreadsManager::add): Deleted. (JSC::ActiveMachineThreadsManager::remove): Deleted. (JSC::ActiveMachineThreadsManager::contains): Deleted. (JSC::ActiveMachineThreadsManager::ActiveMachineThreadsManager): Deleted. (JSC::activeMachineThreadsManager): Deleted. (JSC::MachineThreads::~MachineThreads): Deleted. (JSC::MachineThreads::addCurrentThread): Deleted. (): Deleted. (JSC::MachineThreads::removeThread): Deleted. (JSC::MachineThreads::removeThreadIfFound): Deleted. (JSC::MachineThreads::MachineThread::MachineThread): Deleted. (JSC::MachineThreads::MachineThread::getRegisters): Deleted. (JSC::MachineThreads::MachineThread::Registers::stackPointer): Deleted. (JSC::MachineThreads::MachineThread::Registers::framePointer): Deleted. (JSC::MachineThreads::MachineThread::Registers::instructionPointer): Deleted. (JSC::MachineThreads::MachineThread::Registers::llintPC): Deleted. (JSC::MachineThreads::MachineThread::captureStack): Deleted. * heap/MachineStackMarker.h: (JSC::MachineThreads::addCurrentThread): (JSC::MachineThreads::getLock): (JSC::MachineThreads::threads): (JSC::MachineThreads::MachineThread::suspend): Deleted. (JSC::MachineThreads::MachineThread::resume): Deleted. (JSC::MachineThreads::MachineThread::threadID): Deleted. (JSC::MachineThreads::MachineThread::stackBase): Deleted. (JSC::MachineThreads::MachineThread::stackEnd): Deleted. (JSC::MachineThreads::threadsListHead): Deleted. * runtime/SamplingProfiler.cpp: (JSC::FrameWalker::isValidFramePointer): (JSC::SamplingProfiler::SamplingProfiler): (JSC::SamplingProfiler::takeSample): (JSC::SamplingProfiler::noticeCurrentThreadAsJSCExecutionThread): * runtime/SamplingProfiler.h: * wasm/WasmMachineThreads.cpp: (JSC::Wasm::resetInstructionCacheOnAllThreads): Source/WebCore: * page/ResourceUsageThread.h: Source/WebKit: * Shared/AsyncRequest.h: Source/WTF: This patch implements WTF::ThreadGroup. It implements core of JSC::MachineThreads with more reliable way. JSC::MachineThreads was complicated because of managing dead threads. Each JSC::MachineThreads has its own TLS with a registered destructor. And everytime a thread dies, the registered TLS destructor is called. And this destructor will remove the current dying thread from JSC::MachineThreads. However the above implementation is tricky. And each JSC::MachineThreads requires own TLS space, which is not considered in WTF's Windows ThreadSpecific implementation. Current design works well since we only have small number of MachineThreads right now. Instead, we use more reliable way. After introducing WTF::Thread, WTF::Thread has WTF::Thread::didExit, which is called when associated TLS (with WTF::Thread) is destroyed. We leverage this mechanism to remove WTF::Thread from MachineThreads. This patch introduces WTF::ThreadGroup. It is tightly integrated with WTF::Thread: WTF::Thread knows ThreadGroups which includes this thread. And WTF::ThreadGroup of course knows WTF::Threads added to it. WTF::Thread::didExit carefully remove itself from WTF::ThreadGroups. The most important part of this patch is locking. WTF::Thread can die. And WTF::ThreadGroup can die. If we take a design using two fine grain locks in WTF::Thread and WTF::ThreadGroup, we easily encounter dead lock. Consider the following case. 1. When adding WTF::Thread (TH) to WTF::ThreadGroup (THG), we first hold a lock of THG, and hold a lock of TH (locking order is THG -> TH). 2. When TH dies, TH need to hold a lock of TH to iterate THGs. And we hold a lock of THG to unregister TH from it (locking order is TH -> THG). 3. When suspending and resuming THs in THG, we first hold a lock of THG. And then, we hold a lock of TH to suspend and resume it (locking order is THG -> TH). 4. When destroying THG, we need to hold a lock of TH to unregister THG from TH. We can hold a lock of THG before that (locking order is THG -> TH). Then, it easily causes dead lock. We cannot swap the locking order of (2) since iterating THG requires a lock of TH. To solve this problem, we use std::shared_ptr and std::weak_ptr. 1. When adding WTF::Thread (TH) to WTF::ThreadGroup (THG), we first hold THG, and hold a lock of TH. (THG -> TH) 2. When TH dies, TH first hold lock of TH. And we use std::weak_ptr<>::lock() to retain non-destructed ThreadGroups. If some of ThreadGroups are dying, we just ignore them. It is ok because such a ThreadGroup will be destructed. So we do not need to unregister this thread from such a ThreadGroup. Then, we have Vector<std::shared_ptr<ThreadGroup>>. So we unlock a lock of TH. To unregister a thread from thread group, we first hold a lock of THG and then hold a lock of TH. Both lifetime is ensured: THG is retained by std::shared_ptr. And TH is itself. (TH), (THG -> TH). 3. When suspending and resuming THs in THG, we first hold a lock of THG. And then, we hold a lock of TH to suspend and resume it (THG -> TH). 4. When destroying THG, we hold a lock of THG. And hold a lock of TH. During holding THG's lock, registered thread never dies because (2) holds THG lock. (THG -> TH). We also fix suspend and resume locking mechanism to avoid dead lock. We should hold the global lock when suspending and resuming. If we use per-thread lock, the suspended thread can hold the lock of the other threads. It causes dead lock. * WTF.xcodeproj/project.pbxproj: * wtf/AutomaticThread.cpp: * wtf/CMakeLists.txt: * wtf/CrossThreadCopier.h: * wtf/ParkingLot.h: * wtf/ThreadGroup.cpp: Copied from Source/JavaScriptCore/wasm/WasmMachineThreads.cpp. (WTF::ThreadGroup::~ThreadGroup): (WTF::ThreadGroup::add): (WTF::ThreadGroup::addCurrentThread): * wtf/ThreadGroup.h: Copied from Source/JavaScriptCore/wasm/WasmMachineThreads.cpp. (WTF::ThreadGroup::create): (WTF::ThreadGroup::threads): (WTF::ThreadGroup::getLock): (WTF::ThreadGroup::weakFromThis): * wtf/Threading.cpp: (WTF::shouldRemoveThreadFromThreadGroup): (WTF::Thread::didExit): (WTF::Thread::addToThreadGroup): (WTF::Thread::removeFromThreadGroup): * wtf/Threading.h: * wtf/ThreadingPthreads.cpp: (WTF::Thread::resume): (WTF::Thread::getRegisters): * wtf/ThreadingWin.cpp: (WTF::Thread::resume): (WTF::Thread::getRegisters): Tools: Add WTF::ThreadGroup tests. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/ThreadGroup.cpp: Added. (TestWebKitAPI::testThreadGroup): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/191464@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@219653 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-07-19 08:43:57 +00:00
class Thread;
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
class ParkingLot {
[WTF][JSC] Make JSC and WTF aggressively-fast-malloced https://bugs.webkit.org/show_bug.cgi?id=200611 Reviewed by Saam Barati. Source/JavaScriptCore: This patch aggressively puts many classes into FastMalloc. In JSC side, we grep `std::make_unique` etc. to find potentially system-malloc-allocated classes. After this patch, all the JSC related allocations in JetStream2 cli is done from bmalloc. In the future, it would be nice that we add `WTF::makeUnique<T>` helper function and throw a compile error if `T` is not FastMalloc annotated[1]. Putting WebKit classes in FastMalloc has many benefits. 1. Simply, it is fast. 2. vmmap can tell the amount of memory used for WebKit. 3. bmalloc can isolate WebKit memory allocation from the rest of the world. This is useful since we can know more about what component is corrupting the memory from the memory corruption crash. [1]: https://bugs.webkit.org/show_bug.cgi?id=200620 * API/ObjCCallbackFunction.mm: * assembler/AbstractMacroAssembler.h: * b3/B3PhiChildren.h: * b3/air/AirAllocateRegistersAndStackAndGenerateCode.h: * b3/air/AirDisassembler.h: * bytecode/AccessCaseSnippetParams.h: * bytecode/CallVariant.h: * bytecode/DeferredSourceDump.h: * bytecode/ExecutionCounter.h: * bytecode/GetByIdStatus.h: * bytecode/GetByIdVariant.h: * bytecode/InByIdStatus.h: * bytecode/InByIdVariant.h: * bytecode/InstanceOfStatus.h: * bytecode/InstanceOfVariant.h: * bytecode/PutByIdStatus.h: * bytecode/PutByIdVariant.h: * bytecode/ValueProfile.h: * dfg/DFGAbstractInterpreter.h: * dfg/DFGByteCodeParser.cpp: (JSC::DFG::ByteCodeParser::newVariableAccessData): * dfg/DFGFlowIndexing.h: * dfg/DFGFlowMap.h: * dfg/DFGLiveCatchVariablePreservationPhase.cpp: (JSC::DFG::LiveCatchVariablePreservationPhase::newVariableAccessData): * dfg/DFGMaximalFlushInsertionPhase.cpp: (JSC::DFG::MaximalFlushInsertionPhase::newVariableAccessData): * dfg/DFGOSRExit.h: * dfg/DFGSpeculativeJIT.h: * dfg/DFGVariableAccessData.h: * disassembler/ARM64/A64DOpcode.h: * inspector/remote/socket/RemoteInspectorMessageParser.h: * inspector/remote/socket/RemoteInspectorSocket.h: * inspector/remote/socket/RemoteInspectorSocketEndpoint.h: * jit/PCToCodeOriginMap.h: * runtime/BasicBlockLocation.h: * runtime/DoublePredictionFuzzerAgent.h: * runtime/JSRunLoopTimer.h: * runtime/PromiseDeferredTimer.h: (JSC::PromiseDeferredTimer::create): PromiseDeferredTimer should be allocated as `Ref<>` instead of `std::unique_ptr` since it is inheriting ThreadSafeRefCounted<>. Holding such a class with std::unique_ptr could lead to potentially dangerous operations (like, someone holds it with Ref<> while it is deleted by std::unique_ptr<>). * runtime/RandomizingFuzzerAgent.h: * runtime/SymbolTable.h: * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * tools/JSDollarVM.cpp: * tools/SigillCrashAnalyzer.cpp: * wasm/WasmFormat.h: * wasm/WasmMemory.cpp: * wasm/WasmSignature.h: * yarr/YarrJIT.h: Source/WebCore: Changed the accessor since we changed std::unique_ptr to Ref for this field. No behavior change. * bindings/js/WorkerScriptController.cpp: (WebCore::WorkerScriptController::addTimerSetNotification): (WebCore::WorkerScriptController::removeTimerSetNotification): Source/WTF: WTF has many data structures, in particular, containers. And these containers can be allocated like `std::make_unique<Container>()`. Without WTF_MAKE_FAST_ALLOCATED, this container itself is allocated from the system malloc. This patch attaches WTF_MAKE_FAST_ALLOCATED more aggressively not to allocate them from the system malloc. And we add some `final` to containers and classes that would be never inherited. * wtf/Assertions.cpp: * wtf/Atomics.h: * wtf/AutodrainedPool.h: * wtf/Bag.h: (WTF::Bag::Bag): Deleted. (WTF::Bag::~Bag): Deleted. (WTF::Bag::clear): Deleted. (WTF::Bag::add): Deleted. (WTF::Bag::iterator::iterator): Deleted. (WTF::Bag::iterator::operator! const): Deleted. (WTF::Bag::iterator::operator* const): Deleted. (WTF::Bag::iterator::operator++): Deleted. (WTF::Bag::iterator::operator== const): Deleted. (WTF::Bag::iterator::operator!= const): Deleted. (WTF::Bag::begin): Deleted. (WTF::Bag::begin const): Deleted. (WTF::Bag::end const): Deleted. (WTF::Bag::isEmpty const): Deleted. (WTF::Bag::unwrappedHead const): Deleted. * wtf/BitVector.h: (WTF::BitVector::BitVector): Deleted. (WTF::BitVector::~BitVector): Deleted. (WTF::BitVector::operator=): Deleted. (WTF::BitVector::size const): Deleted. (WTF::BitVector::ensureSize): Deleted. (WTF::BitVector::quickGet const): Deleted. (WTF::BitVector::quickSet): Deleted. (WTF::BitVector::quickClear): Deleted. (WTF::BitVector::get const): Deleted. (WTF::BitVector::contains const): Deleted. (WTF::BitVector::set): Deleted. (WTF::BitVector::add): Deleted. (WTF::BitVector::ensureSizeAndSet): Deleted. (WTF::BitVector::clear): Deleted. (WTF::BitVector::remove): Deleted. (WTF::BitVector::merge): Deleted. (WTF::BitVector::filter): Deleted. (WTF::BitVector::exclude): Deleted. (WTF::BitVector::bitCount const): Deleted. (WTF::BitVector::isEmpty const): Deleted. (WTF::BitVector::findBit const): Deleted. (WTF::BitVector::isEmptyValue const): Deleted. (WTF::BitVector::isDeletedValue const): Deleted. (WTF::BitVector::isEmptyOrDeletedValue const): Deleted. (WTF::BitVector::operator== const): Deleted. (WTF::BitVector::hash const): Deleted. (WTF::BitVector::iterator::iterator): Deleted. (WTF::BitVector::iterator::operator* const): Deleted. (WTF::BitVector::iterator::operator++): Deleted. (WTF::BitVector::iterator::isAtEnd const): Deleted. (WTF::BitVector::iterator::operator== const): Deleted. (WTF::BitVector::iterator::operator!= const): Deleted. (WTF::BitVector::begin const): Deleted. (WTF::BitVector::end const): Deleted. (WTF::BitVector::bitsInPointer): Deleted. (WTF::BitVector::maxInlineBits): Deleted. (WTF::BitVector::byteCount): Deleted. (WTF::BitVector::makeInlineBits): Deleted. (WTF::BitVector::cleanseInlineBits): Deleted. (WTF::BitVector::bitCount): Deleted. (WTF::BitVector::findBitFast const): Deleted. (WTF::BitVector::findBitSimple const): Deleted. (WTF::BitVector::OutOfLineBits::numBits const): Deleted. (WTF::BitVector::OutOfLineBits::numWords const): Deleted. (WTF::BitVector::OutOfLineBits::bits): Deleted. (WTF::BitVector::OutOfLineBits::bits const): Deleted. (WTF::BitVector::OutOfLineBits::OutOfLineBits): Deleted. (WTF::BitVector::isInline const): Deleted. (WTF::BitVector::outOfLineBits const): Deleted. (WTF::BitVector::outOfLineBits): Deleted. (WTF::BitVector::bits): Deleted. (WTF::BitVector::bits const): Deleted. * wtf/Bitmap.h: (WTF::Bitmap::size): Deleted. (WTF::Bitmap::iterator::iterator): Deleted. (WTF::Bitmap::iterator::operator* const): Deleted. (WTF::Bitmap::iterator::operator++): Deleted. (WTF::Bitmap::iterator::operator== const): Deleted. (WTF::Bitmap::iterator::operator!= const): Deleted. (WTF::Bitmap::begin const): Deleted. (WTF::Bitmap::end const): Deleted. * wtf/Box.h: * wtf/BumpPointerAllocator.h: * wtf/CPUTime.h: * wtf/CheckedBoolean.h: * wtf/CommaPrinter.h: (WTF::CommaPrinter::CommaPrinter): Deleted. (WTF::CommaPrinter::dump const): Deleted. (WTF::CommaPrinter::didPrint const): Deleted. * wtf/CompactPointerTuple.h: (WTF::CompactPointerTuple::encodeType): Deleted. (WTF::CompactPointerTuple::decodeType): Deleted. (WTF::CompactPointerTuple::CompactPointerTuple): Deleted. (WTF::CompactPointerTuple::pointer const): Deleted. (WTF::CompactPointerTuple::setPointer): Deleted. (WTF::CompactPointerTuple::type const): Deleted. (WTF::CompactPointerTuple::setType): Deleted. * wtf/CompilationThread.h: (WTF::CompilationScope::CompilationScope): Deleted. (WTF::CompilationScope::~CompilationScope): Deleted. (WTF::CompilationScope::leaveEarly): Deleted. * wtf/CompletionHandler.h: (WTF::CompletionHandler<Out): (WTF::Detail::CallableWrapper<CompletionHandler<Out): (WTF::CompletionHandlerCallingScope::CompletionHandlerCallingScope): Deleted. (WTF::CompletionHandlerCallingScope::~CompletionHandlerCallingScope): Deleted. (WTF::CompletionHandlerCallingScope::CompletionHandler<void): Deleted. * wtf/ConcurrentBuffer.h: (WTF::ConcurrentBuffer::ConcurrentBuffer): Deleted. (WTF::ConcurrentBuffer::~ConcurrentBuffer): Deleted. (WTF::ConcurrentBuffer::growExact): Deleted. (WTF::ConcurrentBuffer::grow): Deleted. (WTF::ConcurrentBuffer::array const): Deleted. (WTF::ConcurrentBuffer::operator[]): Deleted. (WTF::ConcurrentBuffer::operator[] const): Deleted. (WTF::ConcurrentBuffer::createArray): Deleted. * wtf/ConcurrentPtrHashSet.h: (WTF::ConcurrentPtrHashSet::contains): Deleted. (WTF::ConcurrentPtrHashSet::add): Deleted. (WTF::ConcurrentPtrHashSet::size const): Deleted. (WTF::ConcurrentPtrHashSet::Table::maxLoad const): Deleted. (WTF::ConcurrentPtrHashSet::hash): Deleted. (WTF::ConcurrentPtrHashSet::cast): Deleted. (WTF::ConcurrentPtrHashSet::containsImpl const): Deleted. (WTF::ConcurrentPtrHashSet::addImpl): Deleted. * wtf/ConcurrentVector.h: (WTF::ConcurrentVector::~ConcurrentVector): Deleted. (WTF::ConcurrentVector::size const): Deleted. (WTF::ConcurrentVector::isEmpty const): Deleted. (WTF::ConcurrentVector::at): Deleted. (WTF::ConcurrentVector::at const): Deleted. (WTF::ConcurrentVector::operator[]): Deleted. (WTF::ConcurrentVector::operator[] const): Deleted. (WTF::ConcurrentVector::first): Deleted. (WTF::ConcurrentVector::first const): Deleted. (WTF::ConcurrentVector::last): Deleted. (WTF::ConcurrentVector::last const): Deleted. (WTF::ConcurrentVector::takeLast): Deleted. (WTF::ConcurrentVector::append): Deleted. (WTF::ConcurrentVector::alloc): Deleted. (WTF::ConcurrentVector::removeLast): Deleted. (WTF::ConcurrentVector::grow): Deleted. (WTF::ConcurrentVector::begin): Deleted. (WTF::ConcurrentVector::end): Deleted. (WTF::ConcurrentVector::segmentExistsFor): Deleted. (WTF::ConcurrentVector::segmentFor): Deleted. (WTF::ConcurrentVector::subscriptFor): Deleted. (WTF::ConcurrentVector::ensureSegmentsFor): Deleted. (WTF::ConcurrentVector::ensureSegment): Deleted. (WTF::ConcurrentVector::allocateSegment): Deleted. * wtf/Condition.h: (WTF::Condition::waitUntil): Deleted. (WTF::Condition::waitFor): Deleted. (WTF::Condition::wait): Deleted. (WTF::Condition::notifyOne): Deleted. (WTF::Condition::notifyAll): Deleted. * wtf/CountingLock.h: (WTF::CountingLock::LockHooks::lockHook): Deleted. (WTF::CountingLock::LockHooks::unlockHook): Deleted. (WTF::CountingLock::LockHooks::parkHook): Deleted. (WTF::CountingLock::LockHooks::handoffHook): Deleted. (WTF::CountingLock::tryLock): Deleted. (WTF::CountingLock::lock): Deleted. (WTF::CountingLock::unlock): Deleted. (WTF::CountingLock::isHeld const): Deleted. (WTF::CountingLock::isLocked const): Deleted. (WTF::CountingLock::Count::operator bool const): Deleted. (WTF::CountingLock::Count::operator== const): Deleted. (WTF::CountingLock::Count::operator!= const): Deleted. (WTF::CountingLock::tryOptimisticRead): Deleted. (WTF::CountingLock::validate): Deleted. (WTF::CountingLock::doOptimizedRead): Deleted. (WTF::CountingLock::tryOptimisticFencelessRead): Deleted. (WTF::CountingLock::fencelessValidate): Deleted. (WTF::CountingLock::doOptimizedFencelessRead): Deleted. (WTF::CountingLock::getCount): Deleted. * wtf/CrossThreadQueue.h: * wtf/CrossThreadTask.h: * wtf/CryptographicallyRandomNumber.cpp: * wtf/DataMutex.h: * wtf/DateMath.h: * wtf/Deque.h: (WTF::Deque::size const): Deleted. (WTF::Deque::isEmpty const): Deleted. (WTF::Deque::begin): Deleted. (WTF::Deque::end): Deleted. (WTF::Deque::begin const): Deleted. (WTF::Deque::end const): Deleted. (WTF::Deque::rbegin): Deleted. (WTF::Deque::rend): Deleted. (WTF::Deque::rbegin const): Deleted. (WTF::Deque::rend const): Deleted. (WTF::Deque::first): Deleted. (WTF::Deque::first const): Deleted. (WTF::Deque::last): Deleted. (WTF::Deque::last const): Deleted. (WTF::Deque::append): Deleted. * wtf/Dominators.h: * wtf/DoublyLinkedList.h: * wtf/Expected.h: * wtf/FastBitVector.h: * wtf/FileMetadata.h: * wtf/FileSystem.h: * wtf/GraphNodeWorklist.h: * wtf/GregorianDateTime.h: (WTF::GregorianDateTime::GregorianDateTime): Deleted. (WTF::GregorianDateTime::year const): Deleted. (WTF::GregorianDateTime::month const): Deleted. (WTF::GregorianDateTime::yearDay const): Deleted. (WTF::GregorianDateTime::monthDay const): Deleted. (WTF::GregorianDateTime::weekDay const): Deleted. (WTF::GregorianDateTime::hour const): Deleted. (WTF::GregorianDateTime::minute const): Deleted. (WTF::GregorianDateTime::second const): Deleted. (WTF::GregorianDateTime::utcOffset const): Deleted. (WTF::GregorianDateTime::isDST const): Deleted. (WTF::GregorianDateTime::setYear): Deleted. (WTF::GregorianDateTime::setMonth): Deleted. (WTF::GregorianDateTime::setYearDay): Deleted. (WTF::GregorianDateTime::setMonthDay): Deleted. (WTF::GregorianDateTime::setWeekDay): Deleted. (WTF::GregorianDateTime::setHour): Deleted. (WTF::GregorianDateTime::setMinute): Deleted. (WTF::GregorianDateTime::setSecond): Deleted. (WTF::GregorianDateTime::setUtcOffset): Deleted. (WTF::GregorianDateTime::setIsDST): Deleted. (WTF::GregorianDateTime::operator tm const): Deleted. (WTF::GregorianDateTime::copyFrom): Deleted. * wtf/HashTable.h: * wtf/Hasher.h: * wtf/HexNumber.h: * wtf/Indenter.h: * wtf/IndexMap.h: * wtf/IndexSet.h: * wtf/IndexSparseSet.h: * wtf/IndexedContainerIterator.h: * wtf/Insertion.h: * wtf/IteratorAdaptors.h: * wtf/IteratorRange.h: * wtf/KeyValuePair.h: * wtf/ListHashSet.h: (WTF::ListHashSet::begin): Deleted. (WTF::ListHashSet::end): Deleted. (WTF::ListHashSet::begin const): Deleted. (WTF::ListHashSet::end const): Deleted. (WTF::ListHashSet::random): Deleted. (WTF::ListHashSet::random const): Deleted. (WTF::ListHashSet::rbegin): Deleted. (WTF::ListHashSet::rend): Deleted. (WTF::ListHashSet::rbegin const): Deleted. (WTF::ListHashSet::rend const): Deleted. * wtf/Liveness.h: * wtf/LocklessBag.h: (WTF::LocklessBag::LocklessBag): Deleted. (WTF::LocklessBag::add): Deleted. (WTF::LocklessBag::iterate): Deleted. (WTF::LocklessBag::consumeAll): Deleted. (WTF::LocklessBag::consumeAllWithNode): Deleted. (WTF::LocklessBag::~LocklessBag): Deleted. * wtf/LoggingHashID.h: * wtf/MD5.h: * wtf/MachSendRight.h: * wtf/MainThreadData.h: * wtf/Markable.h: * wtf/MediaTime.h: * wtf/MemoryPressureHandler.h: * wtf/MessageQueue.h: (WTF::MessageQueue::MessageQueue): Deleted. * wtf/MetaAllocator.h: * wtf/MonotonicTime.h: (WTF::MonotonicTime::MonotonicTime): Deleted. (WTF::MonotonicTime::fromRawSeconds): Deleted. (WTF::MonotonicTime::infinity): Deleted. (WTF::MonotonicTime::nan): Deleted. (WTF::MonotonicTime::secondsSinceEpoch const): Deleted. (WTF::MonotonicTime::approximateMonotonicTime const): Deleted. (WTF::MonotonicTime::operator bool const): Deleted. (WTF::MonotonicTime::operator+ const): Deleted. (WTF::MonotonicTime::operator- const): Deleted. (WTF::MonotonicTime::operator% const): Deleted. (WTF::MonotonicTime::operator+=): Deleted. (WTF::MonotonicTime::operator-=): Deleted. (WTF::MonotonicTime::operator== const): Deleted. (WTF::MonotonicTime::operator!= const): Deleted. (WTF::MonotonicTime::operator< const): Deleted. (WTF::MonotonicTime::operator> const): Deleted. (WTF::MonotonicTime::operator<= const): Deleted. (WTF::MonotonicTime::operator>= const): Deleted. (WTF::MonotonicTime::isolatedCopy const): Deleted. (WTF::MonotonicTime::encode const): Deleted. (WTF::MonotonicTime::decode): Deleted. * wtf/NaturalLoops.h: * wtf/NoLock.h: * wtf/OSAllocator.h: * wtf/OptionSet.h: * wtf/Optional.h: * wtf/OrderMaker.h: * wtf/Packed.h: (WTF::alignof): * wtf/PackedIntVector.h: (WTF::PackedIntVector::PackedIntVector): Deleted. (WTF::PackedIntVector::operator=): Deleted. (WTF::PackedIntVector::size const): Deleted. (WTF::PackedIntVector::ensureSize): Deleted. (WTF::PackedIntVector::resize): Deleted. (WTF::PackedIntVector::clearAll): Deleted. (WTF::PackedIntVector::get const): Deleted. (WTF::PackedIntVector::set): Deleted. (WTF::PackedIntVector::mask): Deleted. * wtf/PageBlock.h: * wtf/ParallelJobsOpenMP.h: * wtf/ParkingLot.h: * wtf/PriorityQueue.h: (WTF::PriorityQueue::size const): Deleted. (WTF::PriorityQueue::isEmpty const): Deleted. (WTF::PriorityQueue::enqueue): Deleted. (WTF::PriorityQueue::peek const): Deleted. (WTF::PriorityQueue::dequeue): Deleted. (WTF::PriorityQueue::decreaseKey): Deleted. (WTF::PriorityQueue::increaseKey): Deleted. (WTF::PriorityQueue::begin const): Deleted. (WTF::PriorityQueue::end const): Deleted. (WTF::PriorityQueue::isValidHeap const): Deleted. (WTF::PriorityQueue::parentOf): Deleted. (WTF::PriorityQueue::leftChildOf): Deleted. (WTF::PriorityQueue::rightChildOf): Deleted. (WTF::PriorityQueue::siftUp): Deleted. (WTF::PriorityQueue::siftDown): Deleted. * wtf/RandomDevice.h: * wtf/Range.h: * wtf/RangeSet.h: (WTF::RangeSet::RangeSet): Deleted. (WTF::RangeSet::~RangeSet): Deleted. (WTF::RangeSet::add): Deleted. (WTF::RangeSet::contains const): Deleted. (WTF::RangeSet::overlaps const): Deleted. (WTF::RangeSet::clear): Deleted. (WTF::RangeSet::dump const): Deleted. (WTF::RangeSet::dumpRaw const): Deleted. (WTF::RangeSet::begin const): Deleted. (WTF::RangeSet::end const): Deleted. (WTF::RangeSet::addAll): Deleted. (WTF::RangeSet::compact): Deleted. (WTF::RangeSet::overlapsNonEmpty): Deleted. (WTF::RangeSet::subsumesNonEmpty): Deleted. (WTF::RangeSet::findRange const): Deleted. * wtf/RecursableLambda.h: * wtf/RedBlackTree.h: (WTF::RedBlackTree::Node::successor const): Deleted. (WTF::RedBlackTree::Node::predecessor const): Deleted. (WTF::RedBlackTree::Node::successor): Deleted. (WTF::RedBlackTree::Node::predecessor): Deleted. (WTF::RedBlackTree::Node::reset): Deleted. (WTF::RedBlackTree::Node::parent const): Deleted. (WTF::RedBlackTree::Node::setParent): Deleted. (WTF::RedBlackTree::Node::left const): Deleted. (WTF::RedBlackTree::Node::setLeft): Deleted. (WTF::RedBlackTree::Node::right const): Deleted. (WTF::RedBlackTree::Node::setRight): Deleted. (WTF::RedBlackTree::Node::color const): Deleted. (WTF::RedBlackTree::Node::setColor): Deleted. (WTF::RedBlackTree::RedBlackTree): Deleted. (WTF::RedBlackTree::insert): Deleted. (WTF::RedBlackTree::remove): Deleted. (WTF::RedBlackTree::findExact const): Deleted. (WTF::RedBlackTree::findLeastGreaterThanOrEqual const): Deleted. (WTF::RedBlackTree::findGreatestLessThanOrEqual const): Deleted. (WTF::RedBlackTree::first const): Deleted. (WTF::RedBlackTree::last const): Deleted. (WTF::RedBlackTree::size): Deleted. (WTF::RedBlackTree::isEmpty): Deleted. (WTF::RedBlackTree::treeMinimum): Deleted. (WTF::RedBlackTree::treeMaximum): Deleted. (WTF::RedBlackTree::treeInsert): Deleted. (WTF::RedBlackTree::leftRotate): Deleted. (WTF::RedBlackTree::rightRotate): Deleted. (WTF::RedBlackTree::removeFixup): Deleted. * wtf/ResourceUsage.h: * wtf/RunLoop.cpp: * wtf/RunLoopTimer.h: * wtf/SHA1.h: * wtf/Seconds.h: (WTF::Seconds::Seconds): Deleted. (WTF::Seconds::value const): Deleted. (WTF::Seconds::minutes const): Deleted. (WTF::Seconds::seconds const): Deleted. (WTF::Seconds::milliseconds const): Deleted. (WTF::Seconds::microseconds const): Deleted. (WTF::Seconds::nanoseconds const): Deleted. (WTF::Seconds::minutesAs const): Deleted. (WTF::Seconds::secondsAs const): Deleted. (WTF::Seconds::millisecondsAs const): Deleted. (WTF::Seconds::microsecondsAs const): Deleted. (WTF::Seconds::nanosecondsAs const): Deleted. (WTF::Seconds::fromMinutes): Deleted. (WTF::Seconds::fromHours): Deleted. (WTF::Seconds::fromMilliseconds): Deleted. (WTF::Seconds::fromMicroseconds): Deleted. (WTF::Seconds::fromNanoseconds): Deleted. (WTF::Seconds::infinity): Deleted. (WTF::Seconds::nan): Deleted. (WTF::Seconds::operator bool const): Deleted. (WTF::Seconds::operator+ const): Deleted. (WTF::Seconds::operator- const): Deleted. (WTF::Seconds::operator* const): Deleted. (WTF::Seconds::operator/ const): Deleted. (WTF::Seconds::operator% const): Deleted. (WTF::Seconds::operator+=): Deleted. (WTF::Seconds::operator-=): Deleted. (WTF::Seconds::operator*=): Deleted. (WTF::Seconds::operator/=): Deleted. (WTF::Seconds::operator%=): Deleted. (WTF::Seconds::operator== const): Deleted. (WTF::Seconds::operator!= const): Deleted. (WTF::Seconds::operator< const): Deleted. (WTF::Seconds::operator> const): Deleted. (WTF::Seconds::operator<= const): Deleted. (WTF::Seconds::operator>= const): Deleted. (WTF::Seconds::isolatedCopy const): Deleted. (WTF::Seconds::encode const): Deleted. (WTF::Seconds::decode): Deleted. * wtf/SegmentedVector.h: (WTF::SegmentedVector::~SegmentedVector): Deleted. (WTF::SegmentedVector::size const): Deleted. (WTF::SegmentedVector::isEmpty const): Deleted. (WTF::SegmentedVector::at): Deleted. (WTF::SegmentedVector::at const): Deleted. (WTF::SegmentedVector::operator[]): Deleted. (WTF::SegmentedVector::operator[] const): Deleted. (WTF::SegmentedVector::first): Deleted. (WTF::SegmentedVector::first const): Deleted. (WTF::SegmentedVector::last): Deleted. (WTF::SegmentedVector::last const): Deleted. (WTF::SegmentedVector::takeLast): Deleted. (WTF::SegmentedVector::append): Deleted. (WTF::SegmentedVector::alloc): Deleted. (WTF::SegmentedVector::removeLast): Deleted. (WTF::SegmentedVector::grow): Deleted. (WTF::SegmentedVector::clear): Deleted. (WTF::SegmentedVector::begin): Deleted. (WTF::SegmentedVector::end): Deleted. (WTF::SegmentedVector::shrinkToFit): Deleted. (WTF::SegmentedVector::deleteAllSegments): Deleted. (WTF::SegmentedVector::segmentExistsFor): Deleted. (WTF::SegmentedVector::segmentFor): Deleted. (WTF::SegmentedVector::subscriptFor): Deleted. (WTF::SegmentedVector::ensureSegmentsFor): Deleted. (WTF::SegmentedVector::ensureSegment): Deleted. (WTF::SegmentedVector::allocateSegment): Deleted. * wtf/SetForScope.h: * wtf/SingleRootGraph.h: * wtf/SinglyLinkedList.h: * wtf/SmallPtrSet.h: * wtf/SpanningTree.h: * wtf/Spectrum.h: * wtf/StackBounds.h: * wtf/StackShot.h: * wtf/StackShotProfiler.h: * wtf/StackStats.h: * wtf/StackTrace.h: * wtf/StreamBuffer.h: * wtf/SynchronizedFixedQueue.h: (WTF::SynchronizedFixedQueue::create): Deleted. (WTF::SynchronizedFixedQueue::open): Deleted. (WTF::SynchronizedFixedQueue::close): Deleted. (WTF::SynchronizedFixedQueue::isOpen): Deleted. (WTF::SynchronizedFixedQueue::enqueue): Deleted. (WTF::SynchronizedFixedQueue::dequeue): Deleted. (WTF::SynchronizedFixedQueue::SynchronizedFixedQueue): Deleted. * wtf/SystemTracing.h: * wtf/ThreadGroup.h: (WTF::ThreadGroup::create): Deleted. (WTF::ThreadGroup::threads const): Deleted. (WTF::ThreadGroup::getLock): Deleted. (WTF::ThreadGroup::weakFromThis): Deleted. * wtf/ThreadSpecific.h: * wtf/ThreadingPrimitives.h: (WTF::Mutex::impl): Deleted. * wtf/TimeWithDynamicClockType.h: (WTF::TimeWithDynamicClockType::TimeWithDynamicClockType): Deleted. (WTF::TimeWithDynamicClockType::fromRawSeconds): Deleted. (WTF::TimeWithDynamicClockType::secondsSinceEpoch const): Deleted. (WTF::TimeWithDynamicClockType::clockType const): Deleted. (WTF::TimeWithDynamicClockType::withSameClockAndRawSeconds const): Deleted. (WTF::TimeWithDynamicClockType::operator bool const): Deleted. (WTF::TimeWithDynamicClockType::operator+ const): Deleted. (WTF::TimeWithDynamicClockType::operator- const): Deleted. (WTF::TimeWithDynamicClockType::operator+=): Deleted. (WTF::TimeWithDynamicClockType::operator-=): Deleted. (WTF::TimeWithDynamicClockType::operator== const): Deleted. (WTF::TimeWithDynamicClockType::operator!= const): Deleted. * wtf/TimingScope.h: * wtf/TinyLRUCache.h: * wtf/TinyPtrSet.h: * wtf/URLParser.cpp: * wtf/URLParser.h: * wtf/Unexpected.h: * wtf/Variant.h: * wtf/WTFSemaphore.h: (WTF::Semaphore::Semaphore): Deleted. (WTF::Semaphore::signal): Deleted. (WTF::Semaphore::waitUntil): Deleted. (WTF::Semaphore::waitFor): Deleted. (WTF::Semaphore::wait): Deleted. * wtf/WallTime.h: (WTF::WallTime::WallTime): Deleted. (WTF::WallTime::fromRawSeconds): Deleted. (WTF::WallTime::infinity): Deleted. (WTF::WallTime::nan): Deleted. (WTF::WallTime::secondsSinceEpoch const): Deleted. (WTF::WallTime::approximateWallTime const): Deleted. (WTF::WallTime::operator bool const): Deleted. (WTF::WallTime::operator+ const): Deleted. (WTF::WallTime::operator- const): Deleted. (WTF::WallTime::operator+=): Deleted. (WTF::WallTime::operator-=): Deleted. (WTF::WallTime::operator== const): Deleted. (WTF::WallTime::operator!= const): Deleted. (WTF::WallTime::operator< const): Deleted. (WTF::WallTime::operator> const): Deleted. (WTF::WallTime::operator<= const): Deleted. (WTF::WallTime::operator>= const): Deleted. (WTF::WallTime::isolatedCopy const): Deleted. * wtf/WeakHashSet.h: (WTF::WeakHashSet::WeakHashSetConstIterator::WeakHashSetConstIterator): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::get const): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::operator* const): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::operator-> const): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::operator++): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::skipEmptyBuckets): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::operator== const): Deleted. (WTF::WeakHashSet::WeakHashSetConstIterator::operator!= const): Deleted. (WTF::WeakHashSet::WeakHashSet): Deleted. (WTF::WeakHashSet::begin const): Deleted. (WTF::WeakHashSet::end const): Deleted. (WTF::WeakHashSet::add): Deleted. (WTF::WeakHashSet::remove): Deleted. (WTF::WeakHashSet::contains const): Deleted. (WTF::WeakHashSet::capacity const): Deleted. (WTF::WeakHashSet::computesEmpty const): Deleted. (WTF::WeakHashSet::hasNullReferences const): Deleted. (WTF::WeakHashSet::computeSize const): Deleted. (WTF::WeakHashSet::checkConsistency const): Deleted. * wtf/WeakRandom.h: (WTF::WeakRandom::WeakRandom): Deleted. (WTF::WeakRandom::setSeed): Deleted. (WTF::WeakRandom::seed const): Deleted. (WTF::WeakRandom::get): Deleted. (WTF::WeakRandom::getUint32): Deleted. (WTF::WeakRandom::lowOffset): Deleted. (WTF::WeakRandom::highOffset): Deleted. (WTF::WeakRandom::nextState): Deleted. (WTF::WeakRandom::generate): Deleted. (WTF::WeakRandom::advance): Deleted. * wtf/WordLock.h: (WTF::WordLock::lock): Deleted. (WTF::WordLock::unlock): Deleted. (WTF::WordLock::isHeld const): Deleted. (WTF::WordLock::isLocked const): Deleted. (WTF::WordLock::isFullyReset const): Deleted. * wtf/generic/MainThreadGeneric.cpp: * wtf/glib/GMutexLocker.h: * wtf/linux/CurrentProcessMemoryStatus.h: * wtf/posix/ThreadingPOSIX.cpp: (WTF::Semaphore::Semaphore): Deleted. (WTF::Semaphore::~Semaphore): Deleted. (WTF::Semaphore::wait): Deleted. (WTF::Semaphore::post): Deleted. * wtf/text/ASCIILiteral.h: (WTF::ASCIILiteral::operator const char* const): Deleted. (WTF::ASCIILiteral::fromLiteralUnsafe): Deleted. (WTF::ASCIILiteral::null): Deleted. (WTF::ASCIILiteral::characters const): Deleted. (WTF::ASCIILiteral::ASCIILiteral): Deleted. * wtf/text/AtomString.h: (WTF::AtomString::operator=): Deleted. (WTF::AtomString::isHashTableDeletedValue const): Deleted. (WTF::AtomString::existingHash const): Deleted. (WTF::AtomString::operator const String& const): Deleted. (WTF::AtomString::string const): Deleted. (WTF::AtomString::impl const): Deleted. (WTF::AtomString::is8Bit const): Deleted. (WTF::AtomString::characters8 const): Deleted. (WTF::AtomString::characters16 const): Deleted. (WTF::AtomString::length const): Deleted. (WTF::AtomString::operator[] const): Deleted. (WTF::AtomString::contains const): Deleted. (WTF::AtomString::containsIgnoringASCIICase const): Deleted. (WTF::AtomString::find const): Deleted. (WTF::AtomString::findIgnoringASCIICase const): Deleted. (WTF::AtomString::startsWith const): Deleted. (WTF::AtomString::startsWithIgnoringASCIICase const): Deleted. (WTF::AtomString::endsWith const): Deleted. (WTF::AtomString::endsWithIgnoringASCIICase const): Deleted. (WTF::AtomString::toInt const): Deleted. (WTF::AtomString::toDouble const): Deleted. (WTF::AtomString::toFloat const): Deleted. (WTF::AtomString::percentage const): Deleted. (WTF::AtomString::isNull const): Deleted. (WTF::AtomString::isEmpty const): Deleted. (WTF::AtomString::operator NSString * const): Deleted. * wtf/text/AtomStringImpl.h: (WTF::AtomStringImpl::lookUp): Deleted. (WTF::AtomStringImpl::add): Deleted. (WTF::AtomStringImpl::addWithStringTableProvider): Deleted. * wtf/text/CString.h: (WTF::CStringBuffer::data): Deleted. (WTF::CStringBuffer::length const): Deleted. (WTF::CStringBuffer::CStringBuffer): Deleted. (WTF::CStringBuffer::mutableData): Deleted. (WTF::CString::CString): Deleted. (WTF::CString::data const): Deleted. (WTF::CString::length const): Deleted. (WTF::CString::isNull const): Deleted. (WTF::CString::buffer const): Deleted. (WTF::CString::isHashTableDeletedValue const): Deleted. * wtf/text/ExternalStringImpl.h: (WTF::ExternalStringImpl::freeExternalBuffer): Deleted. * wtf/text/LineBreakIteratorPoolICU.h: * wtf/text/NullTextBreakIterator.h: * wtf/text/OrdinalNumber.h: * wtf/text/StringBuffer.h: * wtf/text/StringBuilder.h: * wtf/text/StringConcatenateNumbers.h: * wtf/text/StringHasher.h: * wtf/text/StringImpl.h: * wtf/text/StringView.cpp: * wtf/text/StringView.h: (WTF::StringView::left const): Deleted. (WTF::StringView::right const): Deleted. (WTF::StringView::underlyingStringIsValid const): Deleted. (WTF::StringView::setUnderlyingString): Deleted. * wtf/text/SymbolImpl.h: (WTF::SymbolImpl::StaticSymbolImpl::StaticSymbolImpl): Deleted. (WTF::SymbolImpl::StaticSymbolImpl::operator SymbolImpl&): Deleted. (WTF::PrivateSymbolImpl::PrivateSymbolImpl): Deleted. (WTF::RegisteredSymbolImpl::symbolRegistry const): Deleted. (WTF::RegisteredSymbolImpl::clearSymbolRegistry): Deleted. (WTF::RegisteredSymbolImpl::RegisteredSymbolImpl): Deleted. * wtf/text/SymbolRegistry.h: * wtf/text/TextBreakIterator.h: * wtf/text/TextPosition.h: * wtf/text/TextStream.h: * wtf/text/WTFString.h: (WTF::String::swap): Deleted. (WTF::String::adopt): Deleted. (WTF::String::isNull const): Deleted. (WTF::String::isEmpty const): Deleted. (WTF::String::impl const): Deleted. (WTF::String::releaseImpl): Deleted. (WTF::String::length const): Deleted. (WTF::String::characters8 const): Deleted. (WTF::String::characters16 const): Deleted. (WTF::String::is8Bit const): Deleted. (WTF::String::sizeInBytes const): Deleted. (WTF::String::operator[] const): Deleted. (WTF::String::find const): Deleted. (WTF::String::findIgnoringASCIICase const): Deleted. (WTF::String::reverseFind const): Deleted. (WTF::String::contains const): Deleted. (WTF::String::containsIgnoringASCIICase const): Deleted. (WTF::String::startsWith const): Deleted. (WTF::String::startsWithIgnoringASCIICase const): Deleted. (WTF::String::hasInfixStartingAt const): Deleted. (WTF::String::endsWith const): Deleted. (WTF::String::endsWithIgnoringASCIICase const): Deleted. (WTF::String::hasInfixEndingAt const): Deleted. (WTF::String::append): Deleted. (WTF::String::left const): Deleted. (WTF::String::right const): Deleted. (WTF::String::createUninitialized): Deleted. (WTF::String::fromUTF8WithLatin1Fallback): Deleted. (WTF::String::isAllASCII const): Deleted. (WTF::String::isAllLatin1 const): Deleted. (WTF::String::isSpecialCharacter const): Deleted. (WTF::String::isHashTableDeletedValue const): Deleted. (WTF::String::hash const): Deleted. (WTF::String::existingHash const): Deleted. * wtf/text/cf/TextBreakIteratorCF.h: * wtf/text/icu/TextBreakIteratorICU.h: * wtf/text/icu/UTextProviderLatin1.h: * wtf/threads/BinarySemaphore.h: (WTF::BinarySemaphore::waitFor): Deleted. (WTF::BinarySemaphore::wait): Deleted. * wtf/unicode/Collator.h: * wtf/win/GDIObject.h: * wtf/win/PathWalker.h: * wtf/win/Win32Handle.h: Canonical link: https://commits.webkit.org/214396@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@248546 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-08-12 20:57:15 +00:00
WTF_MAKE_FAST_ALLOCATED;
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
ParkingLot() = delete;
ParkingLot(const ParkingLot&) = delete;
public:
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows https://bugs.webkit.org/show_bug.cgi?id=152045 Reviewed by Andy Estes. Source/JavaScriptCore: Probably the nicest example of why this patch is a good idea is the change in AtomicsObject.cpp. * jit/ICStats.cpp: (JSC::ICStats::ICStats): * runtime/AtomicsObject.cpp: (JSC::atomicsFuncWait): Source/WebCore: No new layout tests because no new behavior. The new WTF time classes have some unit tests in TestWebKitAPI. * fileapi/ThreadableBlobRegistry.cpp: (WebCore::ThreadableBlobRegistry::blobSize): * platform/MainThreadSharedTimer.h: * platform/SharedTimer.h: * platform/ThreadTimers.cpp: (WebCore::ThreadTimers::updateSharedTimer): * platform/cf/MainThreadSharedTimerCF.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/efl/MainThreadSharedTimerEfl.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/glib/MainThreadSharedTimerGLib.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/win/MainThreadSharedTimerWin.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * workers/WorkerRunLoop.cpp: (WebCore::WorkerRunLoop::runInMode): Source/WebKit2: * Platform/IPC/Connection.cpp: (IPC::Connection::SyncMessageState::wait): (IPC::Connection::sendMessage): (IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting): (IPC::Connection::waitForMessage): (IPC::Connection::sendSyncMessage): (IPC::Connection::waitForSyncReply): * Platform/IPC/Connection.h: (IPC::Connection::sendSync): (IPC::Connection::waitForAndDispatchImmediately): * Platform/IPC/MessageSender.h: (IPC::MessageSender::sendSync): * UIProcess/ChildProcessProxy.h: (WebKit::ChildProcessProxy::sendSync): * UIProcess/Network/NetworkProcessProxy.cpp: (WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently): * UIProcess/Storage/StorageManager.cpp: (WebKit::StorageManager::applicationWillTerminate): * UIProcess/WebProcessProxy.cpp: (WebKit::WebProcessProxy::sendProcessWillSuspendImminently): * UIProcess/WebResourceLoadStatisticsStore.cpp: (WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate): * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h: * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm: (-[WKOneShotDisplayLinkHandler displayLinkFired:]): (WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree): (WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay): (WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm: (WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/WKImmediateActionController.mm: (-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]): * UIProcess/mac/WebPageProxyMac.mm: (WebKit::WebPageProxy::stringSelectionForPasteboard): (WebKit::WebPageProxy::dataSelectionForPasteboard): (WebKit::WebPageProxy::readSelectionFromPasteboard): (WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent): (WebKit::WebPageProxy::acceptsFirstMouse): * WebProcess/WebCoreSupport/WebChromeClient.cpp: (WebKit::WebChromeClient::runBeforeUnloadConfirmPanel): (WebKit::WebChromeClient::runJavaScriptAlert): (WebKit::WebChromeClient::runJavaScriptConfirm): (WebKit::WebChromeClient::runJavaScriptPrompt): (WebKit::WebChromeClient::print): (WebKit::WebChromeClient::exceededDatabaseQuota): (WebKit::WebChromeClient::reachedApplicationCacheOriginQuota): * WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp: (WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse): * WebProcess/WebPage/WebPage.cpp: (WebKit::WebPage::postSynchronousMessageForTesting): Source/WTF: We used to use 'double' for all time measurements. Sometimes it was milliseconds, sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time since some epoch. When we spoke of time since epoch, we either used a monotonic clock or a wall clock. The type - always 'double' - never told us what kind of time we had, even though there were roughly six of them (sec interval, ms interval, sec since epoch on wall, ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic). At some point, we thought that it would be a good idea to replace these doubles with std::chrono. But since replacing some things with std::chrono, we found it to be terribly inconvenient: - Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally duration_cast would not even be a thing. - No overflow protection. std::chrono uses integers by default and using anything else is clumsy. But the integer math is done without regard for the rough edges of integer math, so any cast between std::chrono types risks overflow. Any comparison risks overflow because it may do conversions silently. We have even found bugs where some C++ implementations had more overflows than others, which ends up being a special kind of hell. In many cases, the overflow also has nasal demons. It's an error to represent time using integers. It would have been excusable back when floating point math was not guaranteed to be supported on all platforms, but that would have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit for floating point: - Floating point preserves precision under multiplication in all but extreme cases, so using floating point for time means that unit conversions are almost completely lossless. This means that we don't have to think very hard about what units to use. In this patch, we use seconds almost everywhere. We only convert at boundaries, like an API boundary that wants something other than seconds. - Floating point makes it easy to reason about infinity, which is something that time code wants to do a lot. Example: when would you like to timeout? Infinity please! This is the most elegant way of having an API support both a timeout variant and a no-timeout variant. - Floating point does well-understood things when math goes wrong, and these things are pretty well optimized to match what a mathematician would do when computing with real numbers represented using scientific notation with a finite number of significant digits. This means that time math under floating point looks like normal math. On the other hand, std::chrono time math looks like garbage because you have to always check for multiple possible UB corners whenever you touch large integers. Integers that represent time are very likely to be large and you don't have to do much to overflow them. At this time, based on the number of bugs we have already seen due to chrono overflows, I am not certain that we even understand what are all of the corner cases that we should even check for. This patch introduces a new set of timekeeping classes that are all based on double, and all internally use seconds. These classes support algebraic typing. The classes are: - Seconds: this is for measuring a duration. - WallTime: time since epoch according to a wall clock (aka real time clock). - MonotonicTime: time since epoch according to a monotonic clock. - ClockType: enum that says either Wall or Monotonic. - TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a wall time or a monotonic time. All of these classes behave like C++ values and are cheap to copy around since they are very nearly POD. This supports comprehensive conversions between the various time types. Most of this is by way of algebra. Here are just some of the rules we recognize: WallTime = WallTime + Seconds Seconds = WallTime - WallTime MonotonicTime = MonotonicTime + Seconds etc... We support negative, infinite, and NaN times because math. We support conversions between MonotonicTime and WallTime, like: WallTime wt = mt.approximateWallTime() This is called this "approximate" because the only way to do it is to get the current time on both clocks and convert relative to that. Many of our APIs would be happy using whatever notion of time the user wanted to use. For those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This means that if you use a WallTime with Condition::waitUntil, then Condition's internal logic for when it should wake up makes its decision based on the current WallTime - but if you use MonotonicTime then waitUntil will make its decision based on current MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono did not have the concept of a dynamic clock type. This patch does not include conversions between std::chrono and these new time classes, because past experience shows that we're quite bad at getting conversions between std::chrono and anything else right. Also, I didn't need such conversion code because this patch only converts code that transitively touches ParkingLot and Condition. It was easy to get all of that code onto the new time classes. * WTF.xcodeproj/project.pbxproj: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/ClockType.cpp: Added. (WTF::printInternal): * wtf/ClockType.h: Added. * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::waitFor): (WTF::ConditionBase::wait): (WTF::ConditionBase::waitUntilWallClockSeconds): Deleted. (WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted. (WTF::ConditionBase::waitForSeconds): Deleted. (WTF::ConditionBase::waitForSecondsImpl): Deleted. (WTF::ConditionBase::waitForImpl): Deleted. (WTF::ConditionBase::absoluteFromRelative): Deleted. * wtf/CrossThreadQueue.h: (WTF::CrossThreadQueue<DataType>::waitForMessage): * wtf/CurrentTime.cpp: (WTF::sleep): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): Deleted. * wtf/MonotonicTime.cpp: Added. (WTF::MonotonicTime::now): (WTF::MonotonicTime::approximateWallTime): (WTF::MonotonicTime::dump): (WTF::MonotonicTime::sleep): * wtf/MonotonicTime.h: Added. (WTF::MonotonicTime::MonotonicTime): (WTF::MonotonicTime::fromRawDouble): (WTF::MonotonicTime::infinity): (WTF::MonotonicTime::secondsSinceEpoch): (WTF::MonotonicTime::approximateMonotonicTime): (WTF::MonotonicTime::operator bool): (WTF::MonotonicTime::operator+): (WTF::MonotonicTime::operator-): (WTF::MonotonicTime::operator+=): (WTF::MonotonicTime::operator-=): (WTF::MonotonicTime::operator==): (WTF::MonotonicTime::operator!=): (WTF::MonotonicTime::operator<): (WTF::MonotonicTime::operator>): (WTF::MonotonicTime::operator<=): (WTF::MonotonicTime::operator>=): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkCount): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): * wtf/Seconds.cpp: Added. (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::dump): (WTF::Seconds::sleep): * wtf/Seconds.h: Added. (WTF::Seconds::Seconds): (WTF::Seconds::value): (WTF::Seconds::seconds): (WTF::Seconds::milliseconds): (WTF::Seconds::microseconds): (WTF::Seconds::nanoseconds): (WTF::Seconds::fromMilliseconds): (WTF::Seconds::fromMicroseconds): (WTF::Seconds::fromNanoseconds): (WTF::Seconds::infinity): (WTF::Seconds::operator bool): (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::operator*): (WTF::Seconds::operator/): (WTF::Seconds::operator+=): (WTF::Seconds::operator-=): (WTF::Seconds::operator*=): (WTF::Seconds::operator/=): (WTF::Seconds::operator==): (WTF::Seconds::operator!=): (WTF::Seconds::operator<): (WTF::Seconds::operator>): (WTF::Seconds::operator<=): (WTF::Seconds::operator>=): * wtf/TimeWithDynamicClockType.cpp: Added. (WTF::TimeWithDynamicClockType::now): (WTF::TimeWithDynamicClockType::nowWithSameClock): (WTF::TimeWithDynamicClockType::wallTime): (WTF::TimeWithDynamicClockType::monotonicTime): (WTF::TimeWithDynamicClockType::approximateWallTime): (WTF::TimeWithDynamicClockType::approximateMonotonicTime): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator<): (WTF::TimeWithDynamicClockType::operator>): (WTF::TimeWithDynamicClockType::operator<=): (WTF::TimeWithDynamicClockType::operator>=): (WTF::TimeWithDynamicClockType::dump): (WTF::TimeWithDynamicClockType::sleep): * wtf/TimeWithDynamicClockType.h: Added. (WTF::TimeWithDynamicClockType::TimeWithDynamicClockType): (WTF::TimeWithDynamicClockType::fromRawDouble): (WTF::TimeWithDynamicClockType::secondsSinceEpoch): (WTF::TimeWithDynamicClockType::clockType): (WTF::TimeWithDynamicClockType::withSameClockAndRawDouble): (WTF::TimeWithDynamicClockType::operator bool): (WTF::TimeWithDynamicClockType::operator+): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator+=): (WTF::TimeWithDynamicClockType::operator-=): (WTF::TimeWithDynamicClockType::operator==): (WTF::TimeWithDynamicClockType::operator!=): * wtf/WallTime.cpp: Added. (WTF::WallTime::now): (WTF::WallTime::approximateMonotonicTime): (WTF::WallTime::dump): (WTF::WallTime::sleep): * wtf/WallTime.h: Added. (WTF::WallTime::WallTime): (WTF::WallTime::fromRawDouble): (WTF::WallTime::infinity): (WTF::WallTime::secondsSinceEpoch): (WTF::WallTime::approximateWallTime): (WTF::WallTime::operator bool): (WTF::WallTime::operator+): (WTF::WallTime::operator-): (WTF::WallTime::operator+=): (WTF::WallTime::operator-=): (WTF::WallTime::operator==): (WTF::WallTime::operator!=): (WTF::WallTime::operator<): (WTF::WallTime::operator>): (WTF::WallTime::operator<=): (WTF::WallTime::operator>=): * wtf/threads/BinarySemaphore.cpp: (WTF::BinarySemaphore::wait): * wtf/threads/BinarySemaphore.h: Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp: (TestWebKitAPI::ToUpperConverter::stopProducing): (TestWebKitAPI::ToUpperConverter::stopConsuming): * TestWebKitAPI/Tests/WTF/Time.cpp: Added. (WTF::operator<<): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/182152@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
// ParkingLot will accept any kind of time and convert it internally, but this typedef tells
// you what kind of time ParkingLot would be able to use without conversions. It's sad that
// this is WallTime not MonotonicTime, but that's just how OS wait functions work. However,
// because ParkingLot evaluates whether it should wait by checking if your time has passed
// using whatever clock you used, specifying timeouts in MonotonicTime is semantically better.
// For example, if the user sets his computer's clock back during the time that you wanted to
// wait for one second, and you specified the timeout using the MonotonicTime, then ParkingLot
// will be smart enough to know that your one second has elapsed.
typedef WallTime Time;
Replace all remaining uses of WTF::Mutex with WTF::Lock https://bugs.webkit.org/show_bug.cgi?id=148089 Reviewed by Geoffrey Garen. Source/WebCore: No new tests because no new behavior. * Modules/webaudio/AsyncAudioDecoder.cpp: (WebCore::AsyncAudioDecoder::AsyncAudioDecoder): (WebCore::AsyncAudioDecoder::runLoop): * Modules/webaudio/AsyncAudioDecoder.h: * Modules/webaudio/AudioContext.h: * Modules/webaudio/MediaStreamAudioSource.cpp: (WebCore::MediaStreamAudioSource::addAudioConsumer): (WebCore::MediaStreamAudioSource::removeAudioConsumer): (WebCore::MediaStreamAudioSource::setAudioFormat): (WebCore::MediaStreamAudioSource::consumeAudio): * Modules/webaudio/MediaStreamAudioSource.h: * Modules/webdatabase/Database.cpp: (WebCore::Database::close): (WebCore::Database::runTransaction): (WebCore::Database::inProgressTransactionCompleted): (WebCore::Database::hasPendingTransaction): * Modules/webdatabase/Database.h: * Modules/webdatabase/DatabaseTask.cpp: (WebCore::DatabaseTaskSynchronizer::taskCompleted): * Modules/webdatabase/DatabaseTask.h: * Modules/webdatabase/DatabaseThread.cpp: (WebCore::DatabaseThread::start): (WebCore::DatabaseThread::databaseThread): * Modules/webdatabase/DatabaseThread.h: * Modules/webdatabase/DatabaseTracker.cpp: (WebCore::DatabaseTracker::setDatabaseDirectoryPath): (WebCore::DatabaseTracker::canEstablishDatabase): (WebCore::DatabaseTracker::retryCanEstablishDatabase): (WebCore::DatabaseTracker::hasEntryForOrigin): (WebCore::DatabaseTracker::getMaxSizeForDatabase): (WebCore::DatabaseTracker::closeAllDatabases): (WebCore::DatabaseTracker::fullPathForDatabase): (WebCore::DatabaseTracker::origins): (WebCore::DatabaseTracker::databaseNamesForOrigin): (WebCore::DatabaseTracker::detailsForNameAndOrigin): (WebCore::DatabaseTracker::setDatabaseDetails): (WebCore::DatabaseTracker::doneCreatingDatabase): (WebCore::DatabaseTracker::addOpenDatabase): (WebCore::DatabaseTracker::removeOpenDatabase): (WebCore::DatabaseTracker::getOpenDatabases): (WebCore::DatabaseTracker::originLockFor): (WebCore::DatabaseTracker::quotaForOrigin): (WebCore::DatabaseTracker::setQuota): (WebCore::DatabaseTracker::deleteOrigin): (WebCore::DatabaseTracker::deleteDatabase): (WebCore::DatabaseTracker::deleteDatabaseFile): (WebCore::DatabaseTracker::removeDeletedOpenedDatabases): (WebCore::DatabaseTracker::deleteDatabaseFileIfEmpty): (WebCore::DatabaseTracker::openDatabaseMutex): (WebCore::DatabaseTracker::setClient): (WebCore::notificationMutex): (WebCore::DatabaseTracker::scheduleNotifyDatabaseChanged): (WebCore::DatabaseTracker::notifyDatabasesChanged): * Modules/webdatabase/DatabaseTracker.h: * Modules/webdatabase/OriginLock.h: * Modules/webdatabase/SQLCallbackWrapper.h: (WebCore::SQLCallbackWrapper::clear): (WebCore::SQLCallbackWrapper::unwrap): (WebCore::SQLCallbackWrapper::hasCallback): * Modules/webdatabase/SQLTransactionBackend.cpp: (WebCore::SQLTransactionBackend::doCleanup): (WebCore::SQLTransactionBackend::enqueueStatementBackend): (WebCore::SQLTransactionBackend::getNextStatement): * Modules/webdatabase/SQLTransactionBackend.h: * bindings/js/WorkerScriptController.cpp: (WebCore::WorkerScriptController::scheduleExecutionTermination): (WebCore::WorkerScriptController::isExecutionTerminating): * bindings/js/WorkerScriptController.h: * dom/default/PlatformMessagePortChannel.cpp: (WebCore::MessagePortChannel::postMessageToRemote): (WebCore::MessagePortChannel::tryGetMessageFromRemote): (WebCore::MessagePortChannel::isConnectedTo): (WebCore::MessagePortChannel::hasPendingActivity): (WebCore::MessagePortChannel::locallyEntangledPort): (WebCore::PlatformMessagePortChannel::setRemotePort): (WebCore::PlatformMessagePortChannel::entangledChannel): (WebCore::PlatformMessagePortChannel::closeInternal): * dom/default/PlatformMessagePortChannel.h: * loader/icon/IconDatabase.cpp: (WebCore::IconDatabase::removeAllIcons): (WebCore::IconDatabase::synchronousIconForPageURL): (WebCore::IconDatabase::synchronousNativeIconForPageURL): (WebCore::IconDatabase::synchronousIconURLForPageURL): (WebCore::IconDatabase::retainIconForPageURL): (WebCore::IconDatabase::performRetainIconForPageURL): (WebCore::IconDatabase::releaseIconForPageURL): (WebCore::IconDatabase::performReleaseIconForPageURL): (WebCore::IconDatabase::setIconDataForIconURL): (WebCore::IconDatabase::setIconURLForPageURL): (WebCore::IconDatabase::synchronousLoadDecisionForIconURL): (WebCore::IconDatabase::synchronousIconDataKnownForIconURL): (WebCore::IconDatabase::pageURLMappingCount): (WebCore::IconDatabase::retainedPageURLCount): (WebCore::IconDatabase::iconRecordCount): (WebCore::IconDatabase::iconRecordCountWithData): (WebCore::IconDatabase::wakeSyncThread): (WebCore::IconDatabase::scheduleOrDeferSyncTimer): (WebCore::IconDatabase::isOpenBesidesMainThreadCallbacks): (WebCore::IconDatabase::databasePath): (WebCore::IconDatabase::getOrCreatePageURLRecord): (WebCore::IconDatabase::iconDatabaseSyncThread): (WebCore::IconDatabase::performOpenInitialization): (WebCore::IconDatabase::performURLImport): (WebCore::IconDatabase::syncThreadMainLoop): (WebCore::IconDatabase::performPendingRetainAndReleaseOperations): (WebCore::IconDatabase::readFromDatabase): (WebCore::IconDatabase::writeToDatabase): (WebCore::IconDatabase::pruneUnretainedIcons): (WebCore::IconDatabase::cleanupSyncThread): * loader/icon/IconDatabase.h: * page/scrolling/ScrollingTree.cpp: (WebCore::ScrollingTree::shouldHandleWheelEventSynchronously): (WebCore::ScrollingTree::commitNewTreeState): (WebCore::ScrollingTree::setMainFramePinState): (WebCore::ScrollingTree::mainFrameScrollPosition): (WebCore::ScrollingTree::setMainFrameScrollPosition): (WebCore::ScrollingTree::isPointInNonFastScrollableRegion): (WebCore::ScrollingTree::isRubberBandInProgress): (WebCore::ScrollingTree::setMainFrameIsRubberBanding): (WebCore::ScrollingTree::isScrollSnapInProgress): (WebCore::ScrollingTree::setMainFrameIsScrollSnapping): (WebCore::ScrollingTree::setCanRubberBandState): (WebCore::ScrollingTree::rubberBandsAtLeft): (WebCore::ScrollingTree::rubberBandsAtRight): (WebCore::ScrollingTree::rubberBandsAtBottom): (WebCore::ScrollingTree::rubberBandsAtTop): (WebCore::ScrollingTree::setScrollPinningBehavior): (WebCore::ScrollingTree::scrollPinningBehavior): (WebCore::ScrollingTree::willWheelEventStartSwipeGesture): (WebCore::ScrollingTree::latchedNode): (WebCore::ScrollingTree::setLatchedNode): (WebCore::ScrollingTree::clearLatchedNode): * page/scrolling/ScrollingTree.h: * platform/MemoryPressureHandler.h: * platform/audio/HRTFDatabaseLoader.cpp: (WebCore::HRTFDatabaseLoader::loadAsynchronously): (WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion): * platform/audio/HRTFDatabaseLoader.h: * platform/cocoa/MemoryPressureHandlerCocoa.mm: (WebCore::MemoryPressureHandler::setReceivedMemoryPressure): (WebCore::MemoryPressureHandler::clearMemoryPressure): (WebCore::MemoryPressureHandler::shouldWaitForMemoryClearMessage): (WebCore::MemoryPressureHandler::respondToMemoryPressureIfNeeded): * platform/graphics/DisplayRefreshMonitor.cpp: (WebCore::DisplayRefreshMonitor::displayDidRefresh): * platform/graphics/DisplayRefreshMonitor.h: (WebCore::DisplayRefreshMonitor::setMonotonicAnimationStartTime): (WebCore::DisplayRefreshMonitor::mutex): * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp: (WebCore::MediaPlayerPrivateAVFoundation::setDelayCallbacks): (WebCore::MediaPlayerPrivateAVFoundation::clearMainThreadPendingFlag): (WebCore::MediaPlayerPrivateAVFoundation::dispatchNotification): * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h: * platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp: (WebCore::AVFWrapper::callbackContext): (WebCore::AVFWrapper::~AVFWrapper): (WebCore::AVFWrapper::mapLock): (WebCore::AVFWrapper::addToMap): (WebCore::AVFWrapper::removeFromMap): (WebCore::AVFWrapper::periodicTimeObserverCallback): (WebCore::AVFWrapper::processNotification): (WebCore::AVFWrapper::loadPlayableCompletionCallback): (WebCore::AVFWrapper::loadMetadataCompletionCallback): (WebCore::AVFWrapper::seekCompletedCallback): (WebCore::AVFWrapper::processCue): (WebCore::AVFWrapper::legibleOutputCallback): (WebCore::AVFWrapper::processShouldWaitForLoadingOfResource): (WebCore::AVFWrapper::resourceLoaderShouldWaitForLoadingOfRequestedResource): * platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp: (WebCore::InbandTextTrackPrivateGStreamer::handleSample): (WebCore::InbandTextTrackPrivateGStreamer::notifyTrackOfSample): * platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h: * platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp: (WebCore::TrackPrivateBaseGStreamer::tagsChanged): (WebCore::TrackPrivateBaseGStreamer::notifyTrackOfTagsChanged): * platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h: * platform/graphics/mac/DisplayRefreshMonitorMac.cpp: (WebCore::DisplayRefreshMonitorMac::requestRefreshCallback): (WebCore::DisplayRefreshMonitorMac::displayLinkFired): * platform/graphics/win/MediaPlayerPrivateMediaFoundation.cpp: (WebCore::MediaPlayerPrivateMediaFoundation::addListener): (WebCore::MediaPlayerPrivateMediaFoundation::removeListener): (WebCore::MediaPlayerPrivateMediaFoundation::notifyDeleted): (WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::Invoke): (WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::onMediaPlayerDeleted): * platform/graphics/win/MediaPlayerPrivateMediaFoundation.h: * platform/ios/LegacyTileCache.h: * platform/ios/LegacyTileCache.mm: (WebCore::LegacyTileCache::setTilesOpaque): (WebCore::LegacyTileCache::doLayoutTiles): (WebCore::LegacyTileCache::setCurrentScale): (WebCore::LegacyTileCache::commitScaleChange): (WebCore::LegacyTileCache::layoutTilesNow): (WebCore::LegacyTileCache::layoutTilesNowForRect): (WebCore::LegacyTileCache::removeAllNonVisibleTiles): (WebCore::LegacyTileCache::removeAllTiles): (WebCore::LegacyTileCache::removeForegroundTiles): (WebCore::LegacyTileCache::setContentReplacementImage): (WebCore::LegacyTileCache::contentReplacementImage): (WebCore::LegacyTileCache::tileCreationTimerFired): (WebCore::LegacyTileCache::setNeedsDisplayInRect): (WebCore::LegacyTileCache::updateTilingMode): (WebCore::LegacyTileCache::setTilingMode): (WebCore::LegacyTileCache::doPendingRepaints): (WebCore::LegacyTileCache::flushSavedDisplayRects): (WebCore::LegacyTileCache::prepareToDraw): * platform/ios/LegacyTileLayerPool.h: * platform/ios/LegacyTileLayerPool.mm: (WebCore::LegacyTileLayerPool::addLayer): (WebCore::LegacyTileLayerPool::takeLayerWithSize): (WebCore::LegacyTileLayerPool::setCapacity): (WebCore::LegacyTileLayerPool::prune): (WebCore::LegacyTileLayerPool::drain): * platform/network/curl/CurlDownload.cpp: (WebCore::CurlDownloadManager::add): (WebCore::CurlDownloadManager::remove): (WebCore::CurlDownloadManager::getActiveDownloadCount): (WebCore::CurlDownloadManager::getPendingDownloadCount): (WebCore::CurlDownloadManager::stopThreadIfIdle): (WebCore::CurlDownloadManager::updateHandleList): (WebCore::CurlDownload::~CurlDownload): (WebCore::CurlDownload::init): (WebCore::CurlDownload::getTempPath): (WebCore::CurlDownload::getUrl): (WebCore::CurlDownload::getResponse): (WebCore::CurlDownload::closeFile): (WebCore::CurlDownload::didReceiveHeader): (WebCore::CurlDownload::didReceiveData): (WebCore::CurlDownload::didFail): * platform/network/curl/CurlDownload.h: * platform/network/curl/ResourceHandleManager.cpp: (WebCore::cookieJarPath): (WebCore::sharedResourceMutex): (WebCore::curl_lock_callback): (WebCore::curl_unlock_callback): * platform/network/ios/QuickLook.mm: (WebCore::QLDirectoryAttributes): (qlPreviewConverterDictionaryMutex): (WebCore::addQLPreviewConverterWithFileForURL): (WebCore::qlPreviewConverterUTIForURL): (WebCore::removeQLPreviewConverterForURL): (WebCore::safeQLURLForDocumentURLAndResourceURL): * platform/sql/SQLiteDatabase.cpp: (WebCore::SQLiteDatabase::close): (WebCore::SQLiteDatabase::maximumSize): (WebCore::SQLiteDatabase::setMaximumSize): (WebCore::SQLiteDatabase::pageSize): (WebCore::SQLiteDatabase::freeSpaceSize): (WebCore::SQLiteDatabase::totalSize): (WebCore::SQLiteDatabase::runIncrementalVacuumCommand): (WebCore::SQLiteDatabase::setAuthorizer): * platform/sql/SQLiteDatabase.h: (WebCore::SQLiteDatabase::databaseMutex): * platform/sql/SQLiteStatement.cpp: (WebCore::SQLiteStatement::prepare): (WebCore::SQLiteStatement::step): * workers/WorkerThread.cpp: (WebCore::WorkerThread::start): (WebCore::WorkerThread::workerThread): (WebCore::WorkerThread::stop): * workers/WorkerThread.h: Source/WebKit: * Storage/StorageAreaSync.cpp: (WebCore::StorageAreaSync::syncTimerFired): (WebCore::StorageAreaSync::markImported): (WebCore::StorageAreaSync::blockUntilImportComplete): (WebCore::StorageAreaSync::performSync): * Storage/StorageAreaSync.h: * Storage/StorageTracker.cpp: (WebCore::StorageTracker::setDatabaseDirectoryPath): (WebCore::StorageTracker::finishedImportingOriginIdentifiers): (WebCore::StorageTracker::syncImportOriginIdentifiers): (WebCore::StorageTracker::syncFileSystemAndTrackerDatabase): (WebCore::StorageTracker::setOriginDetails): (WebCore::StorageTracker::syncSetOriginDetails): (WebCore::StorageTracker::origins): (WebCore::StorageTracker::deleteAllOrigins): (WebCore::StorageTracker::syncDeleteAllOrigins): (WebCore::StorageTracker::deleteOrigin): (WebCore::StorageTracker::syncDeleteOrigin): (WebCore::StorageTracker::canDeleteOrigin): (WebCore::StorageTracker::cancelDeletingOrigin): (WebCore::StorageTracker::diskUsageForOrigin): * Storage/StorageTracker.h: Source/WebKit/ios: * WebCoreSupport/WebFixedPositionContent.mm: (WebFixedPositionContentDataLock): (-[WebFixedPositionContent scrollOrZoomChanged:]): (-[WebFixedPositionContent overflowScrollPositionForLayer:changedTo:]): (-[WebFixedPositionContent setViewportConstrainedLayers:stickyContainerMap:]): (-[WebFixedPositionContent hasFixedOrStickyPositionLayers]): (-[WebFixedPositionContent minimumOffsetFromFixedPositionLayersToAnchorEdge:ofRect:inLayer:]): Source/WebKit/mac: * Storage/WebDatabaseManager.mm: (transactionBackgroundTaskIdentifierLock): (+[WebDatabaseManager startBackgroundTask]): (+[WebDatabaseManager endBackgroundTask]): * WebView/WebView.mm: (-[WebView _synchronizeCustomFixedPositionLayoutRect]): (-[WebView _setCustomFixedPositionLayoutRectInWebThread:synchronize:]): (-[WebView _setCustomFixedPositionLayoutRect:]): (-[WebView _fetchCustomFixedPositionLayoutRect:]): * WebView/WebViewData.h: Source/WebKit/win: * Plugins/PluginMainThreadScheduler.cpp: (WebCore::PluginMainThreadScheduler::scheduleCall): (WebCore::PluginMainThreadScheduler::registerPlugin): (WebCore::PluginMainThreadScheduler::unregisterPlugin): (WebCore::PluginMainThreadScheduler::dispatchCallsForPlugin): * Plugins/PluginMainThreadScheduler.h: * WebIconDatabase.cpp: (WebIconDatabase::didRemoveAllIcons): (WebIconDatabase::didImportIconURLForPageURL): (WebIconDatabase::deliverNotifications): * WebIconDatabase.h: * WebLocalizableStrings.cpp: (mainBundleLocStrings): (frameworkLocStringsMutex): (findCachedString): (cacheString): Source/WebKit2: * DatabaseProcess/DatabaseProcess.cpp: (WebKit::DatabaseProcess::postDatabaseTask): (WebKit::DatabaseProcess::performNextDatabaseTask): * DatabaseProcess/DatabaseProcess.h: * DatabaseProcess/IndexedDB/UniqueIDBDatabase.cpp: (WebKit::UniqueIDBDatabase::shutdown): (WebKit::UniqueIDBDatabase::postMainThreadTask): (WebKit::UniqueIDBDatabase::performNextMainThreadTask): (WebKit::UniqueIDBDatabase::postDatabaseTask): (WebKit::UniqueIDBDatabase::performNextDatabaseTask): * DatabaseProcess/IndexedDB/UniqueIDBDatabase.h: * Platform/IPC/Connection.cpp: (IPC::Connection::sendSyncMessage): (IPC::Connection::sendSyncMessageFromSecondaryThread): (IPC::Connection::waitForSyncReply): (IPC::Connection::processIncomingSyncReply): (IPC::Connection::connectionDidClose): * Platform/IPC/Connection.h: * Shared/CoordinatedGraphics/CoordinatedGraphicsScene.cpp: (WebKit::CoordinatedGraphicsScene::appendUpdate): * Shared/CoordinatedGraphics/CoordinatedGraphicsScene.h: * Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.cpp: (WebKit::ThreadedCompositor::createCompositingThread): (WebKit::ThreadedCompositor::runCompositingThread): (WebKit::ThreadedCompositor::terminateCompositingThread): * Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.h: * Shared/Network/CustomProtocols/Cocoa/CustomProtocolManagerCocoa.mm: (WebKit::CustomProtocolManager::addCustomProtocol): (WebKit::CustomProtocolManager::removeCustomProtocol): (WebKit::CustomProtocolManager::registerScheme): (WebKit::CustomProtocolManager::unregisterScheme): (WebKit::CustomProtocolManager::supportsScheme): (WebKit::CustomProtocolManager::protocolForID): * Shared/Network/CustomProtocols/CustomProtocolManager.h: * Shared/linux/SeccompFilters/SeccompBroker.cpp: * WebProcess/Plugins/PluginProcessConnectionManager.cpp: (WebKit::PluginProcessConnectionManager::getPluginProcessConnection): (WebKit::PluginProcessConnectionManager::removePluginProcessConnection): (WebKit::PluginProcessConnectionManager::pluginProcessCrashed): * WebProcess/Plugins/PluginProcessConnectionManager.h: * WebProcess/WebPage/EventDispatcher.cpp: (WebKit::EventDispatcher::addScrollingTreeForPage): (WebKit::EventDispatcher::removeScrollingTreeForPage): (WebKit::EventDispatcher::wheelEvent): * WebProcess/WebPage/EventDispatcher.h: * WebProcess/soup/WebKitSoupRequestInputStream.cpp: (webkitSoupRequestInputStreamReadAsync): (webkitSoupRequestInputStreamAddData): Source/WTF: This also beefs up and rationalizes the Condition API, so that it can deal with units of time other than just steady_clock. This makes it easier to port ThreadCondition and std::condition_variable code over to Condition. This patch does not take a position on what kind of time is best; from reading a lot of the uses in WebCore, it seems like our use of double to measure seconds is often nicer than the many different classes in std::chrono. Also added a Condition speed test, to make sure that all of this is a good idea. And indeed it is. The 1-producer/1-consumer scenario with a 100-element queue runs 36x faster using Lock/Condition than Mutex/ThreadCondition when you use the notifyOne()-per-enqueue style. It runs 58x faster with Lock/Condition when you use the notifyAll()-at-boundary style. Note that I have a bug open for making the notifyOne()-per-enqueue style even faster: https://bugs.webkit.org/show_bug.cgi?id=148090. Also, the 10-consumer/10-producer scenario with a 100-element queue runs 20x faster with Lock/Condition for notifyOne()-per-enqueue and 30x faster with notifyAll()-at-boundary. The only way to tweak the test to get Mutex/ThreadCondition to win is to have one producer, one consumer, a 1-element queue, and use the notifyOne()-per-enqueue style. In that case, one of the two threads is going to be waiting most of the time and the test basically measures wake-up latency and nothing else. Because Condition::wait() does a little bit more work than ThreadCondition::wait(), Mutex/ThreadCondition end up running 3% faster in this test case. But if you vary any of the parameters of the test, Mutex/ThreadCondition ends up losing - all it takes is more threads or a queue size of 5 or more. To my knowledge, we never do producer/consumer with a queue bounded to one element precisely because that approach is the least efficient regardless of locking algorithm. For example, neither WTF::MessageQueue nor DFG::Worklist have any bounds on their queue size. So, it seems that replacing all uses of system mutexes and condition variables with our own thing is a great idea. * benchmarks/LockSpeedTest.cpp: * benchmarks/ConditionSpeedTest.cpp: Added. * wtf/Condition.h: (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::waitUntilWallClockSeconds): (WTF::Condition::waitUntilMonotonicClockSeconds): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): (WTF::Condition::waitForSecondsImpl): (WTF::Condition::waitForImpl): (WTF::Condition::absoluteFromRelative): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): (WTF::MessageQueue<DataType>::append): (WTF::MessageQueue<DataType>::appendAndKill): (WTF::MessageQueue<DataType>::appendAndCheckEmpty): (WTF::MessageQueue<DataType>::prepend): (WTF::MessageQueue<DataType>::removeIf): (WTF::MessageQueue<DataType>::isEmpty): (WTF::MessageQueue<DataType>::kill): (WTF::MessageQueue<DataType>::killed): * wtf/ParallelJobsGeneric.cpp: (WTF::ParallelEnvironment::ThreadPrivate::execute): (WTF::ParallelEnvironment::ThreadPrivate::waitForFinish): (WTF::ParallelEnvironment::ThreadPrivate::workerThread): * wtf/ParallelJobsGeneric.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): * wtf/ThreadingPthreads.cpp: (WTF::initializeThreading): * wtf/ThreadingWin.cpp: (WTF::initializeThreading): * wtf/dtoa.cpp: (WTF::pow5mult): * wtf/dtoa.h: Tools: * DumpRenderTree/JavaScriptThreading.cpp: (javaScriptThreadsMutex): (runJavaScriptThread): (startJavaScriptThreads): (stopJavaScriptThreads): * TestWebKitAPI/Tests/WTF/Condition.cpp: Fixed a bug in the test that I found from turning the test into a benchmark. * TestWebKitAPI/Tests/WTF/WorkQueue.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/glib/WorkQueueGLib.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/166252@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188594 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-18 19:31:28 +00:00
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
// Parks the thread in a queue associated with the given address, which cannot be null. The
// parking only succeeds if the validation function returns true while the queue lock is held.
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
//
WTF should have a compact Condition object to use with Lock https://bugs.webkit.org/show_bug.cgi?id=147986 Reviewed by Geoffrey Garen. Source/WTF: Adds a condition variable implementation based on ParkingLot, called simply WTF::Condition. It can be used with WTF::Lock or actually any lock implementation. It should even work with WTF::SpinLock, WTF::Mutex, or std::mutex. Best of all, Condition only requires one byte. ParkingLot almost contained all of the functionality needed to implemenet wait/notify. We could have implemented Condition using a 32-bit (or even 64-bit) version that protects against a notify that happens just before we park. But, this changes the ParkingLot API to give us the ability to run some code between when ParkingLot enqueues the current thread and when it actually sleeps. This callback is called with no locks held, so it can call unlock() on any kind of lock, so long as that lock's unlock() method doesn't recurse into ParkingLot::parkConditionally(). That seems unlikely; unlock() is more likely to call ParkingLot::unparkOne() or unparkAll(). WTF::Lock will never call parkConditionally() inside unlock(), so WTF::Lock is definitely appropriate for use with Condition. Condition supports most of the API that std::condition_variable supports. It does some things to try to reduce footgun potential. The preferred timeout form is waitUntil() which takes an absolute time from the steady_clock. The only relative timeout form also takes a predicate callback, so it's impossible to write the subtly incorrect "while (...) wait_for(...)" idiom. This patch doesn't actually introduce any uses of WTF::Condition other than the unit tests. I'll start switching code over to using WTF::Condition in another patch. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Condition.h: Added. (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): Make this useful assertion be a release assertion. It catches cases where you unlock the lock even though you don't hold it. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): Add the beforeSleep() callback. (WTF::ParkingLot::unparkOne): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): Tools: Add a test for WTF::Condition. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: Added. (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): Change the name of the thread. Canonical link: https://commits.webkit.org/166094@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188400 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 20:42:11 +00:00
// If validation returns false, it will unlock the internal parking queue and then it will
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// return a null ParkResult (wasUnparked = false, token = 0) without doing anything else.
//
// If validation returns true, it will enqueue the thread, unlock the parking queue lock, call
// the beforeSleep function, and then it will sleep so long as the thread continues to be on the
// queue and the timeout hasn't fired. Finally, this returns wasUnparked = true if we actually
// got unparked or wasUnparked = false if the timeout was hit. When wasUnparked = true, the
// token will contain whatever token was returned from the callback to unparkOne(), or 0 if the
// thread was unparked using unparkAll() or the form of unparkOne() that doesn't take a
// callback.
//
// Note that beforeSleep is called with no locks held, so it's OK to do pretty much anything so
// long as you don't recursively call parkConditionally(). You can call unparkOne()/unparkAll()
// though. It's useful to use beforeSleep() to unlock some mutex in the implementation of
WTF should have a compact Condition object to use with Lock https://bugs.webkit.org/show_bug.cgi?id=147986 Reviewed by Geoffrey Garen. Source/WTF: Adds a condition variable implementation based on ParkingLot, called simply WTF::Condition. It can be used with WTF::Lock or actually any lock implementation. It should even work with WTF::SpinLock, WTF::Mutex, or std::mutex. Best of all, Condition only requires one byte. ParkingLot almost contained all of the functionality needed to implemenet wait/notify. We could have implemented Condition using a 32-bit (or even 64-bit) version that protects against a notify that happens just before we park. But, this changes the ParkingLot API to give us the ability to run some code between when ParkingLot enqueues the current thread and when it actually sleeps. This callback is called with no locks held, so it can call unlock() on any kind of lock, so long as that lock's unlock() method doesn't recurse into ParkingLot::parkConditionally(). That seems unlikely; unlock() is more likely to call ParkingLot::unparkOne() or unparkAll(). WTF::Lock will never call parkConditionally() inside unlock(), so WTF::Lock is definitely appropriate for use with Condition. Condition supports most of the API that std::condition_variable supports. It does some things to try to reduce footgun potential. The preferred timeout form is waitUntil() which takes an absolute time from the steady_clock. The only relative timeout form also takes a predicate callback, so it's impossible to write the subtly incorrect "while (...) wait_for(...)" idiom. This patch doesn't actually introduce any uses of WTF::Condition other than the unit tests. I'll start switching code over to using WTF::Condition in another patch. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Condition.h: Added. (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): Make this useful assertion be a release assertion. It catches cases where you unlock the lock even though you don't hold it. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): Add the beforeSleep() callback. (WTF::ParkingLot::unparkOne): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): Tools: Add a test for WTF::Condition. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: Added. (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): Change the name of the thread. Canonical link: https://commits.webkit.org/166094@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188400 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 20:42:11 +00:00
// Condition::wait().
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
struct ParkResult {
bool wasUnparked { false };
intptr_t token { 0 };
};
template<typename ValidationFunctor, typename BeforeSleepFunctor>
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
static ParkResult parkConditionally(
WTF should have a compact Condition object to use with Lock https://bugs.webkit.org/show_bug.cgi?id=147986 Reviewed by Geoffrey Garen. Source/WTF: Adds a condition variable implementation based on ParkingLot, called simply WTF::Condition. It can be used with WTF::Lock or actually any lock implementation. It should even work with WTF::SpinLock, WTF::Mutex, or std::mutex. Best of all, Condition only requires one byte. ParkingLot almost contained all of the functionality needed to implemenet wait/notify. We could have implemented Condition using a 32-bit (or even 64-bit) version that protects against a notify that happens just before we park. But, this changes the ParkingLot API to give us the ability to run some code between when ParkingLot enqueues the current thread and when it actually sleeps. This callback is called with no locks held, so it can call unlock() on any kind of lock, so long as that lock's unlock() method doesn't recurse into ParkingLot::parkConditionally(). That seems unlikely; unlock() is more likely to call ParkingLot::unparkOne() or unparkAll(). WTF::Lock will never call parkConditionally() inside unlock(), so WTF::Lock is definitely appropriate for use with Condition. Condition supports most of the API that std::condition_variable supports. It does some things to try to reduce footgun potential. The preferred timeout form is waitUntil() which takes an absolute time from the steady_clock. The only relative timeout form also takes a predicate callback, so it's impossible to write the subtly incorrect "while (...) wait_for(...)" idiom. This patch doesn't actually introduce any uses of WTF::Condition other than the unit tests. I'll start switching code over to using WTF::Condition in another patch. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Condition.h: Added. (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): Make this useful assertion be a release assertion. It catches cases where you unlock the lock even though you don't hold it. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): Add the beforeSleep() callback. (WTF::ParkingLot::unparkOne): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): Tools: Add a test for WTF::Condition. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: Added. (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): Change the name of the thread. Canonical link: https://commits.webkit.org/166094@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188400 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 20:42:11 +00:00
const void* address,
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression https://bugs.webkit.org/show_bug.cgi?id=160125 Reviewed by Geoffrey Garen and Keith Miller. JSTests: Most of the things I did properly covered by existing tests, but I found some simple cases of unshifting that had sketchy coverage. * stress/array-storage-array-unshift.js: Added. * stress/contiguous-array-unshift.js: Added. * stress/double-array-unshift.js: Added. * stress/int32-array-unshift.js: Added. Source/bmalloc: I needed to tryMemalign, so I added such a thing. * bmalloc/Allocator.cpp: (bmalloc::Allocator::allocate): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): * bmalloc/Allocator.h: * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): * bmalloc/bmalloc.h: (bmalloc::api::tryMemalign): Source/JavaScriptCore: In order to make the GC concurrent (bug 149432), we would either need to enable concurrent copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead from the barriers alone. Considering that MarkedSpace does a decent job of avoiding fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want to get rid of copied space. This change moves copied space's biggest client over to marked space. Moving butterflies to marked space means having them use the new Auxiliary HeapCell allocation path. This is a fairly mechanical change, but it caused performance regressions everywhere, so this change also fixes MarkedSpace's performance issues. At a high level the mechanical changes are: - We use AuxiliaryBarrier instead of CopyBarrier. - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth. - The JITs have to emit inlined marked space allocations instead of inline copy space allocations. - Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't. That's about 1/3 of this change. But this led to performance problems, which I fixed with optimizations that amounted to a major MarkedSpace rewrite: - MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely. - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well. - Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being: 16, 32, 48, 64, 80, 112, 160, 224, 320 The next size class would have been 432, but that's above the 400 byte cutoff. All of this is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what size classes you end up with by doing --dumpSizeClasses=true. - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: https://bugs.webkit.org/show_bug.cgi?id=161581. - Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree. - FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky. - It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch! - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes. All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1% regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a best-case scenario for bump allocation. I think that we should fix bmalloc to make up the difference, but take the hit for now because it's a crazy corner case. By comparison, the alternative approach of using a copy barrier would have cost us 1-2%. That's the real apples-to-apples comparison if your premise is that we should have a concurrent GC. After we finish removing copied space, we will be barrier-ready for concurrent GC: we already have a marking barrier and we simply won't need a copying barrier. This change gets us there for the purposes of our benchmarks, since the remaining clients of copied space are not very important. On the other hand, if we keep copying, then getting barrier-ready would mean adding back the copy barrier, which costs more perf. We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving typed arrays and a few other weird things over to Aux MarkedSpace. This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell, and CellContainer meant that I had to include those files from everywhere. Fortunately, just including JSCInlines.h (instead of manually including the files that includes) is usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something that we were already basically doing. In places where JSCInlines.h would be too much, I just included HeapInlines.h. This got weird, because we previously included HeapInlines.h from JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that meant having to manually include HeapInlines.h from the places that previously got it implicitly via JSObject.h. But that led to more problems for some reason: I started getting build errors because non-JSC files were having trouble including Opcode.h. That's just silly, since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and made it impossible to include it from outside JSC. This was a lot of work, but it was necessary to get the patch to build on all ports. It's also a net win. There were many places in WebCore that were transitively including a *ton* of JSC headers just because of the JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons. * API/JSManagedValue.mm: (-[JSManagedValue initWithValue:]): * API/JSTypedArray.cpp: * API/ObjCCallbackFunction.mm: * API/tests/testapi.mm: (testObjectiveCAPI): (testWeakValue): Deleted. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * Scripts/builtins/builtins_generate_combined_implementation.py: (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py: (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_separate_implementation.py: (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes): * assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::JumpList::link): (JSC::AbstractMacroAssembler::JumpList::linkTo): * assembler/MacroAssembler.h: * assembler/MacroAssemblerARM64.h: (JSC::MacroAssemblerARM64::add32): * assembler/MacroAssemblerCodeRef.cpp: Added. (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): (JSC::MacroAssemblerCodePtr::dumpWithName): (JSC::MacroAssemblerCodePtr::dump): (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): (JSC::MacroAssemblerCodeRef::dump): * assembler/MacroAssemblerCodeRef.h: (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted. (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted. (JSC::MacroAssemblerCodePtr::dump): Deleted. (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted. (JSC::MacroAssemblerCodeRef::dump): Deleted. * b3/B3BasicBlock.cpp: (JSC::B3::BasicBlock::appendBoolConstant): * b3/B3BasicBlock.h: * b3/B3DuplicateTails.cpp: * b3/B3StackmapGenerationParams.h: * b3/testb3.cpp: (JSC::B3::testPatchpointTerminalReturnValue): (JSC::B3::run): * bindings/ScriptValue.cpp: * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeLivenessAnalysis.cpp: * bytecode/BytecodeUseDef.h: * bytecode/CallLinkInfo.cpp: (JSC::CallLinkInfo::callTypeFor): * bytecode/CallLinkInfo.h: (JSC::CallLinkInfo::callTypeFor): Deleted. * bytecode/CallLinkStatus.cpp: * bytecode/CodeBlock.cpp: (JSC::CodeBlock::finishCreation): (JSC::CodeBlock::clearLLIntGetByIdCache): (JSC::CodeBlock::predictedMachineCodeSize): * bytecode/CodeBlock.h: (JSC::CodeBlock::jitCodeMap): Deleted. (JSC::clearLLIntGetByIdCache): Deleted. * bytecode/ExecutionCounter.h: * bytecode/Instruction.h: * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp: (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal): * bytecode/ObjectAllocationProfile.h: (JSC::ObjectAllocationProfile::isNull): (JSC::ObjectAllocationProfile::initialize): * bytecode/Opcode.h: (JSC::padOpcodeName): * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): (JSC::PolymorphicAccess::regenerate): * bytecode/PolymorphicAccess.h: * bytecode/PreciseJumpTargets.cpp: * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecode/UnlinkedCodeBlock.cpp: (JSC::UnlinkedCodeBlock::vm): Deleted. * bytecode/UnlinkedCodeBlock.h: * bytecode/UnlinkedInstructionStream.cpp: * bytecode/UnlinkedInstructionStream.h: * dfg/DFGOperations.cpp: * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT.h: (JSC::DFG::SpeculativeJIT::emitAllocateJSCell): (JSC::DFG::SpeculativeJIT::emitAllocateJSObject): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGStrengthReductionPhase.cpp: (JSC::DFG::StrengthReductionPhase::handleNode): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCompile.cpp: * ftl/FTLJITFinalizer.cpp: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments): (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest): (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell): (JSC::FTL::DFG::LowerDFGToB3::allocateCell): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted. * ftl/FTLOutput.cpp: (JSC::FTL::Output::constBool): (JSC::FTL::Output::add): (JSC::FTL::Output::shl): (JSC::FTL::Output::aShr): (JSC::FTL::Output::lShr): (JSC::FTL::Output::zeroExt): (JSC::FTL::Output::equal): (JSC::FTL::Output::notEqual): (JSC::FTL::Output::above): (JSC::FTL::Output::aboveOrEqual): (JSC::FTL::Output::below): (JSC::FTL::Output::belowOrEqual): (JSC::FTL::Output::greaterThan): (JSC::FTL::Output::greaterThanOrEqual): (JSC::FTL::Output::lessThan): (JSC::FTL::Output::lessThanOrEqual): (JSC::FTL::Output::select): (JSC::FTL::Output::appendSuccessor): (JSC::FTL::Output::addIncomingToPhi): * ftl/FTLOutput.h: * ftl/FTLValueFromBlock.h: (JSC::FTL::ValueFromBlock::operator bool): (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted. * ftl/FTLWeightedTarget.h: (JSC::FTL::WeightedTarget::frequentedBlock): * heap/CellContainer.h: Added. (JSC::CellContainer::CellContainer): (JSC::CellContainer::operator bool): (JSC::CellContainer::isMarkedBlock): (JSC::CellContainer::isLargeAllocation): (JSC::CellContainer::markedBlock): (JSC::CellContainer::largeAllocation): * heap/CellContainerInlines.h: Added. (JSC::CellContainer::isMarked): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::noteMarked): (JSC::CellContainer::cellSize): (JSC::CellContainer::weakSet): (JSC::CellContainer::flipIfNecessary): * heap/ConservativeRoots.cpp: (JSC::ConservativeRoots::ConservativeRoots): (JSC::ConservativeRoots::~ConservativeRoots): (JSC::ConservativeRoots::grow): (JSC::ConservativeRoots::genericAddPointer): (JSC::ConservativeRoots::genericAddSpan): * heap/ConservativeRoots.h: (JSC::ConservativeRoots::roots): * heap/CopyToken.h: * heap/FreeList.cpp: Added. (JSC::FreeList::dump): * heap/FreeList.h: Added. (JSC::FreeList::FreeList): (JSC::FreeList::list): (JSC::FreeList::bump): (JSC::FreeList::operator==): (JSC::FreeList::operator!=): (JSC::FreeList::operator bool): (JSC::FreeList::allocationWillFail): (JSC::FreeList::allocationWillSucceed): * heap/GCTypeMap.h: Added. (JSC::GCTypeMap::operator[]): * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalizeUnconditionalFinalizers): (JSC::Heap::markRoots): (JSC::Heap::copyBackingStores): (JSC::Heap::gatherStackRoots): (JSC::Heap::gatherJSStackRoots): (JSC::Heap::gatherScratchBufferRoots): (JSC::Heap::clearLivenessData): (JSC::Heap::visitSmallStrings): (JSC::Heap::visitConservativeRoots): (JSC::Heap::removeDeadCompilerWorklistEntries): (JSC::Heap::gatherExtraHeapSnapshotData): (JSC::Heap::removeDeadHeapSnapshotNodes): (JSC::Heap::visitProtectedObjects): (JSC::Heap::visitArgumentBuffers): (JSC::Heap::visitException): (JSC::Heap::visitStrongHandles): (JSC::Heap::visitHandleStack): (JSC::Heap::visitSamplingProfiler): (JSC::Heap::traceCodeBlocksAndJITStubRoutines): (JSC::Heap::converge): (JSC::Heap::visitWeakHandles): (JSC::Heap::updateObjectCounts): (JSC::Heap::clearUnmarkedExecutables): (JSC::Heap::deleteUnmarkedCompiledCode): (JSC::Heap::collectAllGarbage): (JSC::Heap::collect): (JSC::Heap::collectWithoutAnySweep): (JSC::Heap::collectImpl): (JSC::Heap::suspendCompilerThreads): (JSC::Heap::willStartCollection): (JSC::Heap::flushOldStructureIDTables): (JSC::Heap::flushWriteBarrierBuffer): (JSC::Heap::stopAllocation): (JSC::Heap::prepareForMarking): (JSC::Heap::reapWeakHandles): (JSC::Heap::pruneStaleEntriesFromWeakGCMaps): (JSC::Heap::sweepArrayBuffers): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): (JSC::MarkedBlockSnapshotFunctor::operator()): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::deleteSourceProviderCaches): (JSC::Heap::notifyIncrementalSweeper): (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks): (JSC::Heap::resetAllocators): (JSC::Heap::updateAllocationLimits): (JSC::Heap::didFinishCollection): (JSC::Heap::resumeCompilerThreads): (JSC::Zombify::visit): (JSC::Heap::forEachCodeBlockImpl): * heap/Heap.h: (JSC::Heap::allocatorForObjectWithoutDestructor): (JSC::Heap::allocatorForObjectWithDestructor): (JSC::Heap::allocatorForAuxiliaryData): (JSC::Heap::jitStubRoutines): (JSC::Heap::codeBlockSet): (JSC::Heap::storageAllocator): Deleted. * heap/HeapCell.h: (JSC::HeapCell::isZapped): Deleted. * heap/HeapCellInlines.h: Added. (JSC::HeapCell::isLargeAllocation): (JSC::HeapCell::cellContainer): (JSC::HeapCell::markedBlock): (JSC::HeapCell::largeAllocation): (JSC::HeapCell::heap): (JSC::HeapCell::vm): (JSC::HeapCell::cellSize): (JSC::HeapCell::allocatorAttributes): (JSC::HeapCell::destructionMode): (JSC::HeapCell::cellKind): * heap/HeapInlines.h: (JSC::Heap::heap): (JSC::Heap::isLive): (JSC::Heap::isMarked): (JSC::Heap::testAndSetMarked): (JSC::Heap::setMarked): (JSC::Heap::cellSize): (JSC::Heap::forEachCodeBlock): (JSC::Heap::allocateObjectOfType): (JSC::Heap::subspaceForObjectOfType): (JSC::Heap::allocatorForObjectOfType): (JSC::Heap::allocateAuxiliary): (JSC::Heap::tryAllocateAuxiliary): (JSC::Heap::tryReallocateAuxiliary): (JSC::Heap::isPointerGCObject): Deleted. (JSC::Heap::isValueGCObject): Deleted. * heap/HeapOperation.cpp: Added. (WTF::printInternal): * heap/HeapOperation.h: * heap/HeapUtil.h: Added. (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): (JSC::HeapUtil::isValueGCObject): * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::sweepNextBlock): * heap/IncrementalSweeper.h: * heap/LargeAllocation.cpp: Added. (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::lastChanceToFinalize): (JSC::LargeAllocation::shrink): (JSC::LargeAllocation::visitWeakSet): (JSC::LargeAllocation::reapWeakSet): (JSC::LargeAllocation::flip): (JSC::LargeAllocation::isEmpty): (JSC::LargeAllocation::sweep): (JSC::LargeAllocation::destroy): (JSC::LargeAllocation::dump): * heap/LargeAllocation.h: Added. (JSC::LargeAllocation::fromCell): (JSC::LargeAllocation::cell): (JSC::LargeAllocation::isLargeAllocation): (JSC::LargeAllocation::heap): (JSC::LargeAllocation::vm): (JSC::LargeAllocation::weakSet): (JSC::LargeAllocation::clearNewlyAllocated): (JSC::LargeAllocation::isNewlyAllocated): (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::isLive): (JSC::LargeAllocation::hasValidCell): (JSC::LargeAllocation::cellSize): (JSC::LargeAllocation::aboveLowerBound): (JSC::LargeAllocation::belowUpperBound): (JSC::LargeAllocation::contains): (JSC::LargeAllocation::attributes): (JSC::LargeAllocation::flipIfNecessary): (JSC::LargeAllocation::flipIfNecessaryConcurrently): (JSC::LargeAllocation::testAndSetMarked): (JSC::LargeAllocation::setMarked): (JSC::LargeAllocation::clearMarked): (JSC::LargeAllocation::noteMarked): (JSC::LargeAllocation::headerSize): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::retire): (JSC::MarkedAllocator::filterNextBlock): (JSC::MarkedAllocator::setNextBlockToSweep): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::blockHeaderSize): (JSC::MarkedAllocator::blockSizeForBytes): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::reset): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::setFreeList): (JSC::isListPagedOut): Deleted. (JSC::MarkedAllocator::tryAllocateHelper): Deleted. (JSC::MarkedAllocator::tryPopFreeList): Deleted. (JSC::MarkedAllocator::tryAllocate): Deleted. (JSC::MarkedAllocator::allocateBlock): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::takeLastActiveBlock): (JSC::MarkedAllocator::offsetOfFreeList): (JSC::MarkedAllocator::offsetOfCellSize): (JSC::MarkedAllocator::tryAllocate): (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted. (JSC::MarkedAllocator::MarkedAllocator): Deleted. (JSC::MarkedAllocator::init): Deleted. (JSC::MarkedAllocator::stopAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): (JSC::SetNewlyAllocatedFunctor::operator()): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::Handle::zap): (JSC::MarkedBlock::Handle::forEachFreeCell): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessarySlow): (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::needsFlip): (JSC::MarkedBlock::Handle::needsFlip): (JSC::MarkedBlock::Handle::willRemoveBlock): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::clearHasAnyMarked): (JSC::MarkedBlock::noteMarkedSlow): (WTF::printInternal): (JSC::MarkedBlock::create): Deleted. (JSC::MarkedBlock::destroy): Deleted. (JSC::MarkedBlock::callDestructor): Deleted. (JSC::MarkedBlock::specializedSweep): Deleted. (JSC::MarkedBlock::sweep): Deleted. (JSC::MarkedBlock::sweepHelper): Deleted. (JSC::MarkedBlock::stopAllocating): Deleted. (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted. (JSC::MarkedBlock::lastChanceToFinalize): Deleted. (JSC::MarkedBlock::resumeAllocating): Deleted. (JSC::MarkedBlock::didRetireBlock): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::VoidFunctor::returnValue): (JSC::MarkedBlock::CountFunctor::CountFunctor): (JSC::MarkedBlock::CountFunctor::count): (JSC::MarkedBlock::CountFunctor::returnValue): (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): (JSC::MarkedBlock::Handle::state): (JSC::MarkedBlock::needsDestruction): (JSC::MarkedBlock::handle): (JSC::MarkedBlock::Handle::block): (JSC::MarkedBlock::firstAtom): (JSC::MarkedBlock::atoms): (JSC::MarkedBlock::isAtomAligned): (JSC::MarkedBlock::Handle::cellAlign): (JSC::MarkedBlock::blockFor): (JSC::MarkedBlock::Handle::allocator): (JSC::MarkedBlock::Handle::heap): (JSC::MarkedBlock::Handle::vm): (JSC::MarkedBlock::vm): (JSC::MarkedBlock::Handle::weakSet): (JSC::MarkedBlock::weakSet): (JSC::MarkedBlock::Handle::shrink): (JSC::MarkedBlock::Handle::visitWeakSet): (JSC::MarkedBlock::Handle::reapWeakSet): (JSC::MarkedBlock::Handle::cellSize): (JSC::MarkedBlock::cellSize): (JSC::MarkedBlock::Handle::attributes): (JSC::MarkedBlock::attributes): (JSC::MarkedBlock::Handle::needsDestruction): (JSC::MarkedBlock::Handle::destruction): (JSC::MarkedBlock::Handle::cellKind): (JSC::MarkedBlock::Handle::markCount): (JSC::MarkedBlock::Handle::size): (JSC::MarkedBlock::atomNumber): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipForEdenCollection): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::Handle::assertFlipped): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::testAndSetMarked): (JSC::MarkedBlock::Handle::isNewlyAllocated): (JSC::MarkedBlock::Handle::setNewlyAllocated): (JSC::MarkedBlock::Handle::clearNewlyAllocated): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::isAtom): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::Handle::needsSweeping): (JSC::MarkedBlock::Handle::isAllocated): (JSC::MarkedBlock::Handle::isMarked): (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::hasAnyMarked): (JSC::MarkedBlock::noteMarked): (WTF::MarkedBlockHash::hash): (JSC::MarkedBlock::FreeList::FreeList): Deleted. (JSC::MarkedBlock::allocator): Deleted. (JSC::MarkedBlock::heap): Deleted. (JSC::MarkedBlock::shrink): Deleted. (JSC::MarkedBlock::visitWeakSet): Deleted. (JSC::MarkedBlock::reapWeakSet): Deleted. (JSC::MarkedBlock::willRemoveBlock): Deleted. (JSC::MarkedBlock::didConsumeFreeList): Deleted. (JSC::MarkedBlock::markCount): Deleted. (JSC::MarkedBlock::isEmpty): Deleted. (JSC::MarkedBlock::destruction): Deleted. (JSC::MarkedBlock::cellKind): Deleted. (JSC::MarkedBlock::size): Deleted. (JSC::MarkedBlock::capacity): Deleted. (JSC::MarkedBlock::setMarked): Deleted. (JSC::MarkedBlock::clearMarked): Deleted. (JSC::MarkedBlock::isNewlyAllocated): Deleted. (JSC::MarkedBlock::setNewlyAllocated): Deleted. (JSC::MarkedBlock::clearNewlyAllocated): Deleted. (JSC::MarkedBlock::isLive): Deleted. (JSC::MarkedBlock::isLiveCell): Deleted. (JSC::MarkedBlock::forEachCell): Deleted. (JSC::MarkedBlock::forEachLiveCell): Deleted. (JSC::MarkedBlock::forEachDeadCell): Deleted. (JSC::MarkedBlock::needsSweeping): Deleted. (JSC::MarkedBlock::isAllocated): Deleted. (JSC::MarkedBlock::isMarkedOrRetired): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::initializeSizeClassForStepSize): (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::lastChanceToFinalize): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::allocateLarge): (JSC::MarkedSpace::tryAllocateLarge): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::zombifySweep): (JSC::MarkedSpace::resetAllocators): (JSC::MarkedSpace::visitWeakSets): (JSC::MarkedSpace::reapWeakSets): (JSC::MarkedSpace::stopAllocating): (JSC::MarkedSpace::prepareForMarking): (JSC::MarkedSpace::resumeAllocating): (JSC::MarkedSpace::isPagedOut): (JSC::MarkedSpace::freeBlock): (JSC::MarkedSpace::freeOrShrinkBlock): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::VerifyMarked::operator()): (JSC::MarkedSpace::flip): (JSC::MarkedSpace::objectCount): (JSC::MarkedSpace::size): (JSC::MarkedSpace::capacity): (JSC::MarkedSpace::addActiveWeakSet): (JSC::MarkedSpace::didAddBlock): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::forEachAllocator): Deleted. (JSC::VerifyMarkedOrRetired::operator()): Deleted. (JSC::MarkedSpace::clearMarks): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::sizeClassToIndex): (JSC::MarkedSpace::indexToSizeClass): (JSC::MarkedSpace::version): (JSC::MarkedSpace::blocksWithNewObjects): (JSC::MarkedSpace::largeAllocations): (JSC::MarkedSpace::largeAllocationsNurseryOffset): (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection): (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin): (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd): (JSC::MarkedSpace::largeAllocationsForThisCollectionSize): (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): (JSC::MarkedSpace::allocatorFor): (JSC::MarkedSpace::destructorAllocatorFor): (JSC::MarkedSpace::auxiliaryAllocatorFor): (JSC::MarkedSpace::allocateWithoutDestructor): (JSC::MarkedSpace::allocateWithDestructor): (JSC::MarkedSpace::allocateAuxiliary): (JSC::MarkedSpace::tryAllocateAuxiliary): (JSC::MarkedSpace::forEachBlock): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::optimalSizeFor): (JSC::MarkedSpace::didAddBlock): Deleted. (JSC::MarkedSpace::didAllocateInBlock): Deleted. (JSC::MarkedSpace::objectCount): Deleted. (JSC::MarkedSpace::size): Deleted. (JSC::MarkedSpace::capacity): Deleted. * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::didStartMarking): (JSC::SlotVisitor::reset): (JSC::SlotVisitor::append): (JSC::SlotVisitor::appendJSCellOrAuxiliary): (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::appendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::noteLiveAuxiliaryCell): (JSC::SlotVisitor::visitChildren): * heap/SlotVisitor.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::WeakBlock): (JSC::WeakBlock::visit): (JSC::WeakBlock::reap): * heap/WeakBlock.h: (JSC::WeakBlock::disconnectContainer): (JSC::WeakBlock::disconnectMarkedBlock): Deleted. * heap/WeakSet.cpp: (JSC::WeakSet::~WeakSet): (JSC::WeakSet::sweep): (JSC::WeakSet::shrink): (JSC::WeakSet::addAllocator): * heap/WeakSet.h: (JSC::WeakSet::container): (JSC::WeakSet::setContainer): (JSC::WeakSet::WeakSet): (JSC::WeakSet::visit): (JSC::WeakSet::shrink): Deleted. * heap/WeakSetInlines.h: (JSC::WeakSet::allocate): * inspector/InjectedScriptManager.cpp: * inspector/JSGlobalObjectInspectorController.cpp: * inspector/JSJavaScriptCallFrame.cpp: * inspector/ScriptDebugServer.cpp: * inspector/agents/InspectorDebuggerAgent.cpp: * interpreter/CachedCall.h: (JSC::CachedCall::CachedCall): * interpreter/Interpreter.cpp: (JSC::loadVarargs): (JSC::StackFrame::sourceID): Deleted. (JSC::StackFrame::sourceURL): Deleted. (JSC::StackFrame::functionName): Deleted. (JSC::StackFrame::computeLineAndColumn): Deleted. (JSC::StackFrame::toString): Deleted. * interpreter/Interpreter.h: (JSC::StackFrame::isNative): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): (JSC::AssemblyHelpers::emitAllocate): (JSC::AssemblyHelpers::emitAllocateJSCell): (JSC::AssemblyHelpers::emitAllocateJSObject): (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): * jit/GCAwareJITStubRoutine.cpp: (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine): * jit/JIT.cpp: (JSC::JIT::compileCTINativeCall): (JSC::JIT::link): * jit/JIT.h: (JSC::JIT::compileCTINativeCall): Deleted. * jit/JITExceptions.cpp: (JSC::genericUnwind): * jit/JITExceptions.h: * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOpcodes32_64.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emitWriteBarrier): * jit/JITThunks.cpp: * jit/JITThunks.h: * jsc.cpp: (functionDescribeArray): (main): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntExceptions.cpp: * llint/LLIntThunks.cpp: * llint/LLIntThunks.h: * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter.cpp: * llint/LowLevelInterpreter32_64.asm: * llint/LowLevelInterpreter64.asm: * parser/ModuleAnalyzer.cpp: * parser/NodeConstructors.h: * parser/Nodes.h: * profiler/ProfilerBytecode.cpp: * profiler/ProfilerBytecode.h: * profiler/ProfilerBytecodeSequence.cpp: * runtime/ArrayConventions.h: (JSC::indexingHeaderForArrayStorage): (JSC::baseIndexingHeaderForArrayStorage): (JSC::indexingHeaderForArray): Deleted. (JSC::baseIndexingHeaderForArray): Deleted. * runtime/ArrayPrototype.cpp: (JSC::arrayProtoFuncSplice): (JSC::concatAppendOne): (JSC::arrayProtoPrivateFuncConcatMemcpy): * runtime/ArrayStorage.h: (JSC::ArrayStorage::vectorLength): (JSC::ArrayStorage::totalSizeFor): (JSC::ArrayStorage::totalSize): (JSC::ArrayStorage::availableVectorLength): (JSC::ArrayStorage::optimalVectorLength): (JSC::ArrayStorage::sizeFor): Deleted. * runtime/AuxiliaryBarrier.h: Added. (JSC::AuxiliaryBarrier::AuxiliaryBarrier): (JSC::AuxiliaryBarrier::clear): (JSC::AuxiliaryBarrier::get): (JSC::AuxiliaryBarrier::slot): (JSC::AuxiliaryBarrier::operator bool): (JSC::AuxiliaryBarrier::setWithoutBarrier): * runtime/AuxiliaryBarrierInlines.h: Added. (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier): (JSC::AuxiliaryBarrier<T>::set): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::availableContiguousVectorLength): (JSC::Butterfly::optimalContiguousVectorLength): (JSC::Butterfly::createUninitialized): (JSC::Butterfly::growArrayRight): * runtime/ClonedArguments.cpp: (JSC::ClonedArguments::createEmpty): * runtime/CommonSlowPathsExceptions.cpp: * runtime/CommonSlowPathsExceptions.h: * runtime/DataView.cpp: * runtime/DirectArguments.h: * runtime/ECMAScriptSpecInternalFunctions.cpp: * runtime/Error.cpp: * runtime/Error.h: * runtime/ErrorInstance.cpp: * runtime/ErrorInstance.h: * runtime/Exception.cpp: * runtime/Exception.h: * runtime/GeneratorFrame.cpp: * runtime/GeneratorPrototype.cpp: * runtime/InternalFunction.cpp: (JSC::InternalFunction::InternalFunction): * runtime/IntlCollator.cpp: * runtime/IntlCollatorConstructor.cpp: * runtime/IntlCollatorPrototype.cpp: * runtime/IntlDateTimeFormat.cpp: * runtime/IntlDateTimeFormatConstructor.cpp: * runtime/IntlDateTimeFormatPrototype.cpp: * runtime/IntlNumberFormat.cpp: * runtime/IntlNumberFormatConstructor.cpp: * runtime/IntlNumberFormatPrototype.cpp: * runtime/IntlObject.cpp: * runtime/IteratorPrototype.cpp: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitialized): (JSC::JSArray::setLengthWritable): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLengthWithArrayStorage): (JSC::JSArray::appendMemcpy): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithArrayStorage): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::createContiguousArrayButterfly): (JSC::createArrayButterfly): (JSC::JSArray::create): (JSC::JSArray::tryCreateUninitialized): Deleted. * runtime/JSArrayBufferView.h: * runtime/JSCInlines.h: * runtime/JSCJSValue.cpp: (JSC::JSValue::dumpInContextAssumingStructure): * runtime/JSCallee.cpp: (JSC::JSCallee::JSCallee): * runtime/JSCell.cpp: (JSC::JSCell::estimatedSize): * runtime/JSCell.h: (JSC::JSCell::cellStateOffset): Deleted. * runtime/JSCellInlines.h: (JSC::ExecState::vm): (JSC::JSCell::classInfo): (JSC::JSCell::callDestructor): (JSC::JSCell::vm): Deleted. * runtime/JSFunction.cpp: (JSC::JSFunction::create): (JSC::JSFunction::allocateAndInitializeRareData): (JSC::JSFunction::initializeRareData): (JSC::JSFunction::getOwnPropertySlot): (JSC::JSFunction::put): (JSC::JSFunction::deleteProperty): (JSC::JSFunction::defineOwnProperty): (JSC::JSFunction::setFunctionName): (JSC::JSFunction::reifyLength): (JSC::JSFunction::reifyName): (JSC::JSFunction::reifyLazyPropertyIfNeeded): (JSC::JSFunction::reifyBoundNameIfNeeded): * runtime/JSFunction.h: * runtime/JSFunctionInlines.h: (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint): (JSC::JSFunction::JSFunction): * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory): * runtime/JSInternalPromise.cpp: * runtime/JSInternalPromiseConstructor.cpp: * runtime/JSInternalPromiseDeferred.cpp: * runtime/JSInternalPromisePrototype.cpp: * runtime/JSJob.cpp: * runtime/JSMapIterator.cpp: * runtime/JSModuleNamespaceObject.cpp: * runtime/JSModuleRecord.cpp: * runtime/JSObject.cpp: (JSC::JSObject::visitButterfly): (JSC::JSObject::notifyPresenceOfIndexedAccessors): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createInitialUndecided): (JSC::JSObject::createInitialInt32): (JSC::JSObject::createInitialDouble): (JSC::JSObject::createInitialContiguous): (JSC::JSObject::createArrayStorage): (JSC::JSObject::createInitialArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::putByIndexBeyondVectorLength): (JSC::JSObject::putDirectIndexBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::increaseVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::growOutOfLineStorage): (JSC::JSObject::copyButterfly): Deleted. (JSC::JSObject::copyBackingStore): Deleted. * runtime/JSObject.h: (JSC::JSObject::globalObject): (JSC::JSObject::putDirectInternal): (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted. * runtime/JSObjectInlines.h: * runtime/JSPromise.cpp: * runtime/JSPromiseConstructor.cpp: * runtime/JSPromiseDeferred.cpp: * runtime/JSPromisePrototype.cpp: * runtime/JSPropertyNameIterator.cpp: * runtime/JSScope.cpp: (JSC::JSScope::resolve): * runtime/JSScope.h: (JSC::JSScope::globalObject): (JSC::JSScope::vm): Deleted. * runtime/JSSetIterator.cpp: * runtime/JSStringIterator.cpp: * runtime/JSTemplateRegistryKey.cpp: * runtime/JSTypedArrayViewConstructor.cpp: * runtime/JSTypedArrayViewPrototype.cpp: * runtime/JSWeakMap.cpp: * runtime/JSWeakSet.cpp: * runtime/MapConstructor.cpp: * runtime/MapIteratorPrototype.cpp: * runtime/MapPrototype.cpp: * runtime/NativeErrorConstructor.cpp: * runtime/NativeStdFunctionCell.cpp: * runtime/Operations.h: (JSC::scribbleFreeCells): (JSC::scribble): * runtime/Options.h: * runtime/PropertyTable.cpp: * runtime/ProxyConstructor.cpp: * runtime/ProxyObject.cpp: * runtime/ProxyRevoke.cpp: * runtime/RegExp.cpp: (JSC::RegExp::match): (JSC::RegExp::matchConcurrently): (JSC::RegExp::matchCompareWithInterpreter): * runtime/RegExp.h: * runtime/RegExpConstructor.h: * runtime/RegExpInlines.h: (JSC::RegExp::matchInline): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): (JSC::createRegExpMatchesArray): * runtime/RegExpPrototype.cpp: (JSC::genericSplit): * runtime/RuntimeType.cpp: * runtime/SamplingProfiler.cpp: (JSC::SamplingProfiler::processUnverifiedStackTraces): * runtime/SetConstructor.cpp: * runtime/SetIteratorPrototype.cpp: * runtime/SetPrototype.cpp: * runtime/StackFrame.cpp: Added. (JSC::StackFrame::sourceID): (JSC::StackFrame::sourceURL): (JSC::StackFrame::functionName): (JSC::StackFrame::computeLineAndColumn): (JSC::StackFrame::toString): * runtime/StackFrame.h: Added. (JSC::StackFrame::isNative): * runtime/StringConstructor.cpp: * runtime/StringIteratorPrototype.cpp: * runtime/StructureInlines.h: (JSC::Structure::propertyTable): * runtime/TemplateRegistry.cpp: * runtime/TestRunnerUtils.cpp: (JSC::finalizeStatsAtEndOfTesting): * runtime/TestRunnerUtils.h: * runtime/TypeProfilerLog.cpp: * runtime/TypeSet.cpp: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::ensureStackCapacityForCLoop): (JSC::VM::isSafeToRecurseSoftCLoop): * runtime/VM.h: * runtime/VMEntryScope.h: * runtime/VMInlines.h: (JSC::VM::ensureStackCapacityFor): (JSC::VM::isSafeToRecurseSoft): * runtime/WeakMapConstructor.cpp: * runtime/WeakMapData.cpp: * runtime/WeakMapPrototype.cpp: * runtime/WeakSetConstructor.cpp: * runtime/WeakSetPrototype.cpp: * testRegExp.cpp: (testOneRegExp): * tools/JSDollarVM.cpp: * tools/JSDollarVMPrototype.cpp: (JSC::JSDollarVMPrototype::isInObjectSpace): Source/WebCore: No new tests because no new WebCore behavior. Just rewiring #includes. * ForwardingHeaders/heap/HeapInlines.h: Added. * ForwardingHeaders/interpreter/Interpreter.h: Removed. * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added. * Modules/indexeddb/IDBCursorWithValue.cpp: * Modules/indexeddb/client/TransactionOperation.cpp: * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp: * Modules/indexeddb/server/UniqueIDBDatabase.cpp: * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp: * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp: * bindings/js/JSClientRectCustom.cpp: * bindings/js/JSDOMBinding.cpp: * bindings/js/JSDOMBinding.h: * bindings/js/JSDeviceMotionEventCustom.cpp: * bindings/js/JSDeviceOrientationEventCustom.cpp: * bindings/js/JSErrorEventCustom.cpp: * bindings/js/JSIDBCursorWithValueCustom.cpp: * bindings/js/JSIDBIndexCustom.cpp: * bindings/js/JSPopStateEventCustom.cpp: * bindings/js/JSWebGL2RenderingContextCustom.cpp: * bindings/js/JSWorkerGlobalScopeCustom.cpp: * bindings/js/WorkerScriptController.cpp: * contentextensions/ContentExtensionParser.cpp: * dom/ErrorEvent.cpp: * html/HTMLCanvasElement.cpp: * html/MediaDocument.cpp: * inspector/CommandLineAPIModule.cpp: * loader/EmptyClients.cpp: * page/CaptionUserPreferences.cpp: * page/Frame.cpp: * page/PageGroup.cpp: * page/UserContentController.cpp: * platform/mock/mediasource/MockBox.cpp: * testing/GCObservation.cpp: Source/WebKit2: Just rewiring some #includes. * UIProcess/ViewGestureController.cpp: * UIProcess/WebPageProxy.cpp: * UIProcess/WebProcessPool.cpp: * UIProcess/WebProcessProxy.cpp: * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp: * WebProcess/Plugins/Netscape/JSNPObject.cpp: Source/WTF: I needed tryFastAlignedMalloc() so I added it. * wtf/FastMalloc.cpp: (WTF::tryFastAlignedMalloc): * wtf/FastMalloc.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): (WTF::ParkingLot::forEach): Deleted. * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::forEach): * wtf/ScopedLambda.h: (WTF::scopedLambdaRef): * wtf/SentinelLinkedList.h: (WTF::SentinelLinkedList::forEach): (WTF::RawNode>::takeFrom): * wtf/SimpleStats.h: (WTF::SimpleStats::operator bool): (WTF::SimpleStats::operator!): Deleted. Tools: * DumpRenderTree/TestRunner.cpp: * DumpRenderTree/mac/DumpRenderTree.mm: (DumpRenderTreeMain): * Scripts/run-jsc-stress-tests: * TestWebKitAPI/Tests/WTF/Vector.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/179778@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
const ValidationFunctor& validation,
const BeforeSleepFunctor& beforeSleep,
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows https://bugs.webkit.org/show_bug.cgi?id=152045 Reviewed by Andy Estes. Source/JavaScriptCore: Probably the nicest example of why this patch is a good idea is the change in AtomicsObject.cpp. * jit/ICStats.cpp: (JSC::ICStats::ICStats): * runtime/AtomicsObject.cpp: (JSC::atomicsFuncWait): Source/WebCore: No new layout tests because no new behavior. The new WTF time classes have some unit tests in TestWebKitAPI. * fileapi/ThreadableBlobRegistry.cpp: (WebCore::ThreadableBlobRegistry::blobSize): * platform/MainThreadSharedTimer.h: * platform/SharedTimer.h: * platform/ThreadTimers.cpp: (WebCore::ThreadTimers::updateSharedTimer): * platform/cf/MainThreadSharedTimerCF.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/efl/MainThreadSharedTimerEfl.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/glib/MainThreadSharedTimerGLib.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/win/MainThreadSharedTimerWin.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * workers/WorkerRunLoop.cpp: (WebCore::WorkerRunLoop::runInMode): Source/WebKit2: * Platform/IPC/Connection.cpp: (IPC::Connection::SyncMessageState::wait): (IPC::Connection::sendMessage): (IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting): (IPC::Connection::waitForMessage): (IPC::Connection::sendSyncMessage): (IPC::Connection::waitForSyncReply): * Platform/IPC/Connection.h: (IPC::Connection::sendSync): (IPC::Connection::waitForAndDispatchImmediately): * Platform/IPC/MessageSender.h: (IPC::MessageSender::sendSync): * UIProcess/ChildProcessProxy.h: (WebKit::ChildProcessProxy::sendSync): * UIProcess/Network/NetworkProcessProxy.cpp: (WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently): * UIProcess/Storage/StorageManager.cpp: (WebKit::StorageManager::applicationWillTerminate): * UIProcess/WebProcessProxy.cpp: (WebKit::WebProcessProxy::sendProcessWillSuspendImminently): * UIProcess/WebResourceLoadStatisticsStore.cpp: (WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate): * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h: * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm: (-[WKOneShotDisplayLinkHandler displayLinkFired:]): (WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree): (WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay): (WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm: (WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/WKImmediateActionController.mm: (-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]): * UIProcess/mac/WebPageProxyMac.mm: (WebKit::WebPageProxy::stringSelectionForPasteboard): (WebKit::WebPageProxy::dataSelectionForPasteboard): (WebKit::WebPageProxy::readSelectionFromPasteboard): (WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent): (WebKit::WebPageProxy::acceptsFirstMouse): * WebProcess/WebCoreSupport/WebChromeClient.cpp: (WebKit::WebChromeClient::runBeforeUnloadConfirmPanel): (WebKit::WebChromeClient::runJavaScriptAlert): (WebKit::WebChromeClient::runJavaScriptConfirm): (WebKit::WebChromeClient::runJavaScriptPrompt): (WebKit::WebChromeClient::print): (WebKit::WebChromeClient::exceededDatabaseQuota): (WebKit::WebChromeClient::reachedApplicationCacheOriginQuota): * WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp: (WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse): * WebProcess/WebPage/WebPage.cpp: (WebKit::WebPage::postSynchronousMessageForTesting): Source/WTF: We used to use 'double' for all time measurements. Sometimes it was milliseconds, sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time since some epoch. When we spoke of time since epoch, we either used a monotonic clock or a wall clock. The type - always 'double' - never told us what kind of time we had, even though there were roughly six of them (sec interval, ms interval, sec since epoch on wall, ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic). At some point, we thought that it would be a good idea to replace these doubles with std::chrono. But since replacing some things with std::chrono, we found it to be terribly inconvenient: - Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally duration_cast would not even be a thing. - No overflow protection. std::chrono uses integers by default and using anything else is clumsy. But the integer math is done without regard for the rough edges of integer math, so any cast between std::chrono types risks overflow. Any comparison risks overflow because it may do conversions silently. We have even found bugs where some C++ implementations had more overflows than others, which ends up being a special kind of hell. In many cases, the overflow also has nasal demons. It's an error to represent time using integers. It would have been excusable back when floating point math was not guaranteed to be supported on all platforms, but that would have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit for floating point: - Floating point preserves precision under multiplication in all but extreme cases, so using floating point for time means that unit conversions are almost completely lossless. This means that we don't have to think very hard about what units to use. In this patch, we use seconds almost everywhere. We only convert at boundaries, like an API boundary that wants something other than seconds. - Floating point makes it easy to reason about infinity, which is something that time code wants to do a lot. Example: when would you like to timeout? Infinity please! This is the most elegant way of having an API support both a timeout variant and a no-timeout variant. - Floating point does well-understood things when math goes wrong, and these things are pretty well optimized to match what a mathematician would do when computing with real numbers represented using scientific notation with a finite number of significant digits. This means that time math under floating point looks like normal math. On the other hand, std::chrono time math looks like garbage because you have to always check for multiple possible UB corners whenever you touch large integers. Integers that represent time are very likely to be large and you don't have to do much to overflow them. At this time, based on the number of bugs we have already seen due to chrono overflows, I am not certain that we even understand what are all of the corner cases that we should even check for. This patch introduces a new set of timekeeping classes that are all based on double, and all internally use seconds. These classes support algebraic typing. The classes are: - Seconds: this is for measuring a duration. - WallTime: time since epoch according to a wall clock (aka real time clock). - MonotonicTime: time since epoch according to a monotonic clock. - ClockType: enum that says either Wall or Monotonic. - TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a wall time or a monotonic time. All of these classes behave like C++ values and are cheap to copy around since they are very nearly POD. This supports comprehensive conversions between the various time types. Most of this is by way of algebra. Here are just some of the rules we recognize: WallTime = WallTime + Seconds Seconds = WallTime - WallTime MonotonicTime = MonotonicTime + Seconds etc... We support negative, infinite, and NaN times because math. We support conversions between MonotonicTime and WallTime, like: WallTime wt = mt.approximateWallTime() This is called this "approximate" because the only way to do it is to get the current time on both clocks and convert relative to that. Many of our APIs would be happy using whatever notion of time the user wanted to use. For those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This means that if you use a WallTime with Condition::waitUntil, then Condition's internal logic for when it should wake up makes its decision based on the current WallTime - but if you use MonotonicTime then waitUntil will make its decision based on current MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono did not have the concept of a dynamic clock type. This patch does not include conversions between std::chrono and these new time classes, because past experience shows that we're quite bad at getting conversions between std::chrono and anything else right. Also, I didn't need such conversion code because this patch only converts code that transitively touches ParkingLot and Condition. It was easy to get all of that code onto the new time classes. * WTF.xcodeproj/project.pbxproj: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/ClockType.cpp: Added. (WTF::printInternal): * wtf/ClockType.h: Added. * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::waitFor): (WTF::ConditionBase::wait): (WTF::ConditionBase::waitUntilWallClockSeconds): Deleted. (WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted. (WTF::ConditionBase::waitForSeconds): Deleted. (WTF::ConditionBase::waitForSecondsImpl): Deleted. (WTF::ConditionBase::waitForImpl): Deleted. (WTF::ConditionBase::absoluteFromRelative): Deleted. * wtf/CrossThreadQueue.h: (WTF::CrossThreadQueue<DataType>::waitForMessage): * wtf/CurrentTime.cpp: (WTF::sleep): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): Deleted. * wtf/MonotonicTime.cpp: Added. (WTF::MonotonicTime::now): (WTF::MonotonicTime::approximateWallTime): (WTF::MonotonicTime::dump): (WTF::MonotonicTime::sleep): * wtf/MonotonicTime.h: Added. (WTF::MonotonicTime::MonotonicTime): (WTF::MonotonicTime::fromRawDouble): (WTF::MonotonicTime::infinity): (WTF::MonotonicTime::secondsSinceEpoch): (WTF::MonotonicTime::approximateMonotonicTime): (WTF::MonotonicTime::operator bool): (WTF::MonotonicTime::operator+): (WTF::MonotonicTime::operator-): (WTF::MonotonicTime::operator+=): (WTF::MonotonicTime::operator-=): (WTF::MonotonicTime::operator==): (WTF::MonotonicTime::operator!=): (WTF::MonotonicTime::operator<): (WTF::MonotonicTime::operator>): (WTF::MonotonicTime::operator<=): (WTF::MonotonicTime::operator>=): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkCount): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): * wtf/Seconds.cpp: Added. (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::dump): (WTF::Seconds::sleep): * wtf/Seconds.h: Added. (WTF::Seconds::Seconds): (WTF::Seconds::value): (WTF::Seconds::seconds): (WTF::Seconds::milliseconds): (WTF::Seconds::microseconds): (WTF::Seconds::nanoseconds): (WTF::Seconds::fromMilliseconds): (WTF::Seconds::fromMicroseconds): (WTF::Seconds::fromNanoseconds): (WTF::Seconds::infinity): (WTF::Seconds::operator bool): (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::operator*): (WTF::Seconds::operator/): (WTF::Seconds::operator+=): (WTF::Seconds::operator-=): (WTF::Seconds::operator*=): (WTF::Seconds::operator/=): (WTF::Seconds::operator==): (WTF::Seconds::operator!=): (WTF::Seconds::operator<): (WTF::Seconds::operator>): (WTF::Seconds::operator<=): (WTF::Seconds::operator>=): * wtf/TimeWithDynamicClockType.cpp: Added. (WTF::TimeWithDynamicClockType::now): (WTF::TimeWithDynamicClockType::nowWithSameClock): (WTF::TimeWithDynamicClockType::wallTime): (WTF::TimeWithDynamicClockType::monotonicTime): (WTF::TimeWithDynamicClockType::approximateWallTime): (WTF::TimeWithDynamicClockType::approximateMonotonicTime): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator<): (WTF::TimeWithDynamicClockType::operator>): (WTF::TimeWithDynamicClockType::operator<=): (WTF::TimeWithDynamicClockType::operator>=): (WTF::TimeWithDynamicClockType::dump): (WTF::TimeWithDynamicClockType::sleep): * wtf/TimeWithDynamicClockType.h: Added. (WTF::TimeWithDynamicClockType::TimeWithDynamicClockType): (WTF::TimeWithDynamicClockType::fromRawDouble): (WTF::TimeWithDynamicClockType::secondsSinceEpoch): (WTF::TimeWithDynamicClockType::clockType): (WTF::TimeWithDynamicClockType::withSameClockAndRawDouble): (WTF::TimeWithDynamicClockType::operator bool): (WTF::TimeWithDynamicClockType::operator+): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator+=): (WTF::TimeWithDynamicClockType::operator-=): (WTF::TimeWithDynamicClockType::operator==): (WTF::TimeWithDynamicClockType::operator!=): * wtf/WallTime.cpp: Added. (WTF::WallTime::now): (WTF::WallTime::approximateMonotonicTime): (WTF::WallTime::dump): (WTF::WallTime::sleep): * wtf/WallTime.h: Added. (WTF::WallTime::WallTime): (WTF::WallTime::fromRawDouble): (WTF::WallTime::infinity): (WTF::WallTime::secondsSinceEpoch): (WTF::WallTime::approximateWallTime): (WTF::WallTime::operator bool): (WTF::WallTime::operator+): (WTF::WallTime::operator-): (WTF::WallTime::operator+=): (WTF::WallTime::operator-=): (WTF::WallTime::operator==): (WTF::WallTime::operator!=): (WTF::WallTime::operator<): (WTF::WallTime::operator>): (WTF::WallTime::operator<=): (WTF::WallTime::operator>=): * wtf/threads/BinarySemaphore.cpp: (WTF::BinarySemaphore::wait): * wtf/threads/BinarySemaphore.h: Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp: (TestWebKitAPI::ToUpperConverter::stopProducing): (TestWebKitAPI::ToUpperConverter::stopConsuming): * TestWebKitAPI/Tests/WTF/Time.cpp: Added. (WTF::operator<<): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/182152@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
const TimeWithDynamicClockType& timeout)
{
return parkConditionallyImpl(
address,
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression https://bugs.webkit.org/show_bug.cgi?id=160125 Reviewed by Geoffrey Garen and Keith Miller. JSTests: Most of the things I did properly covered by existing tests, but I found some simple cases of unshifting that had sketchy coverage. * stress/array-storage-array-unshift.js: Added. * stress/contiguous-array-unshift.js: Added. * stress/double-array-unshift.js: Added. * stress/int32-array-unshift.js: Added. Source/bmalloc: I needed to tryMemalign, so I added such a thing. * bmalloc/Allocator.cpp: (bmalloc::Allocator::allocate): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): * bmalloc/Allocator.h: * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): * bmalloc/bmalloc.h: (bmalloc::api::tryMemalign): Source/JavaScriptCore: In order to make the GC concurrent (bug 149432), we would either need to enable concurrent copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead from the barriers alone. Considering that MarkedSpace does a decent job of avoiding fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want to get rid of copied space. This change moves copied space's biggest client over to marked space. Moving butterflies to marked space means having them use the new Auxiliary HeapCell allocation path. This is a fairly mechanical change, but it caused performance regressions everywhere, so this change also fixes MarkedSpace's performance issues. At a high level the mechanical changes are: - We use AuxiliaryBarrier instead of CopyBarrier. - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth. - The JITs have to emit inlined marked space allocations instead of inline copy space allocations. - Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't. That's about 1/3 of this change. But this led to performance problems, which I fixed with optimizations that amounted to a major MarkedSpace rewrite: - MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely. - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well. - Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being: 16, 32, 48, 64, 80, 112, 160, 224, 320 The next size class would have been 432, but that's above the 400 byte cutoff. All of this is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what size classes you end up with by doing --dumpSizeClasses=true. - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: https://bugs.webkit.org/show_bug.cgi?id=161581. - Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree. - FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky. - It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch! - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes. All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1% regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a best-case scenario for bump allocation. I think that we should fix bmalloc to make up the difference, but take the hit for now because it's a crazy corner case. By comparison, the alternative approach of using a copy barrier would have cost us 1-2%. That's the real apples-to-apples comparison if your premise is that we should have a concurrent GC. After we finish removing copied space, we will be barrier-ready for concurrent GC: we already have a marking barrier and we simply won't need a copying barrier. This change gets us there for the purposes of our benchmarks, since the remaining clients of copied space are not very important. On the other hand, if we keep copying, then getting barrier-ready would mean adding back the copy barrier, which costs more perf. We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving typed arrays and a few other weird things over to Aux MarkedSpace. This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell, and CellContainer meant that I had to include those files from everywhere. Fortunately, just including JSCInlines.h (instead of manually including the files that includes) is usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something that we were already basically doing. In places where JSCInlines.h would be too much, I just included HeapInlines.h. This got weird, because we previously included HeapInlines.h from JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that meant having to manually include HeapInlines.h from the places that previously got it implicitly via JSObject.h. But that led to more problems for some reason: I started getting build errors because non-JSC files were having trouble including Opcode.h. That's just silly, since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and made it impossible to include it from outside JSC. This was a lot of work, but it was necessary to get the patch to build on all ports. It's also a net win. There were many places in WebCore that were transitively including a *ton* of JSC headers just because of the JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons. * API/JSManagedValue.mm: (-[JSManagedValue initWithValue:]): * API/JSTypedArray.cpp: * API/ObjCCallbackFunction.mm: * API/tests/testapi.mm: (testObjectiveCAPI): (testWeakValue): Deleted. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * Scripts/builtins/builtins_generate_combined_implementation.py: (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py: (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_separate_implementation.py: (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes): * assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::JumpList::link): (JSC::AbstractMacroAssembler::JumpList::linkTo): * assembler/MacroAssembler.h: * assembler/MacroAssemblerARM64.h: (JSC::MacroAssemblerARM64::add32): * assembler/MacroAssemblerCodeRef.cpp: Added. (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): (JSC::MacroAssemblerCodePtr::dumpWithName): (JSC::MacroAssemblerCodePtr::dump): (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): (JSC::MacroAssemblerCodeRef::dump): * assembler/MacroAssemblerCodeRef.h: (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted. (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted. (JSC::MacroAssemblerCodePtr::dump): Deleted. (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted. (JSC::MacroAssemblerCodeRef::dump): Deleted. * b3/B3BasicBlock.cpp: (JSC::B3::BasicBlock::appendBoolConstant): * b3/B3BasicBlock.h: * b3/B3DuplicateTails.cpp: * b3/B3StackmapGenerationParams.h: * b3/testb3.cpp: (JSC::B3::testPatchpointTerminalReturnValue): (JSC::B3::run): * bindings/ScriptValue.cpp: * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeLivenessAnalysis.cpp: * bytecode/BytecodeUseDef.h: * bytecode/CallLinkInfo.cpp: (JSC::CallLinkInfo::callTypeFor): * bytecode/CallLinkInfo.h: (JSC::CallLinkInfo::callTypeFor): Deleted. * bytecode/CallLinkStatus.cpp: * bytecode/CodeBlock.cpp: (JSC::CodeBlock::finishCreation): (JSC::CodeBlock::clearLLIntGetByIdCache): (JSC::CodeBlock::predictedMachineCodeSize): * bytecode/CodeBlock.h: (JSC::CodeBlock::jitCodeMap): Deleted. (JSC::clearLLIntGetByIdCache): Deleted. * bytecode/ExecutionCounter.h: * bytecode/Instruction.h: * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp: (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal): * bytecode/ObjectAllocationProfile.h: (JSC::ObjectAllocationProfile::isNull): (JSC::ObjectAllocationProfile::initialize): * bytecode/Opcode.h: (JSC::padOpcodeName): * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): (JSC::PolymorphicAccess::regenerate): * bytecode/PolymorphicAccess.h: * bytecode/PreciseJumpTargets.cpp: * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecode/UnlinkedCodeBlock.cpp: (JSC::UnlinkedCodeBlock::vm): Deleted. * bytecode/UnlinkedCodeBlock.h: * bytecode/UnlinkedInstructionStream.cpp: * bytecode/UnlinkedInstructionStream.h: * dfg/DFGOperations.cpp: * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT.h: (JSC::DFG::SpeculativeJIT::emitAllocateJSCell): (JSC::DFG::SpeculativeJIT::emitAllocateJSObject): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGStrengthReductionPhase.cpp: (JSC::DFG::StrengthReductionPhase::handleNode): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCompile.cpp: * ftl/FTLJITFinalizer.cpp: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments): (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest): (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell): (JSC::FTL::DFG::LowerDFGToB3::allocateCell): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted. * ftl/FTLOutput.cpp: (JSC::FTL::Output::constBool): (JSC::FTL::Output::add): (JSC::FTL::Output::shl): (JSC::FTL::Output::aShr): (JSC::FTL::Output::lShr): (JSC::FTL::Output::zeroExt): (JSC::FTL::Output::equal): (JSC::FTL::Output::notEqual): (JSC::FTL::Output::above): (JSC::FTL::Output::aboveOrEqual): (JSC::FTL::Output::below): (JSC::FTL::Output::belowOrEqual): (JSC::FTL::Output::greaterThan): (JSC::FTL::Output::greaterThanOrEqual): (JSC::FTL::Output::lessThan): (JSC::FTL::Output::lessThanOrEqual): (JSC::FTL::Output::select): (JSC::FTL::Output::appendSuccessor): (JSC::FTL::Output::addIncomingToPhi): * ftl/FTLOutput.h: * ftl/FTLValueFromBlock.h: (JSC::FTL::ValueFromBlock::operator bool): (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted. * ftl/FTLWeightedTarget.h: (JSC::FTL::WeightedTarget::frequentedBlock): * heap/CellContainer.h: Added. (JSC::CellContainer::CellContainer): (JSC::CellContainer::operator bool): (JSC::CellContainer::isMarkedBlock): (JSC::CellContainer::isLargeAllocation): (JSC::CellContainer::markedBlock): (JSC::CellContainer::largeAllocation): * heap/CellContainerInlines.h: Added. (JSC::CellContainer::isMarked): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::noteMarked): (JSC::CellContainer::cellSize): (JSC::CellContainer::weakSet): (JSC::CellContainer::flipIfNecessary): * heap/ConservativeRoots.cpp: (JSC::ConservativeRoots::ConservativeRoots): (JSC::ConservativeRoots::~ConservativeRoots): (JSC::ConservativeRoots::grow): (JSC::ConservativeRoots::genericAddPointer): (JSC::ConservativeRoots::genericAddSpan): * heap/ConservativeRoots.h: (JSC::ConservativeRoots::roots): * heap/CopyToken.h: * heap/FreeList.cpp: Added. (JSC::FreeList::dump): * heap/FreeList.h: Added. (JSC::FreeList::FreeList): (JSC::FreeList::list): (JSC::FreeList::bump): (JSC::FreeList::operator==): (JSC::FreeList::operator!=): (JSC::FreeList::operator bool): (JSC::FreeList::allocationWillFail): (JSC::FreeList::allocationWillSucceed): * heap/GCTypeMap.h: Added. (JSC::GCTypeMap::operator[]): * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalizeUnconditionalFinalizers): (JSC::Heap::markRoots): (JSC::Heap::copyBackingStores): (JSC::Heap::gatherStackRoots): (JSC::Heap::gatherJSStackRoots): (JSC::Heap::gatherScratchBufferRoots): (JSC::Heap::clearLivenessData): (JSC::Heap::visitSmallStrings): (JSC::Heap::visitConservativeRoots): (JSC::Heap::removeDeadCompilerWorklistEntries): (JSC::Heap::gatherExtraHeapSnapshotData): (JSC::Heap::removeDeadHeapSnapshotNodes): (JSC::Heap::visitProtectedObjects): (JSC::Heap::visitArgumentBuffers): (JSC::Heap::visitException): (JSC::Heap::visitStrongHandles): (JSC::Heap::visitHandleStack): (JSC::Heap::visitSamplingProfiler): (JSC::Heap::traceCodeBlocksAndJITStubRoutines): (JSC::Heap::converge): (JSC::Heap::visitWeakHandles): (JSC::Heap::updateObjectCounts): (JSC::Heap::clearUnmarkedExecutables): (JSC::Heap::deleteUnmarkedCompiledCode): (JSC::Heap::collectAllGarbage): (JSC::Heap::collect): (JSC::Heap::collectWithoutAnySweep): (JSC::Heap::collectImpl): (JSC::Heap::suspendCompilerThreads): (JSC::Heap::willStartCollection): (JSC::Heap::flushOldStructureIDTables): (JSC::Heap::flushWriteBarrierBuffer): (JSC::Heap::stopAllocation): (JSC::Heap::prepareForMarking): (JSC::Heap::reapWeakHandles): (JSC::Heap::pruneStaleEntriesFromWeakGCMaps): (JSC::Heap::sweepArrayBuffers): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): (JSC::MarkedBlockSnapshotFunctor::operator()): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::deleteSourceProviderCaches): (JSC::Heap::notifyIncrementalSweeper): (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks): (JSC::Heap::resetAllocators): (JSC::Heap::updateAllocationLimits): (JSC::Heap::didFinishCollection): (JSC::Heap::resumeCompilerThreads): (JSC::Zombify::visit): (JSC::Heap::forEachCodeBlockImpl): * heap/Heap.h: (JSC::Heap::allocatorForObjectWithoutDestructor): (JSC::Heap::allocatorForObjectWithDestructor): (JSC::Heap::allocatorForAuxiliaryData): (JSC::Heap::jitStubRoutines): (JSC::Heap::codeBlockSet): (JSC::Heap::storageAllocator): Deleted. * heap/HeapCell.h: (JSC::HeapCell::isZapped): Deleted. * heap/HeapCellInlines.h: Added. (JSC::HeapCell::isLargeAllocation): (JSC::HeapCell::cellContainer): (JSC::HeapCell::markedBlock): (JSC::HeapCell::largeAllocation): (JSC::HeapCell::heap): (JSC::HeapCell::vm): (JSC::HeapCell::cellSize): (JSC::HeapCell::allocatorAttributes): (JSC::HeapCell::destructionMode): (JSC::HeapCell::cellKind): * heap/HeapInlines.h: (JSC::Heap::heap): (JSC::Heap::isLive): (JSC::Heap::isMarked): (JSC::Heap::testAndSetMarked): (JSC::Heap::setMarked): (JSC::Heap::cellSize): (JSC::Heap::forEachCodeBlock): (JSC::Heap::allocateObjectOfType): (JSC::Heap::subspaceForObjectOfType): (JSC::Heap::allocatorForObjectOfType): (JSC::Heap::allocateAuxiliary): (JSC::Heap::tryAllocateAuxiliary): (JSC::Heap::tryReallocateAuxiliary): (JSC::Heap::isPointerGCObject): Deleted. (JSC::Heap::isValueGCObject): Deleted. * heap/HeapOperation.cpp: Added. (WTF::printInternal): * heap/HeapOperation.h: * heap/HeapUtil.h: Added. (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): (JSC::HeapUtil::isValueGCObject): * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::sweepNextBlock): * heap/IncrementalSweeper.h: * heap/LargeAllocation.cpp: Added. (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::lastChanceToFinalize): (JSC::LargeAllocation::shrink): (JSC::LargeAllocation::visitWeakSet): (JSC::LargeAllocation::reapWeakSet): (JSC::LargeAllocation::flip): (JSC::LargeAllocation::isEmpty): (JSC::LargeAllocation::sweep): (JSC::LargeAllocation::destroy): (JSC::LargeAllocation::dump): * heap/LargeAllocation.h: Added. (JSC::LargeAllocation::fromCell): (JSC::LargeAllocation::cell): (JSC::LargeAllocation::isLargeAllocation): (JSC::LargeAllocation::heap): (JSC::LargeAllocation::vm): (JSC::LargeAllocation::weakSet): (JSC::LargeAllocation::clearNewlyAllocated): (JSC::LargeAllocation::isNewlyAllocated): (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::isLive): (JSC::LargeAllocation::hasValidCell): (JSC::LargeAllocation::cellSize): (JSC::LargeAllocation::aboveLowerBound): (JSC::LargeAllocation::belowUpperBound): (JSC::LargeAllocation::contains): (JSC::LargeAllocation::attributes): (JSC::LargeAllocation::flipIfNecessary): (JSC::LargeAllocation::flipIfNecessaryConcurrently): (JSC::LargeAllocation::testAndSetMarked): (JSC::LargeAllocation::setMarked): (JSC::LargeAllocation::clearMarked): (JSC::LargeAllocation::noteMarked): (JSC::LargeAllocation::headerSize): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::retire): (JSC::MarkedAllocator::filterNextBlock): (JSC::MarkedAllocator::setNextBlockToSweep): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::blockHeaderSize): (JSC::MarkedAllocator::blockSizeForBytes): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::reset): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::setFreeList): (JSC::isListPagedOut): Deleted. (JSC::MarkedAllocator::tryAllocateHelper): Deleted. (JSC::MarkedAllocator::tryPopFreeList): Deleted. (JSC::MarkedAllocator::tryAllocate): Deleted. (JSC::MarkedAllocator::allocateBlock): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::takeLastActiveBlock): (JSC::MarkedAllocator::offsetOfFreeList): (JSC::MarkedAllocator::offsetOfCellSize): (JSC::MarkedAllocator::tryAllocate): (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted. (JSC::MarkedAllocator::MarkedAllocator): Deleted. (JSC::MarkedAllocator::init): Deleted. (JSC::MarkedAllocator::stopAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): (JSC::SetNewlyAllocatedFunctor::operator()): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::Handle::zap): (JSC::MarkedBlock::Handle::forEachFreeCell): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessarySlow): (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::needsFlip): (JSC::MarkedBlock::Handle::needsFlip): (JSC::MarkedBlock::Handle::willRemoveBlock): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::clearHasAnyMarked): (JSC::MarkedBlock::noteMarkedSlow): (WTF::printInternal): (JSC::MarkedBlock::create): Deleted. (JSC::MarkedBlock::destroy): Deleted. (JSC::MarkedBlock::callDestructor): Deleted. (JSC::MarkedBlock::specializedSweep): Deleted. (JSC::MarkedBlock::sweep): Deleted. (JSC::MarkedBlock::sweepHelper): Deleted. (JSC::MarkedBlock::stopAllocating): Deleted. (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted. (JSC::MarkedBlock::lastChanceToFinalize): Deleted. (JSC::MarkedBlock::resumeAllocating): Deleted. (JSC::MarkedBlock::didRetireBlock): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::VoidFunctor::returnValue): (JSC::MarkedBlock::CountFunctor::CountFunctor): (JSC::MarkedBlock::CountFunctor::count): (JSC::MarkedBlock::CountFunctor::returnValue): (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): (JSC::MarkedBlock::Handle::state): (JSC::MarkedBlock::needsDestruction): (JSC::MarkedBlock::handle): (JSC::MarkedBlock::Handle::block): (JSC::MarkedBlock::firstAtom): (JSC::MarkedBlock::atoms): (JSC::MarkedBlock::isAtomAligned): (JSC::MarkedBlock::Handle::cellAlign): (JSC::MarkedBlock::blockFor): (JSC::MarkedBlock::Handle::allocator): (JSC::MarkedBlock::Handle::heap): (JSC::MarkedBlock::Handle::vm): (JSC::MarkedBlock::vm): (JSC::MarkedBlock::Handle::weakSet): (JSC::MarkedBlock::weakSet): (JSC::MarkedBlock::Handle::shrink): (JSC::MarkedBlock::Handle::visitWeakSet): (JSC::MarkedBlock::Handle::reapWeakSet): (JSC::MarkedBlock::Handle::cellSize): (JSC::MarkedBlock::cellSize): (JSC::MarkedBlock::Handle::attributes): (JSC::MarkedBlock::attributes): (JSC::MarkedBlock::Handle::needsDestruction): (JSC::MarkedBlock::Handle::destruction): (JSC::MarkedBlock::Handle::cellKind): (JSC::MarkedBlock::Handle::markCount): (JSC::MarkedBlock::Handle::size): (JSC::MarkedBlock::atomNumber): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipForEdenCollection): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::Handle::assertFlipped): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::testAndSetMarked): (JSC::MarkedBlock::Handle::isNewlyAllocated): (JSC::MarkedBlock::Handle::setNewlyAllocated): (JSC::MarkedBlock::Handle::clearNewlyAllocated): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::isAtom): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::Handle::needsSweeping): (JSC::MarkedBlock::Handle::isAllocated): (JSC::MarkedBlock::Handle::isMarked): (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::hasAnyMarked): (JSC::MarkedBlock::noteMarked): (WTF::MarkedBlockHash::hash): (JSC::MarkedBlock::FreeList::FreeList): Deleted. (JSC::MarkedBlock::allocator): Deleted. (JSC::MarkedBlock::heap): Deleted. (JSC::MarkedBlock::shrink): Deleted. (JSC::MarkedBlock::visitWeakSet): Deleted. (JSC::MarkedBlock::reapWeakSet): Deleted. (JSC::MarkedBlock::willRemoveBlock): Deleted. (JSC::MarkedBlock::didConsumeFreeList): Deleted. (JSC::MarkedBlock::markCount): Deleted. (JSC::MarkedBlock::isEmpty): Deleted. (JSC::MarkedBlock::destruction): Deleted. (JSC::MarkedBlock::cellKind): Deleted. (JSC::MarkedBlock::size): Deleted. (JSC::MarkedBlock::capacity): Deleted. (JSC::MarkedBlock::setMarked): Deleted. (JSC::MarkedBlock::clearMarked): Deleted. (JSC::MarkedBlock::isNewlyAllocated): Deleted. (JSC::MarkedBlock::setNewlyAllocated): Deleted. (JSC::MarkedBlock::clearNewlyAllocated): Deleted. (JSC::MarkedBlock::isLive): Deleted. (JSC::MarkedBlock::isLiveCell): Deleted. (JSC::MarkedBlock::forEachCell): Deleted. (JSC::MarkedBlock::forEachLiveCell): Deleted. (JSC::MarkedBlock::forEachDeadCell): Deleted. (JSC::MarkedBlock::needsSweeping): Deleted. (JSC::MarkedBlock::isAllocated): Deleted. (JSC::MarkedBlock::isMarkedOrRetired): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::initializeSizeClassForStepSize): (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::lastChanceToFinalize): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::allocateLarge): (JSC::MarkedSpace::tryAllocateLarge): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::zombifySweep): (JSC::MarkedSpace::resetAllocators): (JSC::MarkedSpace::visitWeakSets): (JSC::MarkedSpace::reapWeakSets): (JSC::MarkedSpace::stopAllocating): (JSC::MarkedSpace::prepareForMarking): (JSC::MarkedSpace::resumeAllocating): (JSC::MarkedSpace::isPagedOut): (JSC::MarkedSpace::freeBlock): (JSC::MarkedSpace::freeOrShrinkBlock): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::VerifyMarked::operator()): (JSC::MarkedSpace::flip): (JSC::MarkedSpace::objectCount): (JSC::MarkedSpace::size): (JSC::MarkedSpace::capacity): (JSC::MarkedSpace::addActiveWeakSet): (JSC::MarkedSpace::didAddBlock): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::forEachAllocator): Deleted. (JSC::VerifyMarkedOrRetired::operator()): Deleted. (JSC::MarkedSpace::clearMarks): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::sizeClassToIndex): (JSC::MarkedSpace::indexToSizeClass): (JSC::MarkedSpace::version): (JSC::MarkedSpace::blocksWithNewObjects): (JSC::MarkedSpace::largeAllocations): (JSC::MarkedSpace::largeAllocationsNurseryOffset): (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection): (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin): (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd): (JSC::MarkedSpace::largeAllocationsForThisCollectionSize): (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): (JSC::MarkedSpace::allocatorFor): (JSC::MarkedSpace::destructorAllocatorFor): (JSC::MarkedSpace::auxiliaryAllocatorFor): (JSC::MarkedSpace::allocateWithoutDestructor): (JSC::MarkedSpace::allocateWithDestructor): (JSC::MarkedSpace::allocateAuxiliary): (JSC::MarkedSpace::tryAllocateAuxiliary): (JSC::MarkedSpace::forEachBlock): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::optimalSizeFor): (JSC::MarkedSpace::didAddBlock): Deleted. (JSC::MarkedSpace::didAllocateInBlock): Deleted. (JSC::MarkedSpace::objectCount): Deleted. (JSC::MarkedSpace::size): Deleted. (JSC::MarkedSpace::capacity): Deleted. * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::didStartMarking): (JSC::SlotVisitor::reset): (JSC::SlotVisitor::append): (JSC::SlotVisitor::appendJSCellOrAuxiliary): (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::appendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::noteLiveAuxiliaryCell): (JSC::SlotVisitor::visitChildren): * heap/SlotVisitor.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::WeakBlock): (JSC::WeakBlock::visit): (JSC::WeakBlock::reap): * heap/WeakBlock.h: (JSC::WeakBlock::disconnectContainer): (JSC::WeakBlock::disconnectMarkedBlock): Deleted. * heap/WeakSet.cpp: (JSC::WeakSet::~WeakSet): (JSC::WeakSet::sweep): (JSC::WeakSet::shrink): (JSC::WeakSet::addAllocator): * heap/WeakSet.h: (JSC::WeakSet::container): (JSC::WeakSet::setContainer): (JSC::WeakSet::WeakSet): (JSC::WeakSet::visit): (JSC::WeakSet::shrink): Deleted. * heap/WeakSetInlines.h: (JSC::WeakSet::allocate): * inspector/InjectedScriptManager.cpp: * inspector/JSGlobalObjectInspectorController.cpp: * inspector/JSJavaScriptCallFrame.cpp: * inspector/ScriptDebugServer.cpp: * inspector/agents/InspectorDebuggerAgent.cpp: * interpreter/CachedCall.h: (JSC::CachedCall::CachedCall): * interpreter/Interpreter.cpp: (JSC::loadVarargs): (JSC::StackFrame::sourceID): Deleted. (JSC::StackFrame::sourceURL): Deleted. (JSC::StackFrame::functionName): Deleted. (JSC::StackFrame::computeLineAndColumn): Deleted. (JSC::StackFrame::toString): Deleted. * interpreter/Interpreter.h: (JSC::StackFrame::isNative): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): (JSC::AssemblyHelpers::emitAllocate): (JSC::AssemblyHelpers::emitAllocateJSCell): (JSC::AssemblyHelpers::emitAllocateJSObject): (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): * jit/GCAwareJITStubRoutine.cpp: (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine): * jit/JIT.cpp: (JSC::JIT::compileCTINativeCall): (JSC::JIT::link): * jit/JIT.h: (JSC::JIT::compileCTINativeCall): Deleted. * jit/JITExceptions.cpp: (JSC::genericUnwind): * jit/JITExceptions.h: * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOpcodes32_64.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emitWriteBarrier): * jit/JITThunks.cpp: * jit/JITThunks.h: * jsc.cpp: (functionDescribeArray): (main): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntExceptions.cpp: * llint/LLIntThunks.cpp: * llint/LLIntThunks.h: * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter.cpp: * llint/LowLevelInterpreter32_64.asm: * llint/LowLevelInterpreter64.asm: * parser/ModuleAnalyzer.cpp: * parser/NodeConstructors.h: * parser/Nodes.h: * profiler/ProfilerBytecode.cpp: * profiler/ProfilerBytecode.h: * profiler/ProfilerBytecodeSequence.cpp: * runtime/ArrayConventions.h: (JSC::indexingHeaderForArrayStorage): (JSC::baseIndexingHeaderForArrayStorage): (JSC::indexingHeaderForArray): Deleted. (JSC::baseIndexingHeaderForArray): Deleted. * runtime/ArrayPrototype.cpp: (JSC::arrayProtoFuncSplice): (JSC::concatAppendOne): (JSC::arrayProtoPrivateFuncConcatMemcpy): * runtime/ArrayStorage.h: (JSC::ArrayStorage::vectorLength): (JSC::ArrayStorage::totalSizeFor): (JSC::ArrayStorage::totalSize): (JSC::ArrayStorage::availableVectorLength): (JSC::ArrayStorage::optimalVectorLength): (JSC::ArrayStorage::sizeFor): Deleted. * runtime/AuxiliaryBarrier.h: Added. (JSC::AuxiliaryBarrier::AuxiliaryBarrier): (JSC::AuxiliaryBarrier::clear): (JSC::AuxiliaryBarrier::get): (JSC::AuxiliaryBarrier::slot): (JSC::AuxiliaryBarrier::operator bool): (JSC::AuxiliaryBarrier::setWithoutBarrier): * runtime/AuxiliaryBarrierInlines.h: Added. (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier): (JSC::AuxiliaryBarrier<T>::set): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::availableContiguousVectorLength): (JSC::Butterfly::optimalContiguousVectorLength): (JSC::Butterfly::createUninitialized): (JSC::Butterfly::growArrayRight): * runtime/ClonedArguments.cpp: (JSC::ClonedArguments::createEmpty): * runtime/CommonSlowPathsExceptions.cpp: * runtime/CommonSlowPathsExceptions.h: * runtime/DataView.cpp: * runtime/DirectArguments.h: * runtime/ECMAScriptSpecInternalFunctions.cpp: * runtime/Error.cpp: * runtime/Error.h: * runtime/ErrorInstance.cpp: * runtime/ErrorInstance.h: * runtime/Exception.cpp: * runtime/Exception.h: * runtime/GeneratorFrame.cpp: * runtime/GeneratorPrototype.cpp: * runtime/InternalFunction.cpp: (JSC::InternalFunction::InternalFunction): * runtime/IntlCollator.cpp: * runtime/IntlCollatorConstructor.cpp: * runtime/IntlCollatorPrototype.cpp: * runtime/IntlDateTimeFormat.cpp: * runtime/IntlDateTimeFormatConstructor.cpp: * runtime/IntlDateTimeFormatPrototype.cpp: * runtime/IntlNumberFormat.cpp: * runtime/IntlNumberFormatConstructor.cpp: * runtime/IntlNumberFormatPrototype.cpp: * runtime/IntlObject.cpp: * runtime/IteratorPrototype.cpp: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitialized): (JSC::JSArray::setLengthWritable): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLengthWithArrayStorage): (JSC::JSArray::appendMemcpy): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithArrayStorage): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::createContiguousArrayButterfly): (JSC::createArrayButterfly): (JSC::JSArray::create): (JSC::JSArray::tryCreateUninitialized): Deleted. * runtime/JSArrayBufferView.h: * runtime/JSCInlines.h: * runtime/JSCJSValue.cpp: (JSC::JSValue::dumpInContextAssumingStructure): * runtime/JSCallee.cpp: (JSC::JSCallee::JSCallee): * runtime/JSCell.cpp: (JSC::JSCell::estimatedSize): * runtime/JSCell.h: (JSC::JSCell::cellStateOffset): Deleted. * runtime/JSCellInlines.h: (JSC::ExecState::vm): (JSC::JSCell::classInfo): (JSC::JSCell::callDestructor): (JSC::JSCell::vm): Deleted. * runtime/JSFunction.cpp: (JSC::JSFunction::create): (JSC::JSFunction::allocateAndInitializeRareData): (JSC::JSFunction::initializeRareData): (JSC::JSFunction::getOwnPropertySlot): (JSC::JSFunction::put): (JSC::JSFunction::deleteProperty): (JSC::JSFunction::defineOwnProperty): (JSC::JSFunction::setFunctionName): (JSC::JSFunction::reifyLength): (JSC::JSFunction::reifyName): (JSC::JSFunction::reifyLazyPropertyIfNeeded): (JSC::JSFunction::reifyBoundNameIfNeeded): * runtime/JSFunction.h: * runtime/JSFunctionInlines.h: (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint): (JSC::JSFunction::JSFunction): * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory): * runtime/JSInternalPromise.cpp: * runtime/JSInternalPromiseConstructor.cpp: * runtime/JSInternalPromiseDeferred.cpp: * runtime/JSInternalPromisePrototype.cpp: * runtime/JSJob.cpp: * runtime/JSMapIterator.cpp: * runtime/JSModuleNamespaceObject.cpp: * runtime/JSModuleRecord.cpp: * runtime/JSObject.cpp: (JSC::JSObject::visitButterfly): (JSC::JSObject::notifyPresenceOfIndexedAccessors): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createInitialUndecided): (JSC::JSObject::createInitialInt32): (JSC::JSObject::createInitialDouble): (JSC::JSObject::createInitialContiguous): (JSC::JSObject::createArrayStorage): (JSC::JSObject::createInitialArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::putByIndexBeyondVectorLength): (JSC::JSObject::putDirectIndexBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::increaseVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::growOutOfLineStorage): (JSC::JSObject::copyButterfly): Deleted. (JSC::JSObject::copyBackingStore): Deleted. * runtime/JSObject.h: (JSC::JSObject::globalObject): (JSC::JSObject::putDirectInternal): (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted. * runtime/JSObjectInlines.h: * runtime/JSPromise.cpp: * runtime/JSPromiseConstructor.cpp: * runtime/JSPromiseDeferred.cpp: * runtime/JSPromisePrototype.cpp: * runtime/JSPropertyNameIterator.cpp: * runtime/JSScope.cpp: (JSC::JSScope::resolve): * runtime/JSScope.h: (JSC::JSScope::globalObject): (JSC::JSScope::vm): Deleted. * runtime/JSSetIterator.cpp: * runtime/JSStringIterator.cpp: * runtime/JSTemplateRegistryKey.cpp: * runtime/JSTypedArrayViewConstructor.cpp: * runtime/JSTypedArrayViewPrototype.cpp: * runtime/JSWeakMap.cpp: * runtime/JSWeakSet.cpp: * runtime/MapConstructor.cpp: * runtime/MapIteratorPrototype.cpp: * runtime/MapPrototype.cpp: * runtime/NativeErrorConstructor.cpp: * runtime/NativeStdFunctionCell.cpp: * runtime/Operations.h: (JSC::scribbleFreeCells): (JSC::scribble): * runtime/Options.h: * runtime/PropertyTable.cpp: * runtime/ProxyConstructor.cpp: * runtime/ProxyObject.cpp: * runtime/ProxyRevoke.cpp: * runtime/RegExp.cpp: (JSC::RegExp::match): (JSC::RegExp::matchConcurrently): (JSC::RegExp::matchCompareWithInterpreter): * runtime/RegExp.h: * runtime/RegExpConstructor.h: * runtime/RegExpInlines.h: (JSC::RegExp::matchInline): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): (JSC::createRegExpMatchesArray): * runtime/RegExpPrototype.cpp: (JSC::genericSplit): * runtime/RuntimeType.cpp: * runtime/SamplingProfiler.cpp: (JSC::SamplingProfiler::processUnverifiedStackTraces): * runtime/SetConstructor.cpp: * runtime/SetIteratorPrototype.cpp: * runtime/SetPrototype.cpp: * runtime/StackFrame.cpp: Added. (JSC::StackFrame::sourceID): (JSC::StackFrame::sourceURL): (JSC::StackFrame::functionName): (JSC::StackFrame::computeLineAndColumn): (JSC::StackFrame::toString): * runtime/StackFrame.h: Added. (JSC::StackFrame::isNative): * runtime/StringConstructor.cpp: * runtime/StringIteratorPrototype.cpp: * runtime/StructureInlines.h: (JSC::Structure::propertyTable): * runtime/TemplateRegistry.cpp: * runtime/TestRunnerUtils.cpp: (JSC::finalizeStatsAtEndOfTesting): * runtime/TestRunnerUtils.h: * runtime/TypeProfilerLog.cpp: * runtime/TypeSet.cpp: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::ensureStackCapacityForCLoop): (JSC::VM::isSafeToRecurseSoftCLoop): * runtime/VM.h: * runtime/VMEntryScope.h: * runtime/VMInlines.h: (JSC::VM::ensureStackCapacityFor): (JSC::VM::isSafeToRecurseSoft): * runtime/WeakMapConstructor.cpp: * runtime/WeakMapData.cpp: * runtime/WeakMapPrototype.cpp: * runtime/WeakSetConstructor.cpp: * runtime/WeakSetPrototype.cpp: * testRegExp.cpp: (testOneRegExp): * tools/JSDollarVM.cpp: * tools/JSDollarVMPrototype.cpp: (JSC::JSDollarVMPrototype::isInObjectSpace): Source/WebCore: No new tests because no new WebCore behavior. Just rewiring #includes. * ForwardingHeaders/heap/HeapInlines.h: Added. * ForwardingHeaders/interpreter/Interpreter.h: Removed. * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added. * Modules/indexeddb/IDBCursorWithValue.cpp: * Modules/indexeddb/client/TransactionOperation.cpp: * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp: * Modules/indexeddb/server/UniqueIDBDatabase.cpp: * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp: * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp: * bindings/js/JSClientRectCustom.cpp: * bindings/js/JSDOMBinding.cpp: * bindings/js/JSDOMBinding.h: * bindings/js/JSDeviceMotionEventCustom.cpp: * bindings/js/JSDeviceOrientationEventCustom.cpp: * bindings/js/JSErrorEventCustom.cpp: * bindings/js/JSIDBCursorWithValueCustom.cpp: * bindings/js/JSIDBIndexCustom.cpp: * bindings/js/JSPopStateEventCustom.cpp: * bindings/js/JSWebGL2RenderingContextCustom.cpp: * bindings/js/JSWorkerGlobalScopeCustom.cpp: * bindings/js/WorkerScriptController.cpp: * contentextensions/ContentExtensionParser.cpp: * dom/ErrorEvent.cpp: * html/HTMLCanvasElement.cpp: * html/MediaDocument.cpp: * inspector/CommandLineAPIModule.cpp: * loader/EmptyClients.cpp: * page/CaptionUserPreferences.cpp: * page/Frame.cpp: * page/PageGroup.cpp: * page/UserContentController.cpp: * platform/mock/mediasource/MockBox.cpp: * testing/GCObservation.cpp: Source/WebKit2: Just rewiring some #includes. * UIProcess/ViewGestureController.cpp: * UIProcess/WebPageProxy.cpp: * UIProcess/WebProcessPool.cpp: * UIProcess/WebProcessProxy.cpp: * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp: * WebProcess/Plugins/Netscape/JSNPObject.cpp: Source/WTF: I needed tryFastAlignedMalloc() so I added it. * wtf/FastMalloc.cpp: (WTF::tryFastAlignedMalloc): * wtf/FastMalloc.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): (WTF::ParkingLot::forEach): Deleted. * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::forEach): * wtf/ScopedLambda.h: (WTF::scopedLambdaRef): * wtf/SentinelLinkedList.h: (WTF::SentinelLinkedList::forEach): (WTF::RawNode>::takeFrom): * wtf/SimpleStats.h: (WTF::SimpleStats::operator bool): (WTF::SimpleStats::operator!): Deleted. Tools: * DumpRenderTree/TestRunner.cpp: * DumpRenderTree/mac/DumpRenderTree.mm: (DumpRenderTreeMain): * Scripts/run-jsc-stress-tests: * TestWebKitAPI/Tests/WTF/Vector.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/179778@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
scopedLambdaRef<bool()>(validation),
scopedLambdaRef<void()>(beforeSleep),
timeout);
}
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
WTF should have a compact Condition object to use with Lock https://bugs.webkit.org/show_bug.cgi?id=147986 Reviewed by Geoffrey Garen. Source/WTF: Adds a condition variable implementation based on ParkingLot, called simply WTF::Condition. It can be used with WTF::Lock or actually any lock implementation. It should even work with WTF::SpinLock, WTF::Mutex, or std::mutex. Best of all, Condition only requires one byte. ParkingLot almost contained all of the functionality needed to implemenet wait/notify. We could have implemented Condition using a 32-bit (or even 64-bit) version that protects against a notify that happens just before we park. But, this changes the ParkingLot API to give us the ability to run some code between when ParkingLot enqueues the current thread and when it actually sleeps. This callback is called with no locks held, so it can call unlock() on any kind of lock, so long as that lock's unlock() method doesn't recurse into ParkingLot::parkConditionally(). That seems unlikely; unlock() is more likely to call ParkingLot::unparkOne() or unparkAll(). WTF::Lock will never call parkConditionally() inside unlock(), so WTF::Lock is definitely appropriate for use with Condition. Condition supports most of the API that std::condition_variable supports. It does some things to try to reduce footgun potential. The preferred timeout form is waitUntil() which takes an absolute time from the steady_clock. The only relative timeout form also takes a predicate callback, so it's impossible to write the subtly incorrect "while (...) wait_for(...)" idiom. This patch doesn't actually introduce any uses of WTF::Condition other than the unit tests. I'll start switching code over to using WTF::Condition in another patch. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Condition.h: Added. (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): Make this useful assertion be a release assertion. It catches cases where you unlock the lock even though you don't hold it. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): Add the beforeSleep() callback. (WTF::ParkingLot::unparkOne): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): Tools: Add a test for WTF::Condition. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: Added. (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): Change the name of the thread. Canonical link: https://commits.webkit.org/166094@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188400 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 20:42:11 +00:00
// Simple version of parkConditionally() that covers the most common case: you want to park
// indefinitely so long as the value at the given address hasn't changed.
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
template<typename T, typename U>
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
static ParkResult compareAndPark(const Atomic<T>* address, U expected)
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
{
return parkConditionally(
address,
[address, expected] () -> bool {
U value = address->load();
return value == expected;
WTF should have a compact Condition object to use with Lock https://bugs.webkit.org/show_bug.cgi?id=147986 Reviewed by Geoffrey Garen. Source/WTF: Adds a condition variable implementation based on ParkingLot, called simply WTF::Condition. It can be used with WTF::Lock or actually any lock implementation. It should even work with WTF::SpinLock, WTF::Mutex, or std::mutex. Best of all, Condition only requires one byte. ParkingLot almost contained all of the functionality needed to implemenet wait/notify. We could have implemented Condition using a 32-bit (or even 64-bit) version that protects against a notify that happens just before we park. But, this changes the ParkingLot API to give us the ability to run some code between when ParkingLot enqueues the current thread and when it actually sleeps. This callback is called with no locks held, so it can call unlock() on any kind of lock, so long as that lock's unlock() method doesn't recurse into ParkingLot::parkConditionally(). That seems unlikely; unlock() is more likely to call ParkingLot::unparkOne() or unparkAll(). WTF::Lock will never call parkConditionally() inside unlock(), so WTF::Lock is definitely appropriate for use with Condition. Condition supports most of the API that std::condition_variable supports. It does some things to try to reduce footgun potential. The preferred timeout form is waitUntil() which takes an absolute time from the steady_clock. The only relative timeout form also takes a predicate callback, so it's impossible to write the subtly incorrect "while (...) wait_for(...)" idiom. This patch doesn't actually introduce any uses of WTF::Condition other than the unit tests. I'll start switching code over to using WTF::Condition in another patch. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Condition.h: Added. (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): Make this useful assertion be a release assertion. It catches cases where you unlock the lock even though you don't hold it. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): Add the beforeSleep() callback. (WTF::ParkingLot::unparkOne): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): Tools: Add a test for WTF::Condition. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: Added. (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): Change the name of the thread. Canonical link: https://commits.webkit.org/166094@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188400 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 20:42:11 +00:00
},
[] () { },
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows https://bugs.webkit.org/show_bug.cgi?id=152045 Reviewed by Andy Estes. Source/JavaScriptCore: Probably the nicest example of why this patch is a good idea is the change in AtomicsObject.cpp. * jit/ICStats.cpp: (JSC::ICStats::ICStats): * runtime/AtomicsObject.cpp: (JSC::atomicsFuncWait): Source/WebCore: No new layout tests because no new behavior. The new WTF time classes have some unit tests in TestWebKitAPI. * fileapi/ThreadableBlobRegistry.cpp: (WebCore::ThreadableBlobRegistry::blobSize): * platform/MainThreadSharedTimer.h: * platform/SharedTimer.h: * platform/ThreadTimers.cpp: (WebCore::ThreadTimers::updateSharedTimer): * platform/cf/MainThreadSharedTimerCF.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/efl/MainThreadSharedTimerEfl.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/glib/MainThreadSharedTimerGLib.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/win/MainThreadSharedTimerWin.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * workers/WorkerRunLoop.cpp: (WebCore::WorkerRunLoop::runInMode): Source/WebKit2: * Platform/IPC/Connection.cpp: (IPC::Connection::SyncMessageState::wait): (IPC::Connection::sendMessage): (IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting): (IPC::Connection::waitForMessage): (IPC::Connection::sendSyncMessage): (IPC::Connection::waitForSyncReply): * Platform/IPC/Connection.h: (IPC::Connection::sendSync): (IPC::Connection::waitForAndDispatchImmediately): * Platform/IPC/MessageSender.h: (IPC::MessageSender::sendSync): * UIProcess/ChildProcessProxy.h: (WebKit::ChildProcessProxy::sendSync): * UIProcess/Network/NetworkProcessProxy.cpp: (WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently): * UIProcess/Storage/StorageManager.cpp: (WebKit::StorageManager::applicationWillTerminate): * UIProcess/WebProcessProxy.cpp: (WebKit::WebProcessProxy::sendProcessWillSuspendImminently): * UIProcess/WebResourceLoadStatisticsStore.cpp: (WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate): * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h: * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm: (-[WKOneShotDisplayLinkHandler displayLinkFired:]): (WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree): (WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay): (WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm: (WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/WKImmediateActionController.mm: (-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]): * UIProcess/mac/WebPageProxyMac.mm: (WebKit::WebPageProxy::stringSelectionForPasteboard): (WebKit::WebPageProxy::dataSelectionForPasteboard): (WebKit::WebPageProxy::readSelectionFromPasteboard): (WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent): (WebKit::WebPageProxy::acceptsFirstMouse): * WebProcess/WebCoreSupport/WebChromeClient.cpp: (WebKit::WebChromeClient::runBeforeUnloadConfirmPanel): (WebKit::WebChromeClient::runJavaScriptAlert): (WebKit::WebChromeClient::runJavaScriptConfirm): (WebKit::WebChromeClient::runJavaScriptPrompt): (WebKit::WebChromeClient::print): (WebKit::WebChromeClient::exceededDatabaseQuota): (WebKit::WebChromeClient::reachedApplicationCacheOriginQuota): * WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp: (WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse): * WebProcess/WebPage/WebPage.cpp: (WebKit::WebPage::postSynchronousMessageForTesting): Source/WTF: We used to use 'double' for all time measurements. Sometimes it was milliseconds, sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time since some epoch. When we spoke of time since epoch, we either used a monotonic clock or a wall clock. The type - always 'double' - never told us what kind of time we had, even though there were roughly six of them (sec interval, ms interval, sec since epoch on wall, ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic). At some point, we thought that it would be a good idea to replace these doubles with std::chrono. But since replacing some things with std::chrono, we found it to be terribly inconvenient: - Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally duration_cast would not even be a thing. - No overflow protection. std::chrono uses integers by default and using anything else is clumsy. But the integer math is done without regard for the rough edges of integer math, so any cast between std::chrono types risks overflow. Any comparison risks overflow because it may do conversions silently. We have even found bugs where some C++ implementations had more overflows than others, which ends up being a special kind of hell. In many cases, the overflow also has nasal demons. It's an error to represent time using integers. It would have been excusable back when floating point math was not guaranteed to be supported on all platforms, but that would have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit for floating point: - Floating point preserves precision under multiplication in all but extreme cases, so using floating point for time means that unit conversions are almost completely lossless. This means that we don't have to think very hard about what units to use. In this patch, we use seconds almost everywhere. We only convert at boundaries, like an API boundary that wants something other than seconds. - Floating point makes it easy to reason about infinity, which is something that time code wants to do a lot. Example: when would you like to timeout? Infinity please! This is the most elegant way of having an API support both a timeout variant and a no-timeout variant. - Floating point does well-understood things when math goes wrong, and these things are pretty well optimized to match what a mathematician would do when computing with real numbers represented using scientific notation with a finite number of significant digits. This means that time math under floating point looks like normal math. On the other hand, std::chrono time math looks like garbage because you have to always check for multiple possible UB corners whenever you touch large integers. Integers that represent time are very likely to be large and you don't have to do much to overflow them. At this time, based on the number of bugs we have already seen due to chrono overflows, I am not certain that we even understand what are all of the corner cases that we should even check for. This patch introduces a new set of timekeeping classes that are all based on double, and all internally use seconds. These classes support algebraic typing. The classes are: - Seconds: this is for measuring a duration. - WallTime: time since epoch according to a wall clock (aka real time clock). - MonotonicTime: time since epoch according to a monotonic clock. - ClockType: enum that says either Wall or Monotonic. - TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a wall time or a monotonic time. All of these classes behave like C++ values and are cheap to copy around since they are very nearly POD. This supports comprehensive conversions between the various time types. Most of this is by way of algebra. Here are just some of the rules we recognize: WallTime = WallTime + Seconds Seconds = WallTime - WallTime MonotonicTime = MonotonicTime + Seconds etc... We support negative, infinite, and NaN times because math. We support conversions between MonotonicTime and WallTime, like: WallTime wt = mt.approximateWallTime() This is called this "approximate" because the only way to do it is to get the current time on both clocks and convert relative to that. Many of our APIs would be happy using whatever notion of time the user wanted to use. For those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This means that if you use a WallTime with Condition::waitUntil, then Condition's internal logic for when it should wake up makes its decision based on the current WallTime - but if you use MonotonicTime then waitUntil will make its decision based on current MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono did not have the concept of a dynamic clock type. This patch does not include conversions between std::chrono and these new time classes, because past experience shows that we're quite bad at getting conversions between std::chrono and anything else right. Also, I didn't need such conversion code because this patch only converts code that transitively touches ParkingLot and Condition. It was easy to get all of that code onto the new time classes. * WTF.xcodeproj/project.pbxproj: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/ClockType.cpp: Added. (WTF::printInternal): * wtf/ClockType.h: Added. * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::waitFor): (WTF::ConditionBase::wait): (WTF::ConditionBase::waitUntilWallClockSeconds): Deleted. (WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted. (WTF::ConditionBase::waitForSeconds): Deleted. (WTF::ConditionBase::waitForSecondsImpl): Deleted. (WTF::ConditionBase::waitForImpl): Deleted. (WTF::ConditionBase::absoluteFromRelative): Deleted. * wtf/CrossThreadQueue.h: (WTF::CrossThreadQueue<DataType>::waitForMessage): * wtf/CurrentTime.cpp: (WTF::sleep): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): Deleted. * wtf/MonotonicTime.cpp: Added. (WTF::MonotonicTime::now): (WTF::MonotonicTime::approximateWallTime): (WTF::MonotonicTime::dump): (WTF::MonotonicTime::sleep): * wtf/MonotonicTime.h: Added. (WTF::MonotonicTime::MonotonicTime): (WTF::MonotonicTime::fromRawDouble): (WTF::MonotonicTime::infinity): (WTF::MonotonicTime::secondsSinceEpoch): (WTF::MonotonicTime::approximateMonotonicTime): (WTF::MonotonicTime::operator bool): (WTF::MonotonicTime::operator+): (WTF::MonotonicTime::operator-): (WTF::MonotonicTime::operator+=): (WTF::MonotonicTime::operator-=): (WTF::MonotonicTime::operator==): (WTF::MonotonicTime::operator!=): (WTF::MonotonicTime::operator<): (WTF::MonotonicTime::operator>): (WTF::MonotonicTime::operator<=): (WTF::MonotonicTime::operator>=): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkCount): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): * wtf/Seconds.cpp: Added. (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::dump): (WTF::Seconds::sleep): * wtf/Seconds.h: Added. (WTF::Seconds::Seconds): (WTF::Seconds::value): (WTF::Seconds::seconds): (WTF::Seconds::milliseconds): (WTF::Seconds::microseconds): (WTF::Seconds::nanoseconds): (WTF::Seconds::fromMilliseconds): (WTF::Seconds::fromMicroseconds): (WTF::Seconds::fromNanoseconds): (WTF::Seconds::infinity): (WTF::Seconds::operator bool): (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::operator*): (WTF::Seconds::operator/): (WTF::Seconds::operator+=): (WTF::Seconds::operator-=): (WTF::Seconds::operator*=): (WTF::Seconds::operator/=): (WTF::Seconds::operator==): (WTF::Seconds::operator!=): (WTF::Seconds::operator<): (WTF::Seconds::operator>): (WTF::Seconds::operator<=): (WTF::Seconds::operator>=): * wtf/TimeWithDynamicClockType.cpp: Added. (WTF::TimeWithDynamicClockType::now): (WTF::TimeWithDynamicClockType::nowWithSameClock): (WTF::TimeWithDynamicClockType::wallTime): (WTF::TimeWithDynamicClockType::monotonicTime): (WTF::TimeWithDynamicClockType::approximateWallTime): (WTF::TimeWithDynamicClockType::approximateMonotonicTime): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator<): (WTF::TimeWithDynamicClockType::operator>): (WTF::TimeWithDynamicClockType::operator<=): (WTF::TimeWithDynamicClockType::operator>=): (WTF::TimeWithDynamicClockType::dump): (WTF::TimeWithDynamicClockType::sleep): * wtf/TimeWithDynamicClockType.h: Added. (WTF::TimeWithDynamicClockType::TimeWithDynamicClockType): (WTF::TimeWithDynamicClockType::fromRawDouble): (WTF::TimeWithDynamicClockType::secondsSinceEpoch): (WTF::TimeWithDynamicClockType::clockType): (WTF::TimeWithDynamicClockType::withSameClockAndRawDouble): (WTF::TimeWithDynamicClockType::operator bool): (WTF::TimeWithDynamicClockType::operator+): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator+=): (WTF::TimeWithDynamicClockType::operator-=): (WTF::TimeWithDynamicClockType::operator==): (WTF::TimeWithDynamicClockType::operator!=): * wtf/WallTime.cpp: Added. (WTF::WallTime::now): (WTF::WallTime::approximateMonotonicTime): (WTF::WallTime::dump): (WTF::WallTime::sleep): * wtf/WallTime.h: Added. (WTF::WallTime::WallTime): (WTF::WallTime::fromRawDouble): (WTF::WallTime::infinity): (WTF::WallTime::secondsSinceEpoch): (WTF::WallTime::approximateWallTime): (WTF::WallTime::operator bool): (WTF::WallTime::operator+): (WTF::WallTime::operator-): (WTF::WallTime::operator+=): (WTF::WallTime::operator-=): (WTF::WallTime::operator==): (WTF::WallTime::operator!=): (WTF::WallTime::operator<): (WTF::WallTime::operator>): (WTF::WallTime::operator<=): (WTF::WallTime::operator>=): * wtf/threads/BinarySemaphore.cpp: (WTF::BinarySemaphore::wait): * wtf/threads/BinarySemaphore.h: Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp: (TestWebKitAPI::ToUpperConverter::stopProducing): (TestWebKitAPI::ToUpperConverter::stopConsuming): * TestWebKitAPI/Tests/WTF/Time.cpp: Added. (WTF::operator<<): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/182152@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
Time::infinity());
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
}
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// Unparking status given to you anytime you unparkOne().
struct UnparkResult {
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// True if some thread was unparked.
bool didUnparkThread { false };
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// True if there may be more threads on this address. This may be conservatively true.
bool mayHaveMoreThreads { false };
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// This bit is randomly set to true indicating that it may be profitable to unlock the lock
// using a fair unlocking protocol. This is most useful when used in conjunction with
// unparkOne(address, callback).
bool timeToBeFair { false };
};
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// Unparks one thread from the queue associated with the given address, which cannot be null.
// Returns true if there may still be other threads on that queue, or false if there definitely
// are no more threads on the queue.
WTF_EXPORT_PRIVATE static UnparkResult unparkOne(const void* address);
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// This is an expert-mode version of unparkOne() that allows for really good thundering herd
// avoidance and eventual stochastic fairness in adaptive mutexes.
//
WTF::Lock should not suffer from the thundering herd https://bugs.webkit.org/show_bug.cgi?id=147947 Reviewed by Geoffrey Garen. Source/WTF: This changes Lock::unlockSlow() to use unparkOne() instead of unparkAll(). The problem with doing this is that it's not obvious after calling unparkOne() if there are any other threads that are still parked on the lock's queue. If we assume that there are and leave the hasParkedBit set, then future calls to unlock() will take the slow path. We don't want that if there aren't actually any threads parked. On the other hand, if we assume that there aren't any threads parked and clear the hasParkedBit, then if there actually were some threads parked, then they may never be awoken since future calls to unlock() won't take slow path and so won't call unparkOne(). In other words, we need a way to be very precise about when we clear the hasParkedBit and we need to do it in a race-free way: it can't be the case that we clear the bit just as some thread gets parked on the queue. A similar problem arises in futexes, and one of the solutions is to have a thread that acquires a lock after parking sets the hasParkedBit. This is what Rusty Russel's usersem does. It's a subtle algorithm. Also, it means that if a thread barges in before the unparked thread runs, then that barging thread will not know that there are threads parked. This could increase the severity of barging. Since ParkingLot is a user-level API, we don't have to worry about the kernel-user security issues and so we can expose callbacks while ParkingLot is holding its internal locks. This change does exactly that for unparkOne(). The new variant of unparkOne() will call a user function while the queue from which we are unparking is locked. The callback is told basic stats about the queue: did we unpark a thread this time, and could there be more threads to unpark in the future. The callback runs while it's impossible for the queue state to change, since the ParkingLot's internal locks for the queue is held. This means that Lock::unlockSlow() can either clear, or leave, the hasParkedBit while releasing the lock inside the callback from unparkOne(). This takes care of the thundering herd problem while also reducing the greed that arises from barging threads. This required some careful reworking of the ParkingLot algorithm. The first thing I noticed was that the ThreadData::shouldPark flag was useless, since it's set exactly when ThreadData::address is non-null. Then I had to make sure that dequeue() could lazily create both hashtables and buckets, since the "callback is called while queue is locked" invariant requires that we didn't exit early due to the hashtable or bucket not being present. Note that all of this is done in such a way that the old unparkOne() and unparkAll() don't have to create any buckets, though they now may create the hashtable. We don't care as much about the hashtable being created by unpark since it's just such an unlikely scenario and it would only happen once. This change reduces the kernel CPU usage of WTF::Lock for the long critical section test by about 8x and makes it always perform as well as WTF::WordLock and WTF::Mutex for that benchmark. * benchmarks/LockSpeedTest.cpp: * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): * wtf/Lock.h: (WTF::LockBase::isLocked): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: * wtf/WordLock.h: (WTF::WordLock::isLocked): (WTF::WordLock::isFullyReset): Tools: Add testing that checks that locks return to a pristine state after contention is over. * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::LockInspector::isFullyReset): (TestWebKitAPI::runLockTest): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/166072@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188374 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 03:51:25 +00:00
// Unparks one thread from the queue associated with the given address, and calls the given
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
// callback while the address is locked. Reports to the callback whether any thread got
// unparked, whether there may be any other threads still on the queue, and whether this may be
// a good time to do fair unlocking. The callback returns an intptr_t token, which is returned
// to the unparked thread via ParkResult::token.
//
// WTF::Lock and WTF::Condition both use this form of unparkOne() because it allows them to use
// the ParkingLot's internal queue lock to serialize some decision-making. For example, if
// UnparkResult::mayHaveMoreThreads is false inside the callback, then we know that at that
// moment nobody can add any threads to the queue because the queue lock is still held. Also,
// WTF::Lock uses the timeToBeFair and token mechanism to implement eventual fairness.
ScopedLambda should have a lifetime story that makes sense to the compiler https://bugs.webkit.org/show_bug.cgi?id=158118 Reviewed by Mark Lam. Source/WTF: Prior to this change, there were two lifetime bugs in ScopedLambda: - scopedLambda(Functor&&) would bind Functor to const lambda&, so the resulting ScopedLambdaFunctor would hold a reference to the original lambda. This would have surprising behavior; for example it meant that this code was wrong: auto l = scopedLambda<things>([&] ...); The solution is to have explicit copy/move versions of scopedLambda() rather than rely on perfect forwarding. - ScopedLambdaFunctor did not override its copy or move operations, so if the compiler did not RVO scopedLambda(), it would return a ScopedLambdaFunctor whose m_arg points to a dead temporary ScopedLambdaFunctor instance. The solution is to have explicit copy/move constructors and operators, which preserve the invariant that ScopedLambda::m_arg points to this. One nice side-effect of all of these constructors and operators being explicit is that we can rely on WTFMove's excellent assertions, which helped catch the first issue. This reverts ParkingLot to use ScopedLambda again. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): * wtf/ScopedLambda.h: (WTF::scopedLambda): Tools: Added a test case. This test crashes before the fix and now it passes. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/ScopedLambda.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/176236@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@201433 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-05-26 21:58:42 +00:00
template<typename Callback>
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression https://bugs.webkit.org/show_bug.cgi?id=160125 Reviewed by Geoffrey Garen and Keith Miller. JSTests: Most of the things I did properly covered by existing tests, but I found some simple cases of unshifting that had sketchy coverage. * stress/array-storage-array-unshift.js: Added. * stress/contiguous-array-unshift.js: Added. * stress/double-array-unshift.js: Added. * stress/int32-array-unshift.js: Added. Source/bmalloc: I needed to tryMemalign, so I added such a thing. * bmalloc/Allocator.cpp: (bmalloc::Allocator::allocate): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): * bmalloc/Allocator.h: * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): * bmalloc/bmalloc.h: (bmalloc::api::tryMemalign): Source/JavaScriptCore: In order to make the GC concurrent (bug 149432), we would either need to enable concurrent copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead from the barriers alone. Considering that MarkedSpace does a decent job of avoiding fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want to get rid of copied space. This change moves copied space's biggest client over to marked space. Moving butterflies to marked space means having them use the new Auxiliary HeapCell allocation path. This is a fairly mechanical change, but it caused performance regressions everywhere, so this change also fixes MarkedSpace's performance issues. At a high level the mechanical changes are: - We use AuxiliaryBarrier instead of CopyBarrier. - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth. - The JITs have to emit inlined marked space allocations instead of inline copy space allocations. - Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't. That's about 1/3 of this change. But this led to performance problems, which I fixed with optimizations that amounted to a major MarkedSpace rewrite: - MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely. - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well. - Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being: 16, 32, 48, 64, 80, 112, 160, 224, 320 The next size class would have been 432, but that's above the 400 byte cutoff. All of this is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what size classes you end up with by doing --dumpSizeClasses=true. - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: https://bugs.webkit.org/show_bug.cgi?id=161581. - Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree. - FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky. - It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch! - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes. All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1% regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a best-case scenario for bump allocation. I think that we should fix bmalloc to make up the difference, but take the hit for now because it's a crazy corner case. By comparison, the alternative approach of using a copy barrier would have cost us 1-2%. That's the real apples-to-apples comparison if your premise is that we should have a concurrent GC. After we finish removing copied space, we will be barrier-ready for concurrent GC: we already have a marking barrier and we simply won't need a copying barrier. This change gets us there for the purposes of our benchmarks, since the remaining clients of copied space are not very important. On the other hand, if we keep copying, then getting barrier-ready would mean adding back the copy barrier, which costs more perf. We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving typed arrays and a few other weird things over to Aux MarkedSpace. This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell, and CellContainer meant that I had to include those files from everywhere. Fortunately, just including JSCInlines.h (instead of manually including the files that includes) is usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something that we were already basically doing. In places where JSCInlines.h would be too much, I just included HeapInlines.h. This got weird, because we previously included HeapInlines.h from JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that meant having to manually include HeapInlines.h from the places that previously got it implicitly via JSObject.h. But that led to more problems for some reason: I started getting build errors because non-JSC files were having trouble including Opcode.h. That's just silly, since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and made it impossible to include it from outside JSC. This was a lot of work, but it was necessary to get the patch to build on all ports. It's also a net win. There were many places in WebCore that were transitively including a *ton* of JSC headers just because of the JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons. * API/JSManagedValue.mm: (-[JSManagedValue initWithValue:]): * API/JSTypedArray.cpp: * API/ObjCCallbackFunction.mm: * API/tests/testapi.mm: (testObjectiveCAPI): (testWeakValue): Deleted. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * Scripts/builtins/builtins_generate_combined_implementation.py: (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py: (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_separate_implementation.py: (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes): * assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::JumpList::link): (JSC::AbstractMacroAssembler::JumpList::linkTo): * assembler/MacroAssembler.h: * assembler/MacroAssemblerARM64.h: (JSC::MacroAssemblerARM64::add32): * assembler/MacroAssemblerCodeRef.cpp: Added. (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): (JSC::MacroAssemblerCodePtr::dumpWithName): (JSC::MacroAssemblerCodePtr::dump): (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): (JSC::MacroAssemblerCodeRef::dump): * assembler/MacroAssemblerCodeRef.h: (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted. (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted. (JSC::MacroAssemblerCodePtr::dump): Deleted. (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted. (JSC::MacroAssemblerCodeRef::dump): Deleted. * b3/B3BasicBlock.cpp: (JSC::B3::BasicBlock::appendBoolConstant): * b3/B3BasicBlock.h: * b3/B3DuplicateTails.cpp: * b3/B3StackmapGenerationParams.h: * b3/testb3.cpp: (JSC::B3::testPatchpointTerminalReturnValue): (JSC::B3::run): * bindings/ScriptValue.cpp: * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeLivenessAnalysis.cpp: * bytecode/BytecodeUseDef.h: * bytecode/CallLinkInfo.cpp: (JSC::CallLinkInfo::callTypeFor): * bytecode/CallLinkInfo.h: (JSC::CallLinkInfo::callTypeFor): Deleted. * bytecode/CallLinkStatus.cpp: * bytecode/CodeBlock.cpp: (JSC::CodeBlock::finishCreation): (JSC::CodeBlock::clearLLIntGetByIdCache): (JSC::CodeBlock::predictedMachineCodeSize): * bytecode/CodeBlock.h: (JSC::CodeBlock::jitCodeMap): Deleted. (JSC::clearLLIntGetByIdCache): Deleted. * bytecode/ExecutionCounter.h: * bytecode/Instruction.h: * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp: (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal): * bytecode/ObjectAllocationProfile.h: (JSC::ObjectAllocationProfile::isNull): (JSC::ObjectAllocationProfile::initialize): * bytecode/Opcode.h: (JSC::padOpcodeName): * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): (JSC::PolymorphicAccess::regenerate): * bytecode/PolymorphicAccess.h: * bytecode/PreciseJumpTargets.cpp: * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecode/UnlinkedCodeBlock.cpp: (JSC::UnlinkedCodeBlock::vm): Deleted. * bytecode/UnlinkedCodeBlock.h: * bytecode/UnlinkedInstructionStream.cpp: * bytecode/UnlinkedInstructionStream.h: * dfg/DFGOperations.cpp: * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT.h: (JSC::DFG::SpeculativeJIT::emitAllocateJSCell): (JSC::DFG::SpeculativeJIT::emitAllocateJSObject): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGStrengthReductionPhase.cpp: (JSC::DFG::StrengthReductionPhase::handleNode): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCompile.cpp: * ftl/FTLJITFinalizer.cpp: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments): (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest): (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell): (JSC::FTL::DFG::LowerDFGToB3::allocateCell): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted. * ftl/FTLOutput.cpp: (JSC::FTL::Output::constBool): (JSC::FTL::Output::add): (JSC::FTL::Output::shl): (JSC::FTL::Output::aShr): (JSC::FTL::Output::lShr): (JSC::FTL::Output::zeroExt): (JSC::FTL::Output::equal): (JSC::FTL::Output::notEqual): (JSC::FTL::Output::above): (JSC::FTL::Output::aboveOrEqual): (JSC::FTL::Output::below): (JSC::FTL::Output::belowOrEqual): (JSC::FTL::Output::greaterThan): (JSC::FTL::Output::greaterThanOrEqual): (JSC::FTL::Output::lessThan): (JSC::FTL::Output::lessThanOrEqual): (JSC::FTL::Output::select): (JSC::FTL::Output::appendSuccessor): (JSC::FTL::Output::addIncomingToPhi): * ftl/FTLOutput.h: * ftl/FTLValueFromBlock.h: (JSC::FTL::ValueFromBlock::operator bool): (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted. * ftl/FTLWeightedTarget.h: (JSC::FTL::WeightedTarget::frequentedBlock): * heap/CellContainer.h: Added. (JSC::CellContainer::CellContainer): (JSC::CellContainer::operator bool): (JSC::CellContainer::isMarkedBlock): (JSC::CellContainer::isLargeAllocation): (JSC::CellContainer::markedBlock): (JSC::CellContainer::largeAllocation): * heap/CellContainerInlines.h: Added. (JSC::CellContainer::isMarked): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::noteMarked): (JSC::CellContainer::cellSize): (JSC::CellContainer::weakSet): (JSC::CellContainer::flipIfNecessary): * heap/ConservativeRoots.cpp: (JSC::ConservativeRoots::ConservativeRoots): (JSC::ConservativeRoots::~ConservativeRoots): (JSC::ConservativeRoots::grow): (JSC::ConservativeRoots::genericAddPointer): (JSC::ConservativeRoots::genericAddSpan): * heap/ConservativeRoots.h: (JSC::ConservativeRoots::roots): * heap/CopyToken.h: * heap/FreeList.cpp: Added. (JSC::FreeList::dump): * heap/FreeList.h: Added. (JSC::FreeList::FreeList): (JSC::FreeList::list): (JSC::FreeList::bump): (JSC::FreeList::operator==): (JSC::FreeList::operator!=): (JSC::FreeList::operator bool): (JSC::FreeList::allocationWillFail): (JSC::FreeList::allocationWillSucceed): * heap/GCTypeMap.h: Added. (JSC::GCTypeMap::operator[]): * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalizeUnconditionalFinalizers): (JSC::Heap::markRoots): (JSC::Heap::copyBackingStores): (JSC::Heap::gatherStackRoots): (JSC::Heap::gatherJSStackRoots): (JSC::Heap::gatherScratchBufferRoots): (JSC::Heap::clearLivenessData): (JSC::Heap::visitSmallStrings): (JSC::Heap::visitConservativeRoots): (JSC::Heap::removeDeadCompilerWorklistEntries): (JSC::Heap::gatherExtraHeapSnapshotData): (JSC::Heap::removeDeadHeapSnapshotNodes): (JSC::Heap::visitProtectedObjects): (JSC::Heap::visitArgumentBuffers): (JSC::Heap::visitException): (JSC::Heap::visitStrongHandles): (JSC::Heap::visitHandleStack): (JSC::Heap::visitSamplingProfiler): (JSC::Heap::traceCodeBlocksAndJITStubRoutines): (JSC::Heap::converge): (JSC::Heap::visitWeakHandles): (JSC::Heap::updateObjectCounts): (JSC::Heap::clearUnmarkedExecutables): (JSC::Heap::deleteUnmarkedCompiledCode): (JSC::Heap::collectAllGarbage): (JSC::Heap::collect): (JSC::Heap::collectWithoutAnySweep): (JSC::Heap::collectImpl): (JSC::Heap::suspendCompilerThreads): (JSC::Heap::willStartCollection): (JSC::Heap::flushOldStructureIDTables): (JSC::Heap::flushWriteBarrierBuffer): (JSC::Heap::stopAllocation): (JSC::Heap::prepareForMarking): (JSC::Heap::reapWeakHandles): (JSC::Heap::pruneStaleEntriesFromWeakGCMaps): (JSC::Heap::sweepArrayBuffers): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): (JSC::MarkedBlockSnapshotFunctor::operator()): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::deleteSourceProviderCaches): (JSC::Heap::notifyIncrementalSweeper): (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks): (JSC::Heap::resetAllocators): (JSC::Heap::updateAllocationLimits): (JSC::Heap::didFinishCollection): (JSC::Heap::resumeCompilerThreads): (JSC::Zombify::visit): (JSC::Heap::forEachCodeBlockImpl): * heap/Heap.h: (JSC::Heap::allocatorForObjectWithoutDestructor): (JSC::Heap::allocatorForObjectWithDestructor): (JSC::Heap::allocatorForAuxiliaryData): (JSC::Heap::jitStubRoutines): (JSC::Heap::codeBlockSet): (JSC::Heap::storageAllocator): Deleted. * heap/HeapCell.h: (JSC::HeapCell::isZapped): Deleted. * heap/HeapCellInlines.h: Added. (JSC::HeapCell::isLargeAllocation): (JSC::HeapCell::cellContainer): (JSC::HeapCell::markedBlock): (JSC::HeapCell::largeAllocation): (JSC::HeapCell::heap): (JSC::HeapCell::vm): (JSC::HeapCell::cellSize): (JSC::HeapCell::allocatorAttributes): (JSC::HeapCell::destructionMode): (JSC::HeapCell::cellKind): * heap/HeapInlines.h: (JSC::Heap::heap): (JSC::Heap::isLive): (JSC::Heap::isMarked): (JSC::Heap::testAndSetMarked): (JSC::Heap::setMarked): (JSC::Heap::cellSize): (JSC::Heap::forEachCodeBlock): (JSC::Heap::allocateObjectOfType): (JSC::Heap::subspaceForObjectOfType): (JSC::Heap::allocatorForObjectOfType): (JSC::Heap::allocateAuxiliary): (JSC::Heap::tryAllocateAuxiliary): (JSC::Heap::tryReallocateAuxiliary): (JSC::Heap::isPointerGCObject): Deleted. (JSC::Heap::isValueGCObject): Deleted. * heap/HeapOperation.cpp: Added. (WTF::printInternal): * heap/HeapOperation.h: * heap/HeapUtil.h: Added. (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): (JSC::HeapUtil::isValueGCObject): * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::sweepNextBlock): * heap/IncrementalSweeper.h: * heap/LargeAllocation.cpp: Added. (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::lastChanceToFinalize): (JSC::LargeAllocation::shrink): (JSC::LargeAllocation::visitWeakSet): (JSC::LargeAllocation::reapWeakSet): (JSC::LargeAllocation::flip): (JSC::LargeAllocation::isEmpty): (JSC::LargeAllocation::sweep): (JSC::LargeAllocation::destroy): (JSC::LargeAllocation::dump): * heap/LargeAllocation.h: Added. (JSC::LargeAllocation::fromCell): (JSC::LargeAllocation::cell): (JSC::LargeAllocation::isLargeAllocation): (JSC::LargeAllocation::heap): (JSC::LargeAllocation::vm): (JSC::LargeAllocation::weakSet): (JSC::LargeAllocation::clearNewlyAllocated): (JSC::LargeAllocation::isNewlyAllocated): (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::isLive): (JSC::LargeAllocation::hasValidCell): (JSC::LargeAllocation::cellSize): (JSC::LargeAllocation::aboveLowerBound): (JSC::LargeAllocation::belowUpperBound): (JSC::LargeAllocation::contains): (JSC::LargeAllocation::attributes): (JSC::LargeAllocation::flipIfNecessary): (JSC::LargeAllocation::flipIfNecessaryConcurrently): (JSC::LargeAllocation::testAndSetMarked): (JSC::LargeAllocation::setMarked): (JSC::LargeAllocation::clearMarked): (JSC::LargeAllocation::noteMarked): (JSC::LargeAllocation::headerSize): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::retire): (JSC::MarkedAllocator::filterNextBlock): (JSC::MarkedAllocator::setNextBlockToSweep): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::blockHeaderSize): (JSC::MarkedAllocator::blockSizeForBytes): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::reset): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::setFreeList): (JSC::isListPagedOut): Deleted. (JSC::MarkedAllocator::tryAllocateHelper): Deleted. (JSC::MarkedAllocator::tryPopFreeList): Deleted. (JSC::MarkedAllocator::tryAllocate): Deleted. (JSC::MarkedAllocator::allocateBlock): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::takeLastActiveBlock): (JSC::MarkedAllocator::offsetOfFreeList): (JSC::MarkedAllocator::offsetOfCellSize): (JSC::MarkedAllocator::tryAllocate): (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted. (JSC::MarkedAllocator::MarkedAllocator): Deleted. (JSC::MarkedAllocator::init): Deleted. (JSC::MarkedAllocator::stopAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): (JSC::SetNewlyAllocatedFunctor::operator()): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::Handle::zap): (JSC::MarkedBlock::Handle::forEachFreeCell): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessarySlow): (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::needsFlip): (JSC::MarkedBlock::Handle::needsFlip): (JSC::MarkedBlock::Handle::willRemoveBlock): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::clearHasAnyMarked): (JSC::MarkedBlock::noteMarkedSlow): (WTF::printInternal): (JSC::MarkedBlock::create): Deleted. (JSC::MarkedBlock::destroy): Deleted. (JSC::MarkedBlock::callDestructor): Deleted. (JSC::MarkedBlock::specializedSweep): Deleted. (JSC::MarkedBlock::sweep): Deleted. (JSC::MarkedBlock::sweepHelper): Deleted. (JSC::MarkedBlock::stopAllocating): Deleted. (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted. (JSC::MarkedBlock::lastChanceToFinalize): Deleted. (JSC::MarkedBlock::resumeAllocating): Deleted. (JSC::MarkedBlock::didRetireBlock): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::VoidFunctor::returnValue): (JSC::MarkedBlock::CountFunctor::CountFunctor): (JSC::MarkedBlock::CountFunctor::count): (JSC::MarkedBlock::CountFunctor::returnValue): (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): (JSC::MarkedBlock::Handle::state): (JSC::MarkedBlock::needsDestruction): (JSC::MarkedBlock::handle): (JSC::MarkedBlock::Handle::block): (JSC::MarkedBlock::firstAtom): (JSC::MarkedBlock::atoms): (JSC::MarkedBlock::isAtomAligned): (JSC::MarkedBlock::Handle::cellAlign): (JSC::MarkedBlock::blockFor): (JSC::MarkedBlock::Handle::allocator): (JSC::MarkedBlock::Handle::heap): (JSC::MarkedBlock::Handle::vm): (JSC::MarkedBlock::vm): (JSC::MarkedBlock::Handle::weakSet): (JSC::MarkedBlock::weakSet): (JSC::MarkedBlock::Handle::shrink): (JSC::MarkedBlock::Handle::visitWeakSet): (JSC::MarkedBlock::Handle::reapWeakSet): (JSC::MarkedBlock::Handle::cellSize): (JSC::MarkedBlock::cellSize): (JSC::MarkedBlock::Handle::attributes): (JSC::MarkedBlock::attributes): (JSC::MarkedBlock::Handle::needsDestruction): (JSC::MarkedBlock::Handle::destruction): (JSC::MarkedBlock::Handle::cellKind): (JSC::MarkedBlock::Handle::markCount): (JSC::MarkedBlock::Handle::size): (JSC::MarkedBlock::atomNumber): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipForEdenCollection): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::Handle::assertFlipped): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::testAndSetMarked): (JSC::MarkedBlock::Handle::isNewlyAllocated): (JSC::MarkedBlock::Handle::setNewlyAllocated): (JSC::MarkedBlock::Handle::clearNewlyAllocated): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::isAtom): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::Handle::needsSweeping): (JSC::MarkedBlock::Handle::isAllocated): (JSC::MarkedBlock::Handle::isMarked): (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::hasAnyMarked): (JSC::MarkedBlock::noteMarked): (WTF::MarkedBlockHash::hash): (JSC::MarkedBlock::FreeList::FreeList): Deleted. (JSC::MarkedBlock::allocator): Deleted. (JSC::MarkedBlock::heap): Deleted. (JSC::MarkedBlock::shrink): Deleted. (JSC::MarkedBlock::visitWeakSet): Deleted. (JSC::MarkedBlock::reapWeakSet): Deleted. (JSC::MarkedBlock::willRemoveBlock): Deleted. (JSC::MarkedBlock::didConsumeFreeList): Deleted. (JSC::MarkedBlock::markCount): Deleted. (JSC::MarkedBlock::isEmpty): Deleted. (JSC::MarkedBlock::destruction): Deleted. (JSC::MarkedBlock::cellKind): Deleted. (JSC::MarkedBlock::size): Deleted. (JSC::MarkedBlock::capacity): Deleted. (JSC::MarkedBlock::setMarked): Deleted. (JSC::MarkedBlock::clearMarked): Deleted. (JSC::MarkedBlock::isNewlyAllocated): Deleted. (JSC::MarkedBlock::setNewlyAllocated): Deleted. (JSC::MarkedBlock::clearNewlyAllocated): Deleted. (JSC::MarkedBlock::isLive): Deleted. (JSC::MarkedBlock::isLiveCell): Deleted. (JSC::MarkedBlock::forEachCell): Deleted. (JSC::MarkedBlock::forEachLiveCell): Deleted. (JSC::MarkedBlock::forEachDeadCell): Deleted. (JSC::MarkedBlock::needsSweeping): Deleted. (JSC::MarkedBlock::isAllocated): Deleted. (JSC::MarkedBlock::isMarkedOrRetired): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::initializeSizeClassForStepSize): (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::lastChanceToFinalize): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::allocateLarge): (JSC::MarkedSpace::tryAllocateLarge): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::zombifySweep): (JSC::MarkedSpace::resetAllocators): (JSC::MarkedSpace::visitWeakSets): (JSC::MarkedSpace::reapWeakSets): (JSC::MarkedSpace::stopAllocating): (JSC::MarkedSpace::prepareForMarking): (JSC::MarkedSpace::resumeAllocating): (JSC::MarkedSpace::isPagedOut): (JSC::MarkedSpace::freeBlock): (JSC::MarkedSpace::freeOrShrinkBlock): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::VerifyMarked::operator()): (JSC::MarkedSpace::flip): (JSC::MarkedSpace::objectCount): (JSC::MarkedSpace::size): (JSC::MarkedSpace::capacity): (JSC::MarkedSpace::addActiveWeakSet): (JSC::MarkedSpace::didAddBlock): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::forEachAllocator): Deleted. (JSC::VerifyMarkedOrRetired::operator()): Deleted. (JSC::MarkedSpace::clearMarks): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::sizeClassToIndex): (JSC::MarkedSpace::indexToSizeClass): (JSC::MarkedSpace::version): (JSC::MarkedSpace::blocksWithNewObjects): (JSC::MarkedSpace::largeAllocations): (JSC::MarkedSpace::largeAllocationsNurseryOffset): (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection): (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin): (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd): (JSC::MarkedSpace::largeAllocationsForThisCollectionSize): (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): (JSC::MarkedSpace::allocatorFor): (JSC::MarkedSpace::destructorAllocatorFor): (JSC::MarkedSpace::auxiliaryAllocatorFor): (JSC::MarkedSpace::allocateWithoutDestructor): (JSC::MarkedSpace::allocateWithDestructor): (JSC::MarkedSpace::allocateAuxiliary): (JSC::MarkedSpace::tryAllocateAuxiliary): (JSC::MarkedSpace::forEachBlock): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::optimalSizeFor): (JSC::MarkedSpace::didAddBlock): Deleted. (JSC::MarkedSpace::didAllocateInBlock): Deleted. (JSC::MarkedSpace::objectCount): Deleted. (JSC::MarkedSpace::size): Deleted. (JSC::MarkedSpace::capacity): Deleted. * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::didStartMarking): (JSC::SlotVisitor::reset): (JSC::SlotVisitor::append): (JSC::SlotVisitor::appendJSCellOrAuxiliary): (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::appendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::noteLiveAuxiliaryCell): (JSC::SlotVisitor::visitChildren): * heap/SlotVisitor.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::WeakBlock): (JSC::WeakBlock::visit): (JSC::WeakBlock::reap): * heap/WeakBlock.h: (JSC::WeakBlock::disconnectContainer): (JSC::WeakBlock::disconnectMarkedBlock): Deleted. * heap/WeakSet.cpp: (JSC::WeakSet::~WeakSet): (JSC::WeakSet::sweep): (JSC::WeakSet::shrink): (JSC::WeakSet::addAllocator): * heap/WeakSet.h: (JSC::WeakSet::container): (JSC::WeakSet::setContainer): (JSC::WeakSet::WeakSet): (JSC::WeakSet::visit): (JSC::WeakSet::shrink): Deleted. * heap/WeakSetInlines.h: (JSC::WeakSet::allocate): * inspector/InjectedScriptManager.cpp: * inspector/JSGlobalObjectInspectorController.cpp: * inspector/JSJavaScriptCallFrame.cpp: * inspector/ScriptDebugServer.cpp: * inspector/agents/InspectorDebuggerAgent.cpp: * interpreter/CachedCall.h: (JSC::CachedCall::CachedCall): * interpreter/Interpreter.cpp: (JSC::loadVarargs): (JSC::StackFrame::sourceID): Deleted. (JSC::StackFrame::sourceURL): Deleted. (JSC::StackFrame::functionName): Deleted. (JSC::StackFrame::computeLineAndColumn): Deleted. (JSC::StackFrame::toString): Deleted. * interpreter/Interpreter.h: (JSC::StackFrame::isNative): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): (JSC::AssemblyHelpers::emitAllocate): (JSC::AssemblyHelpers::emitAllocateJSCell): (JSC::AssemblyHelpers::emitAllocateJSObject): (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): * jit/GCAwareJITStubRoutine.cpp: (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine): * jit/JIT.cpp: (JSC::JIT::compileCTINativeCall): (JSC::JIT::link): * jit/JIT.h: (JSC::JIT::compileCTINativeCall): Deleted. * jit/JITExceptions.cpp: (JSC::genericUnwind): * jit/JITExceptions.h: * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOpcodes32_64.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emitWriteBarrier): * jit/JITThunks.cpp: * jit/JITThunks.h: * jsc.cpp: (functionDescribeArray): (main): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntExceptions.cpp: * llint/LLIntThunks.cpp: * llint/LLIntThunks.h: * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter.cpp: * llint/LowLevelInterpreter32_64.asm: * llint/LowLevelInterpreter64.asm: * parser/ModuleAnalyzer.cpp: * parser/NodeConstructors.h: * parser/Nodes.h: * profiler/ProfilerBytecode.cpp: * profiler/ProfilerBytecode.h: * profiler/ProfilerBytecodeSequence.cpp: * runtime/ArrayConventions.h: (JSC::indexingHeaderForArrayStorage): (JSC::baseIndexingHeaderForArrayStorage): (JSC::indexingHeaderForArray): Deleted. (JSC::baseIndexingHeaderForArray): Deleted. * runtime/ArrayPrototype.cpp: (JSC::arrayProtoFuncSplice): (JSC::concatAppendOne): (JSC::arrayProtoPrivateFuncConcatMemcpy): * runtime/ArrayStorage.h: (JSC::ArrayStorage::vectorLength): (JSC::ArrayStorage::totalSizeFor): (JSC::ArrayStorage::totalSize): (JSC::ArrayStorage::availableVectorLength): (JSC::ArrayStorage::optimalVectorLength): (JSC::ArrayStorage::sizeFor): Deleted. * runtime/AuxiliaryBarrier.h: Added. (JSC::AuxiliaryBarrier::AuxiliaryBarrier): (JSC::AuxiliaryBarrier::clear): (JSC::AuxiliaryBarrier::get): (JSC::AuxiliaryBarrier::slot): (JSC::AuxiliaryBarrier::operator bool): (JSC::AuxiliaryBarrier::setWithoutBarrier): * runtime/AuxiliaryBarrierInlines.h: Added. (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier): (JSC::AuxiliaryBarrier<T>::set): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::availableContiguousVectorLength): (JSC::Butterfly::optimalContiguousVectorLength): (JSC::Butterfly::createUninitialized): (JSC::Butterfly::growArrayRight): * runtime/ClonedArguments.cpp: (JSC::ClonedArguments::createEmpty): * runtime/CommonSlowPathsExceptions.cpp: * runtime/CommonSlowPathsExceptions.h: * runtime/DataView.cpp: * runtime/DirectArguments.h: * runtime/ECMAScriptSpecInternalFunctions.cpp: * runtime/Error.cpp: * runtime/Error.h: * runtime/ErrorInstance.cpp: * runtime/ErrorInstance.h: * runtime/Exception.cpp: * runtime/Exception.h: * runtime/GeneratorFrame.cpp: * runtime/GeneratorPrototype.cpp: * runtime/InternalFunction.cpp: (JSC::InternalFunction::InternalFunction): * runtime/IntlCollator.cpp: * runtime/IntlCollatorConstructor.cpp: * runtime/IntlCollatorPrototype.cpp: * runtime/IntlDateTimeFormat.cpp: * runtime/IntlDateTimeFormatConstructor.cpp: * runtime/IntlDateTimeFormatPrototype.cpp: * runtime/IntlNumberFormat.cpp: * runtime/IntlNumberFormatConstructor.cpp: * runtime/IntlNumberFormatPrototype.cpp: * runtime/IntlObject.cpp: * runtime/IteratorPrototype.cpp: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitialized): (JSC::JSArray::setLengthWritable): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLengthWithArrayStorage): (JSC::JSArray::appendMemcpy): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithArrayStorage): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::createContiguousArrayButterfly): (JSC::createArrayButterfly): (JSC::JSArray::create): (JSC::JSArray::tryCreateUninitialized): Deleted. * runtime/JSArrayBufferView.h: * runtime/JSCInlines.h: * runtime/JSCJSValue.cpp: (JSC::JSValue::dumpInContextAssumingStructure): * runtime/JSCallee.cpp: (JSC::JSCallee::JSCallee): * runtime/JSCell.cpp: (JSC::JSCell::estimatedSize): * runtime/JSCell.h: (JSC::JSCell::cellStateOffset): Deleted. * runtime/JSCellInlines.h: (JSC::ExecState::vm): (JSC::JSCell::classInfo): (JSC::JSCell::callDestructor): (JSC::JSCell::vm): Deleted. * runtime/JSFunction.cpp: (JSC::JSFunction::create): (JSC::JSFunction::allocateAndInitializeRareData): (JSC::JSFunction::initializeRareData): (JSC::JSFunction::getOwnPropertySlot): (JSC::JSFunction::put): (JSC::JSFunction::deleteProperty): (JSC::JSFunction::defineOwnProperty): (JSC::JSFunction::setFunctionName): (JSC::JSFunction::reifyLength): (JSC::JSFunction::reifyName): (JSC::JSFunction::reifyLazyPropertyIfNeeded): (JSC::JSFunction::reifyBoundNameIfNeeded): * runtime/JSFunction.h: * runtime/JSFunctionInlines.h: (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint): (JSC::JSFunction::JSFunction): * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory): * runtime/JSInternalPromise.cpp: * runtime/JSInternalPromiseConstructor.cpp: * runtime/JSInternalPromiseDeferred.cpp: * runtime/JSInternalPromisePrototype.cpp: * runtime/JSJob.cpp: * runtime/JSMapIterator.cpp: * runtime/JSModuleNamespaceObject.cpp: * runtime/JSModuleRecord.cpp: * runtime/JSObject.cpp: (JSC::JSObject::visitButterfly): (JSC::JSObject::notifyPresenceOfIndexedAccessors): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createInitialUndecided): (JSC::JSObject::createInitialInt32): (JSC::JSObject::createInitialDouble): (JSC::JSObject::createInitialContiguous): (JSC::JSObject::createArrayStorage): (JSC::JSObject::createInitialArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::putByIndexBeyondVectorLength): (JSC::JSObject::putDirectIndexBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::increaseVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::growOutOfLineStorage): (JSC::JSObject::copyButterfly): Deleted. (JSC::JSObject::copyBackingStore): Deleted. * runtime/JSObject.h: (JSC::JSObject::globalObject): (JSC::JSObject::putDirectInternal): (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted. * runtime/JSObjectInlines.h: * runtime/JSPromise.cpp: * runtime/JSPromiseConstructor.cpp: * runtime/JSPromiseDeferred.cpp: * runtime/JSPromisePrototype.cpp: * runtime/JSPropertyNameIterator.cpp: * runtime/JSScope.cpp: (JSC::JSScope::resolve): * runtime/JSScope.h: (JSC::JSScope::globalObject): (JSC::JSScope::vm): Deleted. * runtime/JSSetIterator.cpp: * runtime/JSStringIterator.cpp: * runtime/JSTemplateRegistryKey.cpp: * runtime/JSTypedArrayViewConstructor.cpp: * runtime/JSTypedArrayViewPrototype.cpp: * runtime/JSWeakMap.cpp: * runtime/JSWeakSet.cpp: * runtime/MapConstructor.cpp: * runtime/MapIteratorPrototype.cpp: * runtime/MapPrototype.cpp: * runtime/NativeErrorConstructor.cpp: * runtime/NativeStdFunctionCell.cpp: * runtime/Operations.h: (JSC::scribbleFreeCells): (JSC::scribble): * runtime/Options.h: * runtime/PropertyTable.cpp: * runtime/ProxyConstructor.cpp: * runtime/ProxyObject.cpp: * runtime/ProxyRevoke.cpp: * runtime/RegExp.cpp: (JSC::RegExp::match): (JSC::RegExp::matchConcurrently): (JSC::RegExp::matchCompareWithInterpreter): * runtime/RegExp.h: * runtime/RegExpConstructor.h: * runtime/RegExpInlines.h: (JSC::RegExp::matchInline): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): (JSC::createRegExpMatchesArray): * runtime/RegExpPrototype.cpp: (JSC::genericSplit): * runtime/RuntimeType.cpp: * runtime/SamplingProfiler.cpp: (JSC::SamplingProfiler::processUnverifiedStackTraces): * runtime/SetConstructor.cpp: * runtime/SetIteratorPrototype.cpp: * runtime/SetPrototype.cpp: * runtime/StackFrame.cpp: Added. (JSC::StackFrame::sourceID): (JSC::StackFrame::sourceURL): (JSC::StackFrame::functionName): (JSC::StackFrame::computeLineAndColumn): (JSC::StackFrame::toString): * runtime/StackFrame.h: Added. (JSC::StackFrame::isNative): * runtime/StringConstructor.cpp: * runtime/StringIteratorPrototype.cpp: * runtime/StructureInlines.h: (JSC::Structure::propertyTable): * runtime/TemplateRegistry.cpp: * runtime/TestRunnerUtils.cpp: (JSC::finalizeStatsAtEndOfTesting): * runtime/TestRunnerUtils.h: * runtime/TypeProfilerLog.cpp: * runtime/TypeSet.cpp: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::ensureStackCapacityForCLoop): (JSC::VM::isSafeToRecurseSoftCLoop): * runtime/VM.h: * runtime/VMEntryScope.h: * runtime/VMInlines.h: (JSC::VM::ensureStackCapacityFor): (JSC::VM::isSafeToRecurseSoft): * runtime/WeakMapConstructor.cpp: * runtime/WeakMapData.cpp: * runtime/WeakMapPrototype.cpp: * runtime/WeakSetConstructor.cpp: * runtime/WeakSetPrototype.cpp: * testRegExp.cpp: (testOneRegExp): * tools/JSDollarVM.cpp: * tools/JSDollarVMPrototype.cpp: (JSC::JSDollarVMPrototype::isInObjectSpace): Source/WebCore: No new tests because no new WebCore behavior. Just rewiring #includes. * ForwardingHeaders/heap/HeapInlines.h: Added. * ForwardingHeaders/interpreter/Interpreter.h: Removed. * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added. * Modules/indexeddb/IDBCursorWithValue.cpp: * Modules/indexeddb/client/TransactionOperation.cpp: * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp: * Modules/indexeddb/server/UniqueIDBDatabase.cpp: * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp: * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp: * bindings/js/JSClientRectCustom.cpp: * bindings/js/JSDOMBinding.cpp: * bindings/js/JSDOMBinding.h: * bindings/js/JSDeviceMotionEventCustom.cpp: * bindings/js/JSDeviceOrientationEventCustom.cpp: * bindings/js/JSErrorEventCustom.cpp: * bindings/js/JSIDBCursorWithValueCustom.cpp: * bindings/js/JSIDBIndexCustom.cpp: * bindings/js/JSPopStateEventCustom.cpp: * bindings/js/JSWebGL2RenderingContextCustom.cpp: * bindings/js/JSWorkerGlobalScopeCustom.cpp: * bindings/js/WorkerScriptController.cpp: * contentextensions/ContentExtensionParser.cpp: * dom/ErrorEvent.cpp: * html/HTMLCanvasElement.cpp: * html/MediaDocument.cpp: * inspector/CommandLineAPIModule.cpp: * loader/EmptyClients.cpp: * page/CaptionUserPreferences.cpp: * page/Frame.cpp: * page/PageGroup.cpp: * page/UserContentController.cpp: * platform/mock/mediasource/MockBox.cpp: * testing/GCObservation.cpp: Source/WebKit2: Just rewiring some #includes. * UIProcess/ViewGestureController.cpp: * UIProcess/WebPageProxy.cpp: * UIProcess/WebProcessPool.cpp: * UIProcess/WebProcessProxy.cpp: * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp: * WebProcess/Plugins/Netscape/JSNPObject.cpp: Source/WTF: I needed tryFastAlignedMalloc() so I added it. * wtf/FastMalloc.cpp: (WTF::tryFastAlignedMalloc): * wtf/FastMalloc.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): (WTF::ParkingLot::forEach): Deleted. * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::forEach): * wtf/ScopedLambda.h: (WTF::scopedLambdaRef): * wtf/SentinelLinkedList.h: (WTF::SentinelLinkedList::forEach): (WTF::RawNode>::takeFrom): * wtf/SimpleStats.h: (WTF::SimpleStats::operator bool): (WTF::SimpleStats::operator!): Deleted. Tools: * DumpRenderTree/TestRunner.cpp: * DumpRenderTree/mac/DumpRenderTree.mm: (DumpRenderTreeMain): * Scripts/run-jsc-stress-tests: * TestWebKitAPI/Tests/WTF/Vector.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/179778@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
static void unparkOne(const void* address, const Callback& callback)
ScopedLambda should have a lifetime story that makes sense to the compiler https://bugs.webkit.org/show_bug.cgi?id=158118 Reviewed by Mark Lam. Source/WTF: Prior to this change, there were two lifetime bugs in ScopedLambda: - scopedLambda(Functor&&) would bind Functor to const lambda&, so the resulting ScopedLambdaFunctor would hold a reference to the original lambda. This would have surprising behavior; for example it meant that this code was wrong: auto l = scopedLambda<things>([&] ...); The solution is to have explicit copy/move versions of scopedLambda() rather than rely on perfect forwarding. - ScopedLambdaFunctor did not override its copy or move operations, so if the compiler did not RVO scopedLambda(), it would return a ScopedLambdaFunctor whose m_arg points to a dead temporary ScopedLambdaFunctor instance. The solution is to have explicit copy/move constructors and operators, which preserve the invariant that ScopedLambda::m_arg points to this. One nice side-effect of all of these constructors and operators being explicit is that we can rely on WTFMove's excellent assertions, which helped catch the first issue. This reverts ParkingLot to use ScopedLambda again. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): * wtf/ScopedLambda.h: (WTF::scopedLambda): Tools: Added a test case. This test crashes before the fix and now it passes. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/ScopedLambda.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/176236@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@201433 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-05-26 21:58:42 +00:00
{
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression https://bugs.webkit.org/show_bug.cgi?id=160125 Reviewed by Geoffrey Garen and Keith Miller. JSTests: Most of the things I did properly covered by existing tests, but I found some simple cases of unshifting that had sketchy coverage. * stress/array-storage-array-unshift.js: Added. * stress/contiguous-array-unshift.js: Added. * stress/double-array-unshift.js: Added. * stress/int32-array-unshift.js: Added. Source/bmalloc: I needed to tryMemalign, so I added such a thing. * bmalloc/Allocator.cpp: (bmalloc::Allocator::allocate): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): * bmalloc/Allocator.h: * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): * bmalloc/bmalloc.h: (bmalloc::api::tryMemalign): Source/JavaScriptCore: In order to make the GC concurrent (bug 149432), we would either need to enable concurrent copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead from the barriers alone. Considering that MarkedSpace does a decent job of avoiding fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want to get rid of copied space. This change moves copied space's biggest client over to marked space. Moving butterflies to marked space means having them use the new Auxiliary HeapCell allocation path. This is a fairly mechanical change, but it caused performance regressions everywhere, so this change also fixes MarkedSpace's performance issues. At a high level the mechanical changes are: - We use AuxiliaryBarrier instead of CopyBarrier. - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth. - The JITs have to emit inlined marked space allocations instead of inline copy space allocations. - Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't. That's about 1/3 of this change. But this led to performance problems, which I fixed with optimizations that amounted to a major MarkedSpace rewrite: - MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely. - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well. - Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being: 16, 32, 48, 64, 80, 112, 160, 224, 320 The next size class would have been 432, but that's above the 400 byte cutoff. All of this is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what size classes you end up with by doing --dumpSizeClasses=true. - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: https://bugs.webkit.org/show_bug.cgi?id=161581. - Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree. - FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky. - It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch! - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes. All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1% regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a best-case scenario for bump allocation. I think that we should fix bmalloc to make up the difference, but take the hit for now because it's a crazy corner case. By comparison, the alternative approach of using a copy barrier would have cost us 1-2%. That's the real apples-to-apples comparison if your premise is that we should have a concurrent GC. After we finish removing copied space, we will be barrier-ready for concurrent GC: we already have a marking barrier and we simply won't need a copying barrier. This change gets us there for the purposes of our benchmarks, since the remaining clients of copied space are not very important. On the other hand, if we keep copying, then getting barrier-ready would mean adding back the copy barrier, which costs more perf. We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving typed arrays and a few other weird things over to Aux MarkedSpace. This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell, and CellContainer meant that I had to include those files from everywhere. Fortunately, just including JSCInlines.h (instead of manually including the files that includes) is usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something that we were already basically doing. In places where JSCInlines.h would be too much, I just included HeapInlines.h. This got weird, because we previously included HeapInlines.h from JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that meant having to manually include HeapInlines.h from the places that previously got it implicitly via JSObject.h. But that led to more problems for some reason: I started getting build errors because non-JSC files were having trouble including Opcode.h. That's just silly, since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and made it impossible to include it from outside JSC. This was a lot of work, but it was necessary to get the patch to build on all ports. It's also a net win. There were many places in WebCore that were transitively including a *ton* of JSC headers just because of the JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons. * API/JSManagedValue.mm: (-[JSManagedValue initWithValue:]): * API/JSTypedArray.cpp: * API/ObjCCallbackFunction.mm: * API/tests/testapi.mm: (testObjectiveCAPI): (testWeakValue): Deleted. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * Scripts/builtins/builtins_generate_combined_implementation.py: (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py: (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_separate_implementation.py: (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes): * assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::JumpList::link): (JSC::AbstractMacroAssembler::JumpList::linkTo): * assembler/MacroAssembler.h: * assembler/MacroAssemblerARM64.h: (JSC::MacroAssemblerARM64::add32): * assembler/MacroAssemblerCodeRef.cpp: Added. (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): (JSC::MacroAssemblerCodePtr::dumpWithName): (JSC::MacroAssemblerCodePtr::dump): (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): (JSC::MacroAssemblerCodeRef::dump): * assembler/MacroAssemblerCodeRef.h: (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted. (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted. (JSC::MacroAssemblerCodePtr::dump): Deleted. (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted. (JSC::MacroAssemblerCodeRef::dump): Deleted. * b3/B3BasicBlock.cpp: (JSC::B3::BasicBlock::appendBoolConstant): * b3/B3BasicBlock.h: * b3/B3DuplicateTails.cpp: * b3/B3StackmapGenerationParams.h: * b3/testb3.cpp: (JSC::B3::testPatchpointTerminalReturnValue): (JSC::B3::run): * bindings/ScriptValue.cpp: * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeLivenessAnalysis.cpp: * bytecode/BytecodeUseDef.h: * bytecode/CallLinkInfo.cpp: (JSC::CallLinkInfo::callTypeFor): * bytecode/CallLinkInfo.h: (JSC::CallLinkInfo::callTypeFor): Deleted. * bytecode/CallLinkStatus.cpp: * bytecode/CodeBlock.cpp: (JSC::CodeBlock::finishCreation): (JSC::CodeBlock::clearLLIntGetByIdCache): (JSC::CodeBlock::predictedMachineCodeSize): * bytecode/CodeBlock.h: (JSC::CodeBlock::jitCodeMap): Deleted. (JSC::clearLLIntGetByIdCache): Deleted. * bytecode/ExecutionCounter.h: * bytecode/Instruction.h: * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp: (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal): * bytecode/ObjectAllocationProfile.h: (JSC::ObjectAllocationProfile::isNull): (JSC::ObjectAllocationProfile::initialize): * bytecode/Opcode.h: (JSC::padOpcodeName): * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): (JSC::PolymorphicAccess::regenerate): * bytecode/PolymorphicAccess.h: * bytecode/PreciseJumpTargets.cpp: * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecode/UnlinkedCodeBlock.cpp: (JSC::UnlinkedCodeBlock::vm): Deleted. * bytecode/UnlinkedCodeBlock.h: * bytecode/UnlinkedInstructionStream.cpp: * bytecode/UnlinkedInstructionStream.h: * dfg/DFGOperations.cpp: * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT.h: (JSC::DFG::SpeculativeJIT::emitAllocateJSCell): (JSC::DFG::SpeculativeJIT::emitAllocateJSObject): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGStrengthReductionPhase.cpp: (JSC::DFG::StrengthReductionPhase::handleNode): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCompile.cpp: * ftl/FTLJITFinalizer.cpp: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments): (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest): (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell): (JSC::FTL::DFG::LowerDFGToB3::allocateCell): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted. * ftl/FTLOutput.cpp: (JSC::FTL::Output::constBool): (JSC::FTL::Output::add): (JSC::FTL::Output::shl): (JSC::FTL::Output::aShr): (JSC::FTL::Output::lShr): (JSC::FTL::Output::zeroExt): (JSC::FTL::Output::equal): (JSC::FTL::Output::notEqual): (JSC::FTL::Output::above): (JSC::FTL::Output::aboveOrEqual): (JSC::FTL::Output::below): (JSC::FTL::Output::belowOrEqual): (JSC::FTL::Output::greaterThan): (JSC::FTL::Output::greaterThanOrEqual): (JSC::FTL::Output::lessThan): (JSC::FTL::Output::lessThanOrEqual): (JSC::FTL::Output::select): (JSC::FTL::Output::appendSuccessor): (JSC::FTL::Output::addIncomingToPhi): * ftl/FTLOutput.h: * ftl/FTLValueFromBlock.h: (JSC::FTL::ValueFromBlock::operator bool): (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted. * ftl/FTLWeightedTarget.h: (JSC::FTL::WeightedTarget::frequentedBlock): * heap/CellContainer.h: Added. (JSC::CellContainer::CellContainer): (JSC::CellContainer::operator bool): (JSC::CellContainer::isMarkedBlock): (JSC::CellContainer::isLargeAllocation): (JSC::CellContainer::markedBlock): (JSC::CellContainer::largeAllocation): * heap/CellContainerInlines.h: Added. (JSC::CellContainer::isMarked): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::noteMarked): (JSC::CellContainer::cellSize): (JSC::CellContainer::weakSet): (JSC::CellContainer::flipIfNecessary): * heap/ConservativeRoots.cpp: (JSC::ConservativeRoots::ConservativeRoots): (JSC::ConservativeRoots::~ConservativeRoots): (JSC::ConservativeRoots::grow): (JSC::ConservativeRoots::genericAddPointer): (JSC::ConservativeRoots::genericAddSpan): * heap/ConservativeRoots.h: (JSC::ConservativeRoots::roots): * heap/CopyToken.h: * heap/FreeList.cpp: Added. (JSC::FreeList::dump): * heap/FreeList.h: Added. (JSC::FreeList::FreeList): (JSC::FreeList::list): (JSC::FreeList::bump): (JSC::FreeList::operator==): (JSC::FreeList::operator!=): (JSC::FreeList::operator bool): (JSC::FreeList::allocationWillFail): (JSC::FreeList::allocationWillSucceed): * heap/GCTypeMap.h: Added. (JSC::GCTypeMap::operator[]): * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalizeUnconditionalFinalizers): (JSC::Heap::markRoots): (JSC::Heap::copyBackingStores): (JSC::Heap::gatherStackRoots): (JSC::Heap::gatherJSStackRoots): (JSC::Heap::gatherScratchBufferRoots): (JSC::Heap::clearLivenessData): (JSC::Heap::visitSmallStrings): (JSC::Heap::visitConservativeRoots): (JSC::Heap::removeDeadCompilerWorklistEntries): (JSC::Heap::gatherExtraHeapSnapshotData): (JSC::Heap::removeDeadHeapSnapshotNodes): (JSC::Heap::visitProtectedObjects): (JSC::Heap::visitArgumentBuffers): (JSC::Heap::visitException): (JSC::Heap::visitStrongHandles): (JSC::Heap::visitHandleStack): (JSC::Heap::visitSamplingProfiler): (JSC::Heap::traceCodeBlocksAndJITStubRoutines): (JSC::Heap::converge): (JSC::Heap::visitWeakHandles): (JSC::Heap::updateObjectCounts): (JSC::Heap::clearUnmarkedExecutables): (JSC::Heap::deleteUnmarkedCompiledCode): (JSC::Heap::collectAllGarbage): (JSC::Heap::collect): (JSC::Heap::collectWithoutAnySweep): (JSC::Heap::collectImpl): (JSC::Heap::suspendCompilerThreads): (JSC::Heap::willStartCollection): (JSC::Heap::flushOldStructureIDTables): (JSC::Heap::flushWriteBarrierBuffer): (JSC::Heap::stopAllocation): (JSC::Heap::prepareForMarking): (JSC::Heap::reapWeakHandles): (JSC::Heap::pruneStaleEntriesFromWeakGCMaps): (JSC::Heap::sweepArrayBuffers): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): (JSC::MarkedBlockSnapshotFunctor::operator()): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::deleteSourceProviderCaches): (JSC::Heap::notifyIncrementalSweeper): (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks): (JSC::Heap::resetAllocators): (JSC::Heap::updateAllocationLimits): (JSC::Heap::didFinishCollection): (JSC::Heap::resumeCompilerThreads): (JSC::Zombify::visit): (JSC::Heap::forEachCodeBlockImpl): * heap/Heap.h: (JSC::Heap::allocatorForObjectWithoutDestructor): (JSC::Heap::allocatorForObjectWithDestructor): (JSC::Heap::allocatorForAuxiliaryData): (JSC::Heap::jitStubRoutines): (JSC::Heap::codeBlockSet): (JSC::Heap::storageAllocator): Deleted. * heap/HeapCell.h: (JSC::HeapCell::isZapped): Deleted. * heap/HeapCellInlines.h: Added. (JSC::HeapCell::isLargeAllocation): (JSC::HeapCell::cellContainer): (JSC::HeapCell::markedBlock): (JSC::HeapCell::largeAllocation): (JSC::HeapCell::heap): (JSC::HeapCell::vm): (JSC::HeapCell::cellSize): (JSC::HeapCell::allocatorAttributes): (JSC::HeapCell::destructionMode): (JSC::HeapCell::cellKind): * heap/HeapInlines.h: (JSC::Heap::heap): (JSC::Heap::isLive): (JSC::Heap::isMarked): (JSC::Heap::testAndSetMarked): (JSC::Heap::setMarked): (JSC::Heap::cellSize): (JSC::Heap::forEachCodeBlock): (JSC::Heap::allocateObjectOfType): (JSC::Heap::subspaceForObjectOfType): (JSC::Heap::allocatorForObjectOfType): (JSC::Heap::allocateAuxiliary): (JSC::Heap::tryAllocateAuxiliary): (JSC::Heap::tryReallocateAuxiliary): (JSC::Heap::isPointerGCObject): Deleted. (JSC::Heap::isValueGCObject): Deleted. * heap/HeapOperation.cpp: Added. (WTF::printInternal): * heap/HeapOperation.h: * heap/HeapUtil.h: Added. (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): (JSC::HeapUtil::isValueGCObject): * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::sweepNextBlock): * heap/IncrementalSweeper.h: * heap/LargeAllocation.cpp: Added. (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::lastChanceToFinalize): (JSC::LargeAllocation::shrink): (JSC::LargeAllocation::visitWeakSet): (JSC::LargeAllocation::reapWeakSet): (JSC::LargeAllocation::flip): (JSC::LargeAllocation::isEmpty): (JSC::LargeAllocation::sweep): (JSC::LargeAllocation::destroy): (JSC::LargeAllocation::dump): * heap/LargeAllocation.h: Added. (JSC::LargeAllocation::fromCell): (JSC::LargeAllocation::cell): (JSC::LargeAllocation::isLargeAllocation): (JSC::LargeAllocation::heap): (JSC::LargeAllocation::vm): (JSC::LargeAllocation::weakSet): (JSC::LargeAllocation::clearNewlyAllocated): (JSC::LargeAllocation::isNewlyAllocated): (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::isLive): (JSC::LargeAllocation::hasValidCell): (JSC::LargeAllocation::cellSize): (JSC::LargeAllocation::aboveLowerBound): (JSC::LargeAllocation::belowUpperBound): (JSC::LargeAllocation::contains): (JSC::LargeAllocation::attributes): (JSC::LargeAllocation::flipIfNecessary): (JSC::LargeAllocation::flipIfNecessaryConcurrently): (JSC::LargeAllocation::testAndSetMarked): (JSC::LargeAllocation::setMarked): (JSC::LargeAllocation::clearMarked): (JSC::LargeAllocation::noteMarked): (JSC::LargeAllocation::headerSize): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::retire): (JSC::MarkedAllocator::filterNextBlock): (JSC::MarkedAllocator::setNextBlockToSweep): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::blockHeaderSize): (JSC::MarkedAllocator::blockSizeForBytes): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::reset): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::setFreeList): (JSC::isListPagedOut): Deleted. (JSC::MarkedAllocator::tryAllocateHelper): Deleted. (JSC::MarkedAllocator::tryPopFreeList): Deleted. (JSC::MarkedAllocator::tryAllocate): Deleted. (JSC::MarkedAllocator::allocateBlock): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::takeLastActiveBlock): (JSC::MarkedAllocator::offsetOfFreeList): (JSC::MarkedAllocator::offsetOfCellSize): (JSC::MarkedAllocator::tryAllocate): (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted. (JSC::MarkedAllocator::MarkedAllocator): Deleted. (JSC::MarkedAllocator::init): Deleted. (JSC::MarkedAllocator::stopAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): (JSC::SetNewlyAllocatedFunctor::operator()): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::Handle::zap): (JSC::MarkedBlock::Handle::forEachFreeCell): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessarySlow): (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::needsFlip): (JSC::MarkedBlock::Handle::needsFlip): (JSC::MarkedBlock::Handle::willRemoveBlock): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::clearHasAnyMarked): (JSC::MarkedBlock::noteMarkedSlow): (WTF::printInternal): (JSC::MarkedBlock::create): Deleted. (JSC::MarkedBlock::destroy): Deleted. (JSC::MarkedBlock::callDestructor): Deleted. (JSC::MarkedBlock::specializedSweep): Deleted. (JSC::MarkedBlock::sweep): Deleted. (JSC::MarkedBlock::sweepHelper): Deleted. (JSC::MarkedBlock::stopAllocating): Deleted. (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted. (JSC::MarkedBlock::lastChanceToFinalize): Deleted. (JSC::MarkedBlock::resumeAllocating): Deleted. (JSC::MarkedBlock::didRetireBlock): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::VoidFunctor::returnValue): (JSC::MarkedBlock::CountFunctor::CountFunctor): (JSC::MarkedBlock::CountFunctor::count): (JSC::MarkedBlock::CountFunctor::returnValue): (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): (JSC::MarkedBlock::Handle::state): (JSC::MarkedBlock::needsDestruction): (JSC::MarkedBlock::handle): (JSC::MarkedBlock::Handle::block): (JSC::MarkedBlock::firstAtom): (JSC::MarkedBlock::atoms): (JSC::MarkedBlock::isAtomAligned): (JSC::MarkedBlock::Handle::cellAlign): (JSC::MarkedBlock::blockFor): (JSC::MarkedBlock::Handle::allocator): (JSC::MarkedBlock::Handle::heap): (JSC::MarkedBlock::Handle::vm): (JSC::MarkedBlock::vm): (JSC::MarkedBlock::Handle::weakSet): (JSC::MarkedBlock::weakSet): (JSC::MarkedBlock::Handle::shrink): (JSC::MarkedBlock::Handle::visitWeakSet): (JSC::MarkedBlock::Handle::reapWeakSet): (JSC::MarkedBlock::Handle::cellSize): (JSC::MarkedBlock::cellSize): (JSC::MarkedBlock::Handle::attributes): (JSC::MarkedBlock::attributes): (JSC::MarkedBlock::Handle::needsDestruction): (JSC::MarkedBlock::Handle::destruction): (JSC::MarkedBlock::Handle::cellKind): (JSC::MarkedBlock::Handle::markCount): (JSC::MarkedBlock::Handle::size): (JSC::MarkedBlock::atomNumber): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipForEdenCollection): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::Handle::assertFlipped): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::testAndSetMarked): (JSC::MarkedBlock::Handle::isNewlyAllocated): (JSC::MarkedBlock::Handle::setNewlyAllocated): (JSC::MarkedBlock::Handle::clearNewlyAllocated): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::isAtom): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::Handle::needsSweeping): (JSC::MarkedBlock::Handle::isAllocated): (JSC::MarkedBlock::Handle::isMarked): (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::hasAnyMarked): (JSC::MarkedBlock::noteMarked): (WTF::MarkedBlockHash::hash): (JSC::MarkedBlock::FreeList::FreeList): Deleted. (JSC::MarkedBlock::allocator): Deleted. (JSC::MarkedBlock::heap): Deleted. (JSC::MarkedBlock::shrink): Deleted. (JSC::MarkedBlock::visitWeakSet): Deleted. (JSC::MarkedBlock::reapWeakSet): Deleted. (JSC::MarkedBlock::willRemoveBlock): Deleted. (JSC::MarkedBlock::didConsumeFreeList): Deleted. (JSC::MarkedBlock::markCount): Deleted. (JSC::MarkedBlock::isEmpty): Deleted. (JSC::MarkedBlock::destruction): Deleted. (JSC::MarkedBlock::cellKind): Deleted. (JSC::MarkedBlock::size): Deleted. (JSC::MarkedBlock::capacity): Deleted. (JSC::MarkedBlock::setMarked): Deleted. (JSC::MarkedBlock::clearMarked): Deleted. (JSC::MarkedBlock::isNewlyAllocated): Deleted. (JSC::MarkedBlock::setNewlyAllocated): Deleted. (JSC::MarkedBlock::clearNewlyAllocated): Deleted. (JSC::MarkedBlock::isLive): Deleted. (JSC::MarkedBlock::isLiveCell): Deleted. (JSC::MarkedBlock::forEachCell): Deleted. (JSC::MarkedBlock::forEachLiveCell): Deleted. (JSC::MarkedBlock::forEachDeadCell): Deleted. (JSC::MarkedBlock::needsSweeping): Deleted. (JSC::MarkedBlock::isAllocated): Deleted. (JSC::MarkedBlock::isMarkedOrRetired): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::initializeSizeClassForStepSize): (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::lastChanceToFinalize): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::allocateLarge): (JSC::MarkedSpace::tryAllocateLarge): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::zombifySweep): (JSC::MarkedSpace::resetAllocators): (JSC::MarkedSpace::visitWeakSets): (JSC::MarkedSpace::reapWeakSets): (JSC::MarkedSpace::stopAllocating): (JSC::MarkedSpace::prepareForMarking): (JSC::MarkedSpace::resumeAllocating): (JSC::MarkedSpace::isPagedOut): (JSC::MarkedSpace::freeBlock): (JSC::MarkedSpace::freeOrShrinkBlock): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::VerifyMarked::operator()): (JSC::MarkedSpace::flip): (JSC::MarkedSpace::objectCount): (JSC::MarkedSpace::size): (JSC::MarkedSpace::capacity): (JSC::MarkedSpace::addActiveWeakSet): (JSC::MarkedSpace::didAddBlock): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::forEachAllocator): Deleted. (JSC::VerifyMarkedOrRetired::operator()): Deleted. (JSC::MarkedSpace::clearMarks): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::sizeClassToIndex): (JSC::MarkedSpace::indexToSizeClass): (JSC::MarkedSpace::version): (JSC::MarkedSpace::blocksWithNewObjects): (JSC::MarkedSpace::largeAllocations): (JSC::MarkedSpace::largeAllocationsNurseryOffset): (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection): (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin): (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd): (JSC::MarkedSpace::largeAllocationsForThisCollectionSize): (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): (JSC::MarkedSpace::allocatorFor): (JSC::MarkedSpace::destructorAllocatorFor): (JSC::MarkedSpace::auxiliaryAllocatorFor): (JSC::MarkedSpace::allocateWithoutDestructor): (JSC::MarkedSpace::allocateWithDestructor): (JSC::MarkedSpace::allocateAuxiliary): (JSC::MarkedSpace::tryAllocateAuxiliary): (JSC::MarkedSpace::forEachBlock): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::optimalSizeFor): (JSC::MarkedSpace::didAddBlock): Deleted. (JSC::MarkedSpace::didAllocateInBlock): Deleted. (JSC::MarkedSpace::objectCount): Deleted. (JSC::MarkedSpace::size): Deleted. (JSC::MarkedSpace::capacity): Deleted. * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::didStartMarking): (JSC::SlotVisitor::reset): (JSC::SlotVisitor::append): (JSC::SlotVisitor::appendJSCellOrAuxiliary): (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::appendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::noteLiveAuxiliaryCell): (JSC::SlotVisitor::visitChildren): * heap/SlotVisitor.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::WeakBlock): (JSC::WeakBlock::visit): (JSC::WeakBlock::reap): * heap/WeakBlock.h: (JSC::WeakBlock::disconnectContainer): (JSC::WeakBlock::disconnectMarkedBlock): Deleted. * heap/WeakSet.cpp: (JSC::WeakSet::~WeakSet): (JSC::WeakSet::sweep): (JSC::WeakSet::shrink): (JSC::WeakSet::addAllocator): * heap/WeakSet.h: (JSC::WeakSet::container): (JSC::WeakSet::setContainer): (JSC::WeakSet::WeakSet): (JSC::WeakSet::visit): (JSC::WeakSet::shrink): Deleted. * heap/WeakSetInlines.h: (JSC::WeakSet::allocate): * inspector/InjectedScriptManager.cpp: * inspector/JSGlobalObjectInspectorController.cpp: * inspector/JSJavaScriptCallFrame.cpp: * inspector/ScriptDebugServer.cpp: * inspector/agents/InspectorDebuggerAgent.cpp: * interpreter/CachedCall.h: (JSC::CachedCall::CachedCall): * interpreter/Interpreter.cpp: (JSC::loadVarargs): (JSC::StackFrame::sourceID): Deleted. (JSC::StackFrame::sourceURL): Deleted. (JSC::StackFrame::functionName): Deleted. (JSC::StackFrame::computeLineAndColumn): Deleted. (JSC::StackFrame::toString): Deleted. * interpreter/Interpreter.h: (JSC::StackFrame::isNative): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): (JSC::AssemblyHelpers::emitAllocate): (JSC::AssemblyHelpers::emitAllocateJSCell): (JSC::AssemblyHelpers::emitAllocateJSObject): (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): * jit/GCAwareJITStubRoutine.cpp: (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine): * jit/JIT.cpp: (JSC::JIT::compileCTINativeCall): (JSC::JIT::link): * jit/JIT.h: (JSC::JIT::compileCTINativeCall): Deleted. * jit/JITExceptions.cpp: (JSC::genericUnwind): * jit/JITExceptions.h: * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOpcodes32_64.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emitWriteBarrier): * jit/JITThunks.cpp: * jit/JITThunks.h: * jsc.cpp: (functionDescribeArray): (main): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntExceptions.cpp: * llint/LLIntThunks.cpp: * llint/LLIntThunks.h: * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter.cpp: * llint/LowLevelInterpreter32_64.asm: * llint/LowLevelInterpreter64.asm: * parser/ModuleAnalyzer.cpp: * parser/NodeConstructors.h: * parser/Nodes.h: * profiler/ProfilerBytecode.cpp: * profiler/ProfilerBytecode.h: * profiler/ProfilerBytecodeSequence.cpp: * runtime/ArrayConventions.h: (JSC::indexingHeaderForArrayStorage): (JSC::baseIndexingHeaderForArrayStorage): (JSC::indexingHeaderForArray): Deleted. (JSC::baseIndexingHeaderForArray): Deleted. * runtime/ArrayPrototype.cpp: (JSC::arrayProtoFuncSplice): (JSC::concatAppendOne): (JSC::arrayProtoPrivateFuncConcatMemcpy): * runtime/ArrayStorage.h: (JSC::ArrayStorage::vectorLength): (JSC::ArrayStorage::totalSizeFor): (JSC::ArrayStorage::totalSize): (JSC::ArrayStorage::availableVectorLength): (JSC::ArrayStorage::optimalVectorLength): (JSC::ArrayStorage::sizeFor): Deleted. * runtime/AuxiliaryBarrier.h: Added. (JSC::AuxiliaryBarrier::AuxiliaryBarrier): (JSC::AuxiliaryBarrier::clear): (JSC::AuxiliaryBarrier::get): (JSC::AuxiliaryBarrier::slot): (JSC::AuxiliaryBarrier::operator bool): (JSC::AuxiliaryBarrier::setWithoutBarrier): * runtime/AuxiliaryBarrierInlines.h: Added. (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier): (JSC::AuxiliaryBarrier<T>::set): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::availableContiguousVectorLength): (JSC::Butterfly::optimalContiguousVectorLength): (JSC::Butterfly::createUninitialized): (JSC::Butterfly::growArrayRight): * runtime/ClonedArguments.cpp: (JSC::ClonedArguments::createEmpty): * runtime/CommonSlowPathsExceptions.cpp: * runtime/CommonSlowPathsExceptions.h: * runtime/DataView.cpp: * runtime/DirectArguments.h: * runtime/ECMAScriptSpecInternalFunctions.cpp: * runtime/Error.cpp: * runtime/Error.h: * runtime/ErrorInstance.cpp: * runtime/ErrorInstance.h: * runtime/Exception.cpp: * runtime/Exception.h: * runtime/GeneratorFrame.cpp: * runtime/GeneratorPrototype.cpp: * runtime/InternalFunction.cpp: (JSC::InternalFunction::InternalFunction): * runtime/IntlCollator.cpp: * runtime/IntlCollatorConstructor.cpp: * runtime/IntlCollatorPrototype.cpp: * runtime/IntlDateTimeFormat.cpp: * runtime/IntlDateTimeFormatConstructor.cpp: * runtime/IntlDateTimeFormatPrototype.cpp: * runtime/IntlNumberFormat.cpp: * runtime/IntlNumberFormatConstructor.cpp: * runtime/IntlNumberFormatPrototype.cpp: * runtime/IntlObject.cpp: * runtime/IteratorPrototype.cpp: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitialized): (JSC::JSArray::setLengthWritable): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLengthWithArrayStorage): (JSC::JSArray::appendMemcpy): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithArrayStorage): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::createContiguousArrayButterfly): (JSC::createArrayButterfly): (JSC::JSArray::create): (JSC::JSArray::tryCreateUninitialized): Deleted. * runtime/JSArrayBufferView.h: * runtime/JSCInlines.h: * runtime/JSCJSValue.cpp: (JSC::JSValue::dumpInContextAssumingStructure): * runtime/JSCallee.cpp: (JSC::JSCallee::JSCallee): * runtime/JSCell.cpp: (JSC::JSCell::estimatedSize): * runtime/JSCell.h: (JSC::JSCell::cellStateOffset): Deleted. * runtime/JSCellInlines.h: (JSC::ExecState::vm): (JSC::JSCell::classInfo): (JSC::JSCell::callDestructor): (JSC::JSCell::vm): Deleted. * runtime/JSFunction.cpp: (JSC::JSFunction::create): (JSC::JSFunction::allocateAndInitializeRareData): (JSC::JSFunction::initializeRareData): (JSC::JSFunction::getOwnPropertySlot): (JSC::JSFunction::put): (JSC::JSFunction::deleteProperty): (JSC::JSFunction::defineOwnProperty): (JSC::JSFunction::setFunctionName): (JSC::JSFunction::reifyLength): (JSC::JSFunction::reifyName): (JSC::JSFunction::reifyLazyPropertyIfNeeded): (JSC::JSFunction::reifyBoundNameIfNeeded): * runtime/JSFunction.h: * runtime/JSFunctionInlines.h: (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint): (JSC::JSFunction::JSFunction): * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory): * runtime/JSInternalPromise.cpp: * runtime/JSInternalPromiseConstructor.cpp: * runtime/JSInternalPromiseDeferred.cpp: * runtime/JSInternalPromisePrototype.cpp: * runtime/JSJob.cpp: * runtime/JSMapIterator.cpp: * runtime/JSModuleNamespaceObject.cpp: * runtime/JSModuleRecord.cpp: * runtime/JSObject.cpp: (JSC::JSObject::visitButterfly): (JSC::JSObject::notifyPresenceOfIndexedAccessors): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createInitialUndecided): (JSC::JSObject::createInitialInt32): (JSC::JSObject::createInitialDouble): (JSC::JSObject::createInitialContiguous): (JSC::JSObject::createArrayStorage): (JSC::JSObject::createInitialArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::putByIndexBeyondVectorLength): (JSC::JSObject::putDirectIndexBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::increaseVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::growOutOfLineStorage): (JSC::JSObject::copyButterfly): Deleted. (JSC::JSObject::copyBackingStore): Deleted. * runtime/JSObject.h: (JSC::JSObject::globalObject): (JSC::JSObject::putDirectInternal): (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted. * runtime/JSObjectInlines.h: * runtime/JSPromise.cpp: * runtime/JSPromiseConstructor.cpp: * runtime/JSPromiseDeferred.cpp: * runtime/JSPromisePrototype.cpp: * runtime/JSPropertyNameIterator.cpp: * runtime/JSScope.cpp: (JSC::JSScope::resolve): * runtime/JSScope.h: (JSC::JSScope::globalObject): (JSC::JSScope::vm): Deleted. * runtime/JSSetIterator.cpp: * runtime/JSStringIterator.cpp: * runtime/JSTemplateRegistryKey.cpp: * runtime/JSTypedArrayViewConstructor.cpp: * runtime/JSTypedArrayViewPrototype.cpp: * runtime/JSWeakMap.cpp: * runtime/JSWeakSet.cpp: * runtime/MapConstructor.cpp: * runtime/MapIteratorPrototype.cpp: * runtime/MapPrototype.cpp: * runtime/NativeErrorConstructor.cpp: * runtime/NativeStdFunctionCell.cpp: * runtime/Operations.h: (JSC::scribbleFreeCells): (JSC::scribble): * runtime/Options.h: * runtime/PropertyTable.cpp: * runtime/ProxyConstructor.cpp: * runtime/ProxyObject.cpp: * runtime/ProxyRevoke.cpp: * runtime/RegExp.cpp: (JSC::RegExp::match): (JSC::RegExp::matchConcurrently): (JSC::RegExp::matchCompareWithInterpreter): * runtime/RegExp.h: * runtime/RegExpConstructor.h: * runtime/RegExpInlines.h: (JSC::RegExp::matchInline): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): (JSC::createRegExpMatchesArray): * runtime/RegExpPrototype.cpp: (JSC::genericSplit): * runtime/RuntimeType.cpp: * runtime/SamplingProfiler.cpp: (JSC::SamplingProfiler::processUnverifiedStackTraces): * runtime/SetConstructor.cpp: * runtime/SetIteratorPrototype.cpp: * runtime/SetPrototype.cpp: * runtime/StackFrame.cpp: Added. (JSC::StackFrame::sourceID): (JSC::StackFrame::sourceURL): (JSC::StackFrame::functionName): (JSC::StackFrame::computeLineAndColumn): (JSC::StackFrame::toString): * runtime/StackFrame.h: Added. (JSC::StackFrame::isNative): * runtime/StringConstructor.cpp: * runtime/StringIteratorPrototype.cpp: * runtime/StructureInlines.h: (JSC::Structure::propertyTable): * runtime/TemplateRegistry.cpp: * runtime/TestRunnerUtils.cpp: (JSC::finalizeStatsAtEndOfTesting): * runtime/TestRunnerUtils.h: * runtime/TypeProfilerLog.cpp: * runtime/TypeSet.cpp: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::ensureStackCapacityForCLoop): (JSC::VM::isSafeToRecurseSoftCLoop): * runtime/VM.h: * runtime/VMEntryScope.h: * runtime/VMInlines.h: (JSC::VM::ensureStackCapacityFor): (JSC::VM::isSafeToRecurseSoft): * runtime/WeakMapConstructor.cpp: * runtime/WeakMapData.cpp: * runtime/WeakMapPrototype.cpp: * runtime/WeakSetConstructor.cpp: * runtime/WeakSetPrototype.cpp: * testRegExp.cpp: (testOneRegExp): * tools/JSDollarVM.cpp: * tools/JSDollarVMPrototype.cpp: (JSC::JSDollarVMPrototype::isInObjectSpace): Source/WebCore: No new tests because no new WebCore behavior. Just rewiring #includes. * ForwardingHeaders/heap/HeapInlines.h: Added. * ForwardingHeaders/interpreter/Interpreter.h: Removed. * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added. * Modules/indexeddb/IDBCursorWithValue.cpp: * Modules/indexeddb/client/TransactionOperation.cpp: * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp: * Modules/indexeddb/server/UniqueIDBDatabase.cpp: * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp: * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp: * bindings/js/JSClientRectCustom.cpp: * bindings/js/JSDOMBinding.cpp: * bindings/js/JSDOMBinding.h: * bindings/js/JSDeviceMotionEventCustom.cpp: * bindings/js/JSDeviceOrientationEventCustom.cpp: * bindings/js/JSErrorEventCustom.cpp: * bindings/js/JSIDBCursorWithValueCustom.cpp: * bindings/js/JSIDBIndexCustom.cpp: * bindings/js/JSPopStateEventCustom.cpp: * bindings/js/JSWebGL2RenderingContextCustom.cpp: * bindings/js/JSWorkerGlobalScopeCustom.cpp: * bindings/js/WorkerScriptController.cpp: * contentextensions/ContentExtensionParser.cpp: * dom/ErrorEvent.cpp: * html/HTMLCanvasElement.cpp: * html/MediaDocument.cpp: * inspector/CommandLineAPIModule.cpp: * loader/EmptyClients.cpp: * page/CaptionUserPreferences.cpp: * page/Frame.cpp: * page/PageGroup.cpp: * page/UserContentController.cpp: * platform/mock/mediasource/MockBox.cpp: * testing/GCObservation.cpp: Source/WebKit2: Just rewiring some #includes. * UIProcess/ViewGestureController.cpp: * UIProcess/WebPageProxy.cpp: * UIProcess/WebProcessPool.cpp: * UIProcess/WebProcessProxy.cpp: * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp: * WebProcess/Plugins/Netscape/JSNPObject.cpp: Source/WTF: I needed tryFastAlignedMalloc() so I added it. * wtf/FastMalloc.cpp: (WTF::tryFastAlignedMalloc): * wtf/FastMalloc.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): (WTF::ParkingLot::forEach): Deleted. * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::forEach): * wtf/ScopedLambda.h: (WTF::scopedLambdaRef): * wtf/SentinelLinkedList.h: (WTF::SentinelLinkedList::forEach): (WTF::RawNode>::takeFrom): * wtf/SimpleStats.h: (WTF::SimpleStats::operator bool): (WTF::SimpleStats::operator!): Deleted. Tools: * DumpRenderTree/TestRunner.cpp: * DumpRenderTree/mac/DumpRenderTree.mm: (DumpRenderTreeMain): * Scripts/run-jsc-stress-tests: * TestWebKitAPI/Tests/WTF/Vector.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/179778@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
unparkOneImpl(address, scopedLambdaRef<intptr_t(UnparkResult)>(callback));
ScopedLambda should have a lifetime story that makes sense to the compiler https://bugs.webkit.org/show_bug.cgi?id=158118 Reviewed by Mark Lam. Source/WTF: Prior to this change, there were two lifetime bugs in ScopedLambda: - scopedLambda(Functor&&) would bind Functor to const lambda&, so the resulting ScopedLambdaFunctor would hold a reference to the original lambda. This would have surprising behavior; for example it meant that this code was wrong: auto l = scopedLambda<things>([&] ...); The solution is to have explicit copy/move versions of scopedLambda() rather than rely on perfect forwarding. - ScopedLambdaFunctor did not override its copy or move operations, so if the compiler did not RVO scopedLambda(), it would return a ScopedLambdaFunctor whose m_arg points to a dead temporary ScopedLambdaFunctor instance. The solution is to have explicit copy/move constructors and operators, which preserve the invariant that ScopedLambda::m_arg points to this. One nice side-effect of all of these constructors and operators being explicit is that we can rely on WTFMove's excellent assertions, which helped catch the first issue. This reverts ParkingLot to use ScopedLambda again. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): * wtf/ScopedLambda.h: (WTF::scopedLambda): Tools: Added a test case. This test crashes before the fix and now it passes. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/ScopedLambda.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/176236@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@201433 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-05-26 21:58:42 +00:00
}
JSC should support SharedArrayBuffer https://bugs.webkit.org/show_bug.cgi?id=163986 Reviewed by Keith Miller. JSTests: This adds our own test for the various corner cases of SharedArrayBuffer. This test is meant to check all of the things that don't require concurrency. * stress/SharedArrayBuffer.js: Added. (checkAtomics): (shouldFail): (Symbol): (runAtomic): Source/JavaScriptCore: This implements https://tc39.github.io/ecmascript_sharedmem/shmem.html. There is now a new SharedArrayBuffer type. In the JS runtime, which includes typed array types, the SharedArrayBuffer is a drop-in replacement for ArrayBuffer, even though they are distinct types (new SharedArrayBuffer() instanceof ArrayBuffer == false and vice versa). The DOM will not recognize SharedArrayBuffer, or any typed array that wraps it, to ensure safety. This matches what other browsers intend to do, see https://github.com/tc39/ecmascript_sharedmem/issues/38. API is provided for the DOM to opt into SharedArrayBuffer. One notable place is postMessage, which will share the SharedArrayBuffer's underlying data storage with other workers. This creates a pool of shared memory that the workers can use to talk to each other. There is also an Atomics object in global scope, which exposes sequentially consistent atomic operations: add, and, compareExchange, exchange, load, or, store, sub, and xor. Additionally it exposes a Atomics.isLockFree utility, which takes a byte amount and returns true or false. Also there is Atomics.wake/wait, which neatly map to ParkingLot. Accesses to typed arrays that wrap SharedArrayBuffer are optimized by JSC the same way as always. I believe that DFG and B3 already obey the following memory model, which I believe is a bit weaker than Cambridge and a bit stronger than what is being proposed for SharedArrayBuffer. To predict a program's behavior under the B3 memory model, imagine the space of all possible programs that would result from running an optimizer that adversarially follows B3's transformation rules. B3 transformations are correct if the newly created program is equivalent to the old one, assuming that any opaque effect in IR (like the reads and writes of a patchpoint/call/fence) could perform any load/store that satisfies the B3::Effects summary. Opaque effects are a way of describing an infinite set of programs: any program that only does the effects summarized in B3::Effects belongs to the set. For example, this prevents motion of operations across fences since fences are summarized as opaque effects that could read or write memory. This rule alone is not enough, because it leaves the door open for turning an atomic operation (like a load) into a non-atomic one (like a load followed by a store of the same value back to the same location or multiple loads). This is not an optimization that either our compiler or the CPU would want to do. One way to think of what exactly is forbidden is that B3 transformations that mess with memory accesses can only reorder them or remove them. This means that for any execution of the untransformed program, the corresponding execution of the transformed program (i.e. with the same input arguments and the same programs filled in for the opaque effects) must have the same loads and stores, with some removed and some reordered. This is a fairly simple mental model that B3 and DFG already follow and it's based on existing abstractions for the infinite set of programs inside an opaque effect (DFG's AbstractHeaps and B3's Effects). This patch makes all atomics operations intrinsic, but the DFG doesn't know about any of them yet. That's covered by bug 164108. This ought to be perf-neutral, but I am still running tests to confirm this. I'm also still writing new tests to cover all of the Atomics functionality and the behavior of SAB objects. * API/JSTypedArray.cpp: (JSObjectGetTypedArrayBytesPtr): (JSObjectGetTypedArrayBuffer): (JSObjectMakeArrayBufferWithBytesNoCopy): * API/tests/CompareAndSwapTest.cpp: (Bitmap::concurrentTestAndSet): * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * dfg/DFGDesiredWatchpoints.cpp: (JSC::DFG::ArrayBufferViewWatchpointAdaptor::add): * heap/Heap.cpp: (JSC::Heap::reportExtraMemoryVisited): (JSC::Heap::reportExternalMemoryVisited): * jsc.cpp: (functionTransferArrayBuffer): * runtime/ArrayBuffer.cpp: (JSC::SharedArrayBufferContents::SharedArrayBufferContents): (JSC::SharedArrayBufferContents::~SharedArrayBufferContents): (JSC::ArrayBufferContents::ArrayBufferContents): (JSC::ArrayBufferContents::operator=): (JSC::ArrayBufferContents::~ArrayBufferContents): (JSC::ArrayBufferContents::clear): (JSC::ArrayBufferContents::destroy): (JSC::ArrayBufferContents::reset): (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBufferContents::makeShared): (JSC::ArrayBufferContents::transferTo): (JSC::ArrayBufferContents::copyTo): (JSC::ArrayBufferContents::shareWith): (JSC::ArrayBuffer::create): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): (JSC::ArrayBuffer::createUninitialized): (JSC::ArrayBuffer::tryCreateUninitialized): (JSC::ArrayBuffer::createInternal): (JSC::ArrayBuffer::ArrayBuffer): (JSC::ArrayBuffer::slice): (JSC::ArrayBuffer::sliceImpl): (JSC::ArrayBuffer::makeShared): (JSC::ArrayBuffer::setSharingMode): (JSC::ArrayBuffer::transferTo): (JSC::ArrayBuffer::transfer): Deleted. * runtime/ArrayBuffer.h: (JSC::arrayBufferSharingModeName): (JSC::SharedArrayBufferContents::data): (JSC::ArrayBufferContents::data): (JSC::ArrayBufferContents::sizeInBytes): (JSC::ArrayBufferContents::isShared): (JSC::ArrayBuffer::sharingMode): (JSC::ArrayBuffer::isShared): (JSC::ArrayBuffer::gcSizeEstimateInBytes): (JSC::arrayBufferDestructorNull): Deleted. (JSC::arrayBufferDestructorDefault): Deleted. (JSC::ArrayBufferContents::ArrayBufferContents): Deleted. (JSC::ArrayBufferContents::transfer): Deleted. (JSC::ArrayBufferContents::copyTo): Deleted. (JSC::ArrayBuffer::create): Deleted. (JSC::ArrayBuffer::createAdopted): Deleted. (JSC::ArrayBuffer::createFromBytes): Deleted. (JSC::ArrayBuffer::tryCreate): Deleted. (JSC::ArrayBuffer::createUninitialized): Deleted. (JSC::ArrayBuffer::tryCreateUninitialized): Deleted. (JSC::ArrayBuffer::createInternal): Deleted. (JSC::ArrayBuffer::ArrayBuffer): Deleted. (JSC::ArrayBuffer::slice): Deleted. (JSC::ArrayBuffer::sliceImpl): Deleted. (JSC::ArrayBufferContents::tryAllocate): Deleted. (JSC::ArrayBufferContents::~ArrayBufferContents): Deleted. * runtime/ArrayBufferSharingMode.h: Added. * runtime/ArrayBufferView.h: (JSC::ArrayBufferView::possiblySharedBuffer): (JSC::ArrayBufferView::unsharedBuffer): (JSC::ArrayBufferView::isShared): (JSC::ArrayBufferView::buffer): Deleted. * runtime/AtomicsObject.cpp: Added. (JSC::AtomicsObject::AtomicsObject): (JSC::AtomicsObject::create): (JSC::AtomicsObject::createStructure): (JSC::AtomicsObject::finishCreation): (JSC::atomicsFuncAdd): (JSC::atomicsFuncAnd): (JSC::atomicsFuncCompareExchange): (JSC::atomicsFuncExchange): (JSC::atomicsFuncIsLockFree): (JSC::atomicsFuncLoad): (JSC::atomicsFuncOr): (JSC::atomicsFuncStore): (JSC::atomicsFuncSub): (JSC::atomicsFuncWait): (JSC::atomicsFuncWake): (JSC::atomicsFuncXor): * runtime/AtomicsObject.h: Added. * runtime/CommonIdentifiers.h: * runtime/DataView.cpp: (JSC::DataView::wrap): * runtime/GenericTypedArrayViewInlines.h: (JSC::GenericTypedArrayView<Adaptor>::subarray): * runtime/Intrinsic.h: * runtime/JSArrayBuffer.cpp: (JSC::JSArrayBuffer::finishCreation): (JSC::JSArrayBuffer::isShared): (JSC::JSArrayBuffer::sharingMode): * runtime/JSArrayBuffer.h: (JSC::toPossiblySharedArrayBuffer): (JSC::toUnsharedArrayBuffer): (JSC::JSArrayBuffer::toWrapped): (JSC::toArrayBuffer): Deleted. * runtime/JSArrayBufferConstructor.cpp: (JSC::JSArrayBufferConstructor::JSArrayBufferConstructor): (JSC::JSArrayBufferConstructor::finishCreation): (JSC::JSArrayBufferConstructor::create): (JSC::constructArrayBuffer): * runtime/JSArrayBufferConstructor.h: (JSC::JSArrayBufferConstructor::sharingMode): * runtime/JSArrayBufferPrototype.cpp: (JSC::arrayBufferProtoFuncSlice): (JSC::JSArrayBufferPrototype::JSArrayBufferPrototype): (JSC::JSArrayBufferPrototype::finishCreation): (JSC::JSArrayBufferPrototype::create): * runtime/JSArrayBufferPrototype.h: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::finishCreation): (JSC::JSArrayBufferView::visitChildren): (JSC::JSArrayBufferView::unsharedBuffer): (JSC::JSArrayBufferView::unsharedJSBuffer): (JSC::JSArrayBufferView::possiblySharedJSBuffer): (JSC::JSArrayBufferView::neuter): (JSC::JSArrayBufferView::toWrapped): Deleted. * runtime/JSArrayBufferView.h: (JSC::JSArrayBufferView::jsBuffer): Deleted. * runtime/JSArrayBufferViewInlines.h: (JSC::JSArrayBufferView::isShared): (JSC::JSArrayBufferView::possiblySharedBuffer): (JSC::JSArrayBufferView::possiblySharedImpl): (JSC::JSArrayBufferView::unsharedImpl): (JSC::JSArrayBufferView::byteOffset): (JSC::JSArrayBufferView::toWrapped): (JSC::JSArrayBufferView::buffer): Deleted. (JSC::JSArrayBufferView::impl): Deleted. (JSC::JSArrayBufferView::neuter): Deleted. * runtime/JSDataView.cpp: (JSC::JSDataView::possiblySharedTypedImpl): (JSC::JSDataView::unsharedTypedImpl): (JSC::JSDataView::getTypedArrayImpl): (JSC::JSDataView::typedImpl): Deleted. * runtime/JSDataView.h: (JSC::JSDataView::possiblySharedBuffer): (JSC::JSDataView::unsharedBuffer): (JSC::JSDataView::buffer): Deleted. * runtime/JSDataViewPrototype.cpp: (JSC::dataViewProtoGetterBuffer): * runtime/JSGenericTypedArrayView.h: (JSC::toPossiblySharedNativeTypedView): (JSC::toUnsharedNativeTypedView): (JSC::JSGenericTypedArrayView<Adaptor>::toWrapped): (JSC::JSGenericTypedArrayView::typedImpl): Deleted. (JSC::toNativeTypedView): Deleted. * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::create): (JSC::JSGenericTypedArrayView<Adaptor>::possiblySharedTypedImpl): (JSC::JSGenericTypedArrayView<Adaptor>::unsharedTypedImpl): (JSC::JSGenericTypedArrayView<Adaptor>::getTypedArrayImpl): * runtime/JSGenericTypedArrayViewPrototypeFunctions.h: (JSC::genericTypedArrayViewProtoGetterFuncBuffer): (JSC::genericTypedArrayViewPrivateFuncSubarrayCreate): * runtime/JSGlobalObject.cpp: (JSC::createAtomicsProperty): (JSC::JSGlobalObject::init): (JSC::JSGlobalObject::visitChildren): * runtime/JSGlobalObject.h: (JSC::JSGlobalObject::arrayBufferPrototype): (JSC::JSGlobalObject::arrayBufferStructure): * runtime/MathObject.cpp: * runtime/RuntimeFlags.h: * runtime/SimpleTypedArrayController.cpp: (JSC::SimpleTypedArrayController::toJS): * runtime/TypedArrayType.h: (JSC::typedArrayTypeForType): Source/WebCore: New tests added in the LayoutTests/workers/sab directory. This teaches WebCore that a typed array could be shared or not. By default, WebCore will reject shared typed arrays as if they were not typed arrays. This ensures that we don't get race conditions in code that can't handle it. If you postMessage a SharedArrayBuffer or something that wraps it, you will send the shared memory to the other worker. * Modules/encryptedmedia/CDMSessionClearKey.cpp: (WebCore::CDMSessionClearKey::cachedKeyForKeyID): * Modules/fetch/FetchBody.cpp: (WebCore::FetchBody::extract): * Modules/mediastream/RTCDataChannel.cpp: (WebCore::RTCDataChannel::send): * Modules/webaudio/AudioBuffer.cpp: (WebCore::AudioBuffer::getChannelData): * Modules/websockets/WebSocket.cpp: (WebCore::WebSocket::send): * bindings/js/JSBlobCustom.cpp: (WebCore::constructJSBlob): * bindings/js/JSCryptoAlgorithmDictionary.cpp: (WebCore::createRsaKeyGenParams): * bindings/js/JSCryptoCustom.cpp: (WebCore::JSCrypto::getRandomValues): * bindings/js/JSCryptoOperationData.cpp: (WebCore::cryptoOperationDataFromJSValue): * bindings/js/JSDOMBinding.h: (WebCore::toJS): (WebCore::toPossiblySharedArrayBufferView): (WebCore::toUnsharedArrayBufferView): (WebCore::toPossiblySharedInt8Array): (WebCore::toPossiblySharedInt16Array): (WebCore::toPossiblySharedInt32Array): (WebCore::toPossiblySharedUint8Array): (WebCore::toPossiblySharedUint8ClampedArray): (WebCore::toPossiblySharedUint16Array): (WebCore::toPossiblySharedUint32Array): (WebCore::toPossiblySharedFloat32Array): (WebCore::toPossiblySharedFloat64Array): (WebCore::toUnsharedInt8Array): (WebCore::toUnsharedInt16Array): (WebCore::toUnsharedInt32Array): (WebCore::toUnsharedUint8Array): (WebCore::toUnsharedUint8ClampedArray): (WebCore::toUnsharedUint16Array): (WebCore::toUnsharedUint32Array): (WebCore::toUnsharedFloat32Array): (WebCore::toUnsharedFloat64Array): (WebCore::toArrayBufferView): Deleted. (WebCore::toInt8Array): Deleted. (WebCore::toInt16Array): Deleted. (WebCore::toInt32Array): Deleted. (WebCore::toUint8Array): Deleted. (WebCore::toUint8ClampedArray): Deleted. (WebCore::toUint16Array): Deleted. (WebCore::toUint32Array): Deleted. (WebCore::toFloat32Array): Deleted. (WebCore::toFloat64Array): Deleted. * bindings/js/JSDataCueCustom.cpp: (WebCore::constructJSDataCue): * bindings/js/JSDictionary.cpp: (WebCore::JSDictionary::convertValue): * bindings/js/JSFileCustom.cpp: (WebCore::constructJSFile): * bindings/js/JSMessagePortCustom.cpp: (WebCore::extractTransferables): * bindings/js/JSWebGLRenderingContextBaseCustom.cpp: (WebCore::dataFunctionf): (WebCore::dataFunctioni): (WebCore::dataFunctionMatrix): * bindings/js/JSXMLHttpRequestCustom.cpp: (WebCore::JSXMLHttpRequest::send): * bindings/js/SerializedScriptValue.cpp: (WebCore::CloneSerializer::dumpArrayBufferView): (WebCore::CloneSerializer::dumpIfTerminal): (WebCore::CloneDeserializer::readArrayBufferView): (WebCore::CloneDeserializer::readTerminal): (WebCore::SerializedScriptValue::transferArrayBuffers): * bindings/js/StructuredClone.cpp: (WebCore::structuredCloneArrayBuffer): (WebCore::structuredCloneArrayBufferView): * bindings/scripts/CodeGeneratorJS.pm: (JSValueToNative): * css/FontFace.cpp: (WebCore::FontFace::create): * html/canvas/WebGL2RenderingContext.cpp: (WebCore::WebGL2RenderingContext::bufferData): (WebCore::WebGL2RenderingContext::bufferSubData): * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp: (WebCore::MediaPlayerPrivateAVFoundation::extractKeyURIKeyIDAndCertificateFromInitData): Source/WebKit/mac: Support the RuntimeFlag. * WebView/WebPreferencesPrivate.h: Source/WebKit/win: Support the RuntimeFlag. * Interfaces/IWebPreferencesPrivate.idl: Source/WebKit2: Adds some small things we need for SharedArrayBuffer. * UIProcess/API/C/WKPreferencesRefPrivate.h: * UIProcess/API/Cocoa/WKPreferencesPrivate.h: * WebProcess/InjectedBundle/InjectedBundle.cpp: (WebKit::InjectedBundle::createWebDataFromUint8Array): Source/WTF: Adds some small things we need for SharedArrayBuffer. * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeakRelaxed): (WTF::Atomic::exchangeAdd): (WTF::Atomic::exchangeAnd): (WTF::Atomic::exchangeOr): (WTF::Atomic::exchangeSub): (WTF::Atomic::exchangeXor): (WTF::atomicLoad): (WTF::atomicStore): (WTF::atomicCompareExchangeWeak): (WTF::atomicCompareExchangeWeakRelaxed): (WTF::atomicCompareExchangeStrong): (WTF::atomicExchangeAdd): (WTF::atomicExchangeAnd): (WTF::atomicExchangeOr): (WTF::atomicExchangeSub): (WTF::atomicExchangeXor): (WTF::atomicExchange): (WTF::Atomic::exchangeAndAdd): Deleted. (WTF::weakCompareAndSwap): Deleted. We need to be able to do atomics operations on naked pointers. We also need to be able to do all of the things that std::atomic does. This adds those things and renames weakCompareAndSwap to atomicCompareExchangeWeakRelaxed so that we're using consistent terminology. * wtf/Bitmap.h: (WTF::WordType>::concurrentTestAndSet): Renamed weakCompareAndSwap. (WTF::WordType>::concurrentTestAndClear): Renamed weakCompareAndSwap. * wtf/FastBitVector.h: (WTF::FastBitVector::atomicSetAndCheck): Renamed weakCompareAndSwap. * wtf/ParkingLot.cpp: (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkCount): * wtf/ParkingLot.h: Added unparkCount(), which lets you unpark some bounded number of threads and returns the number of threads unparked. This is just a modest extension of unparkAll(). unparkAll() now just calls unparkCount(ptr, UINT_MAX). Tools: Use the right kind of typed array API. * DumpRenderTree/TestRunner.cpp: (setAudioResultCallback): LayoutTests: Adding tests. This is a work in progress. * workers/sab: Added. * workers/sab/simple-worker-1.js: Added. (onmessage): * workers/sab/simple-worker-2.js: Added. (onmessage): * workers/sab/simple.html: Added. Canonical link: https://commits.webkit.org/181984@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208209 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-01 03:10:00 +00:00
WTF_EXPORT_PRIVATE static unsigned unparkCount(const void* address, unsigned count);
WTF::Lock should not suffer from the thundering herd https://bugs.webkit.org/show_bug.cgi?id=147947 Reviewed by Geoffrey Garen. Source/WTF: This changes Lock::unlockSlow() to use unparkOne() instead of unparkAll(). The problem with doing this is that it's not obvious after calling unparkOne() if there are any other threads that are still parked on the lock's queue. If we assume that there are and leave the hasParkedBit set, then future calls to unlock() will take the slow path. We don't want that if there aren't actually any threads parked. On the other hand, if we assume that there aren't any threads parked and clear the hasParkedBit, then if there actually were some threads parked, then they may never be awoken since future calls to unlock() won't take slow path and so won't call unparkOne(). In other words, we need a way to be very precise about when we clear the hasParkedBit and we need to do it in a race-free way: it can't be the case that we clear the bit just as some thread gets parked on the queue. A similar problem arises in futexes, and one of the solutions is to have a thread that acquires a lock after parking sets the hasParkedBit. This is what Rusty Russel's usersem does. It's a subtle algorithm. Also, it means that if a thread barges in before the unparked thread runs, then that barging thread will not know that there are threads parked. This could increase the severity of barging. Since ParkingLot is a user-level API, we don't have to worry about the kernel-user security issues and so we can expose callbacks while ParkingLot is holding its internal locks. This change does exactly that for unparkOne(). The new variant of unparkOne() will call a user function while the queue from which we are unparking is locked. The callback is told basic stats about the queue: did we unpark a thread this time, and could there be more threads to unpark in the future. The callback runs while it's impossible for the queue state to change, since the ParkingLot's internal locks for the queue is held. This means that Lock::unlockSlow() can either clear, or leave, the hasParkedBit while releasing the lock inside the callback from unparkOne(). This takes care of the thundering herd problem while also reducing the greed that arises from barging threads. This required some careful reworking of the ParkingLot algorithm. The first thing I noticed was that the ThreadData::shouldPark flag was useless, since it's set exactly when ThreadData::address is non-null. Then I had to make sure that dequeue() could lazily create both hashtables and buckets, since the "callback is called while queue is locked" invariant requires that we didn't exit early due to the hashtable or bucket not being present. Note that all of this is done in such a way that the old unparkOne() and unparkAll() don't have to create any buckets, though they now may create the hashtable. We don't care as much about the hashtable being created by unpark since it's just such an unlikely scenario and it would only happen once. This change reduces the kernel CPU usage of WTF::Lock for the long critical section test by about 8x and makes it always perform as well as WTF::WordLock and WTF::Mutex for that benchmark. * benchmarks/LockSpeedTest.cpp: * wtf/Lock.cpp: (WTF::LockBase::unlockSlow): * wtf/Lock.h: (WTF::LockBase::isLocked): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: * wtf/WordLock.h: (WTF::WordLock::isLocked): (WTF::WordLock::isFullyReset): Tools: Add testing that checks that locks return to a pristine state after contention is over. * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::LockInspector::isFullyReset): (TestWebKitAPI::runLockTest): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/166072@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188374 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-13 03:51:25 +00:00
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
// Unparks every thread from the queue associated with the given address, which cannot be null.
WTF_EXPORT_PRIVATE static void unparkAll(const void* address);
// Locks the parking lot and walks all of the parked threads and the addresses they are waiting
// on. Threads that are on the same queue are guaranteed to be walked from first to last, but the
// queues may be randomly interleaved. For example, if the queue for address A1 has T1 and T2 and
// the queue for address A2 has T3 and T4, then you might see iteration orders like:
//
// A1,T1 A1,T2 A2,T3 A2,T4
// A2,T3 A2,T4 A1,T1 A1,T2
// A1,T1 A2,T3 A1,T2 A2,T4
// A1,T1 A2,T3 A2,T4 A1,T2
//
Replace all remaining uses of WTF::Mutex with WTF::Lock https://bugs.webkit.org/show_bug.cgi?id=148089 Reviewed by Geoffrey Garen. Source/WebCore: No new tests because no new behavior. * Modules/webaudio/AsyncAudioDecoder.cpp: (WebCore::AsyncAudioDecoder::AsyncAudioDecoder): (WebCore::AsyncAudioDecoder::runLoop): * Modules/webaudio/AsyncAudioDecoder.h: * Modules/webaudio/AudioContext.h: * Modules/webaudio/MediaStreamAudioSource.cpp: (WebCore::MediaStreamAudioSource::addAudioConsumer): (WebCore::MediaStreamAudioSource::removeAudioConsumer): (WebCore::MediaStreamAudioSource::setAudioFormat): (WebCore::MediaStreamAudioSource::consumeAudio): * Modules/webaudio/MediaStreamAudioSource.h: * Modules/webdatabase/Database.cpp: (WebCore::Database::close): (WebCore::Database::runTransaction): (WebCore::Database::inProgressTransactionCompleted): (WebCore::Database::hasPendingTransaction): * Modules/webdatabase/Database.h: * Modules/webdatabase/DatabaseTask.cpp: (WebCore::DatabaseTaskSynchronizer::taskCompleted): * Modules/webdatabase/DatabaseTask.h: * Modules/webdatabase/DatabaseThread.cpp: (WebCore::DatabaseThread::start): (WebCore::DatabaseThread::databaseThread): * Modules/webdatabase/DatabaseThread.h: * Modules/webdatabase/DatabaseTracker.cpp: (WebCore::DatabaseTracker::setDatabaseDirectoryPath): (WebCore::DatabaseTracker::canEstablishDatabase): (WebCore::DatabaseTracker::retryCanEstablishDatabase): (WebCore::DatabaseTracker::hasEntryForOrigin): (WebCore::DatabaseTracker::getMaxSizeForDatabase): (WebCore::DatabaseTracker::closeAllDatabases): (WebCore::DatabaseTracker::fullPathForDatabase): (WebCore::DatabaseTracker::origins): (WebCore::DatabaseTracker::databaseNamesForOrigin): (WebCore::DatabaseTracker::detailsForNameAndOrigin): (WebCore::DatabaseTracker::setDatabaseDetails): (WebCore::DatabaseTracker::doneCreatingDatabase): (WebCore::DatabaseTracker::addOpenDatabase): (WebCore::DatabaseTracker::removeOpenDatabase): (WebCore::DatabaseTracker::getOpenDatabases): (WebCore::DatabaseTracker::originLockFor): (WebCore::DatabaseTracker::quotaForOrigin): (WebCore::DatabaseTracker::setQuota): (WebCore::DatabaseTracker::deleteOrigin): (WebCore::DatabaseTracker::deleteDatabase): (WebCore::DatabaseTracker::deleteDatabaseFile): (WebCore::DatabaseTracker::removeDeletedOpenedDatabases): (WebCore::DatabaseTracker::deleteDatabaseFileIfEmpty): (WebCore::DatabaseTracker::openDatabaseMutex): (WebCore::DatabaseTracker::setClient): (WebCore::notificationMutex): (WebCore::DatabaseTracker::scheduleNotifyDatabaseChanged): (WebCore::DatabaseTracker::notifyDatabasesChanged): * Modules/webdatabase/DatabaseTracker.h: * Modules/webdatabase/OriginLock.h: * Modules/webdatabase/SQLCallbackWrapper.h: (WebCore::SQLCallbackWrapper::clear): (WebCore::SQLCallbackWrapper::unwrap): (WebCore::SQLCallbackWrapper::hasCallback): * Modules/webdatabase/SQLTransactionBackend.cpp: (WebCore::SQLTransactionBackend::doCleanup): (WebCore::SQLTransactionBackend::enqueueStatementBackend): (WebCore::SQLTransactionBackend::getNextStatement): * Modules/webdatabase/SQLTransactionBackend.h: * bindings/js/WorkerScriptController.cpp: (WebCore::WorkerScriptController::scheduleExecutionTermination): (WebCore::WorkerScriptController::isExecutionTerminating): * bindings/js/WorkerScriptController.h: * dom/default/PlatformMessagePortChannel.cpp: (WebCore::MessagePortChannel::postMessageToRemote): (WebCore::MessagePortChannel::tryGetMessageFromRemote): (WebCore::MessagePortChannel::isConnectedTo): (WebCore::MessagePortChannel::hasPendingActivity): (WebCore::MessagePortChannel::locallyEntangledPort): (WebCore::PlatformMessagePortChannel::setRemotePort): (WebCore::PlatformMessagePortChannel::entangledChannel): (WebCore::PlatformMessagePortChannel::closeInternal): * dom/default/PlatformMessagePortChannel.h: * loader/icon/IconDatabase.cpp: (WebCore::IconDatabase::removeAllIcons): (WebCore::IconDatabase::synchronousIconForPageURL): (WebCore::IconDatabase::synchronousNativeIconForPageURL): (WebCore::IconDatabase::synchronousIconURLForPageURL): (WebCore::IconDatabase::retainIconForPageURL): (WebCore::IconDatabase::performRetainIconForPageURL): (WebCore::IconDatabase::releaseIconForPageURL): (WebCore::IconDatabase::performReleaseIconForPageURL): (WebCore::IconDatabase::setIconDataForIconURL): (WebCore::IconDatabase::setIconURLForPageURL): (WebCore::IconDatabase::synchronousLoadDecisionForIconURL): (WebCore::IconDatabase::synchronousIconDataKnownForIconURL): (WebCore::IconDatabase::pageURLMappingCount): (WebCore::IconDatabase::retainedPageURLCount): (WebCore::IconDatabase::iconRecordCount): (WebCore::IconDatabase::iconRecordCountWithData): (WebCore::IconDatabase::wakeSyncThread): (WebCore::IconDatabase::scheduleOrDeferSyncTimer): (WebCore::IconDatabase::isOpenBesidesMainThreadCallbacks): (WebCore::IconDatabase::databasePath): (WebCore::IconDatabase::getOrCreatePageURLRecord): (WebCore::IconDatabase::iconDatabaseSyncThread): (WebCore::IconDatabase::performOpenInitialization): (WebCore::IconDatabase::performURLImport): (WebCore::IconDatabase::syncThreadMainLoop): (WebCore::IconDatabase::performPendingRetainAndReleaseOperations): (WebCore::IconDatabase::readFromDatabase): (WebCore::IconDatabase::writeToDatabase): (WebCore::IconDatabase::pruneUnretainedIcons): (WebCore::IconDatabase::cleanupSyncThread): * loader/icon/IconDatabase.h: * page/scrolling/ScrollingTree.cpp: (WebCore::ScrollingTree::shouldHandleWheelEventSynchronously): (WebCore::ScrollingTree::commitNewTreeState): (WebCore::ScrollingTree::setMainFramePinState): (WebCore::ScrollingTree::mainFrameScrollPosition): (WebCore::ScrollingTree::setMainFrameScrollPosition): (WebCore::ScrollingTree::isPointInNonFastScrollableRegion): (WebCore::ScrollingTree::isRubberBandInProgress): (WebCore::ScrollingTree::setMainFrameIsRubberBanding): (WebCore::ScrollingTree::isScrollSnapInProgress): (WebCore::ScrollingTree::setMainFrameIsScrollSnapping): (WebCore::ScrollingTree::setCanRubberBandState): (WebCore::ScrollingTree::rubberBandsAtLeft): (WebCore::ScrollingTree::rubberBandsAtRight): (WebCore::ScrollingTree::rubberBandsAtBottom): (WebCore::ScrollingTree::rubberBandsAtTop): (WebCore::ScrollingTree::setScrollPinningBehavior): (WebCore::ScrollingTree::scrollPinningBehavior): (WebCore::ScrollingTree::willWheelEventStartSwipeGesture): (WebCore::ScrollingTree::latchedNode): (WebCore::ScrollingTree::setLatchedNode): (WebCore::ScrollingTree::clearLatchedNode): * page/scrolling/ScrollingTree.h: * platform/MemoryPressureHandler.h: * platform/audio/HRTFDatabaseLoader.cpp: (WebCore::HRTFDatabaseLoader::loadAsynchronously): (WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion): * platform/audio/HRTFDatabaseLoader.h: * platform/cocoa/MemoryPressureHandlerCocoa.mm: (WebCore::MemoryPressureHandler::setReceivedMemoryPressure): (WebCore::MemoryPressureHandler::clearMemoryPressure): (WebCore::MemoryPressureHandler::shouldWaitForMemoryClearMessage): (WebCore::MemoryPressureHandler::respondToMemoryPressureIfNeeded): * platform/graphics/DisplayRefreshMonitor.cpp: (WebCore::DisplayRefreshMonitor::displayDidRefresh): * platform/graphics/DisplayRefreshMonitor.h: (WebCore::DisplayRefreshMonitor::setMonotonicAnimationStartTime): (WebCore::DisplayRefreshMonitor::mutex): * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.cpp: (WebCore::MediaPlayerPrivateAVFoundation::setDelayCallbacks): (WebCore::MediaPlayerPrivateAVFoundation::clearMainThreadPendingFlag): (WebCore::MediaPlayerPrivateAVFoundation::dispatchNotification): * platform/graphics/avfoundation/MediaPlayerPrivateAVFoundation.h: * platform/graphics/avfoundation/cf/MediaPlayerPrivateAVFoundationCF.cpp: (WebCore::AVFWrapper::callbackContext): (WebCore::AVFWrapper::~AVFWrapper): (WebCore::AVFWrapper::mapLock): (WebCore::AVFWrapper::addToMap): (WebCore::AVFWrapper::removeFromMap): (WebCore::AVFWrapper::periodicTimeObserverCallback): (WebCore::AVFWrapper::processNotification): (WebCore::AVFWrapper::loadPlayableCompletionCallback): (WebCore::AVFWrapper::loadMetadataCompletionCallback): (WebCore::AVFWrapper::seekCompletedCallback): (WebCore::AVFWrapper::processCue): (WebCore::AVFWrapper::legibleOutputCallback): (WebCore::AVFWrapper::processShouldWaitForLoadingOfResource): (WebCore::AVFWrapper::resourceLoaderShouldWaitForLoadingOfRequestedResource): * platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp: (WebCore::InbandTextTrackPrivateGStreamer::handleSample): (WebCore::InbandTextTrackPrivateGStreamer::notifyTrackOfSample): * platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h: * platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp: (WebCore::TrackPrivateBaseGStreamer::tagsChanged): (WebCore::TrackPrivateBaseGStreamer::notifyTrackOfTagsChanged): * platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h: * platform/graphics/mac/DisplayRefreshMonitorMac.cpp: (WebCore::DisplayRefreshMonitorMac::requestRefreshCallback): (WebCore::DisplayRefreshMonitorMac::displayLinkFired): * platform/graphics/win/MediaPlayerPrivateMediaFoundation.cpp: (WebCore::MediaPlayerPrivateMediaFoundation::addListener): (WebCore::MediaPlayerPrivateMediaFoundation::removeListener): (WebCore::MediaPlayerPrivateMediaFoundation::notifyDeleted): (WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::Invoke): (WebCore::MediaPlayerPrivateMediaFoundation::AsyncCallback::onMediaPlayerDeleted): * platform/graphics/win/MediaPlayerPrivateMediaFoundation.h: * platform/ios/LegacyTileCache.h: * platform/ios/LegacyTileCache.mm: (WebCore::LegacyTileCache::setTilesOpaque): (WebCore::LegacyTileCache::doLayoutTiles): (WebCore::LegacyTileCache::setCurrentScale): (WebCore::LegacyTileCache::commitScaleChange): (WebCore::LegacyTileCache::layoutTilesNow): (WebCore::LegacyTileCache::layoutTilesNowForRect): (WebCore::LegacyTileCache::removeAllNonVisibleTiles): (WebCore::LegacyTileCache::removeAllTiles): (WebCore::LegacyTileCache::removeForegroundTiles): (WebCore::LegacyTileCache::setContentReplacementImage): (WebCore::LegacyTileCache::contentReplacementImage): (WebCore::LegacyTileCache::tileCreationTimerFired): (WebCore::LegacyTileCache::setNeedsDisplayInRect): (WebCore::LegacyTileCache::updateTilingMode): (WebCore::LegacyTileCache::setTilingMode): (WebCore::LegacyTileCache::doPendingRepaints): (WebCore::LegacyTileCache::flushSavedDisplayRects): (WebCore::LegacyTileCache::prepareToDraw): * platform/ios/LegacyTileLayerPool.h: * platform/ios/LegacyTileLayerPool.mm: (WebCore::LegacyTileLayerPool::addLayer): (WebCore::LegacyTileLayerPool::takeLayerWithSize): (WebCore::LegacyTileLayerPool::setCapacity): (WebCore::LegacyTileLayerPool::prune): (WebCore::LegacyTileLayerPool::drain): * platform/network/curl/CurlDownload.cpp: (WebCore::CurlDownloadManager::add): (WebCore::CurlDownloadManager::remove): (WebCore::CurlDownloadManager::getActiveDownloadCount): (WebCore::CurlDownloadManager::getPendingDownloadCount): (WebCore::CurlDownloadManager::stopThreadIfIdle): (WebCore::CurlDownloadManager::updateHandleList): (WebCore::CurlDownload::~CurlDownload): (WebCore::CurlDownload::init): (WebCore::CurlDownload::getTempPath): (WebCore::CurlDownload::getUrl): (WebCore::CurlDownload::getResponse): (WebCore::CurlDownload::closeFile): (WebCore::CurlDownload::didReceiveHeader): (WebCore::CurlDownload::didReceiveData): (WebCore::CurlDownload::didFail): * platform/network/curl/CurlDownload.h: * platform/network/curl/ResourceHandleManager.cpp: (WebCore::cookieJarPath): (WebCore::sharedResourceMutex): (WebCore::curl_lock_callback): (WebCore::curl_unlock_callback): * platform/network/ios/QuickLook.mm: (WebCore::QLDirectoryAttributes): (qlPreviewConverterDictionaryMutex): (WebCore::addQLPreviewConverterWithFileForURL): (WebCore::qlPreviewConverterUTIForURL): (WebCore::removeQLPreviewConverterForURL): (WebCore::safeQLURLForDocumentURLAndResourceURL): * platform/sql/SQLiteDatabase.cpp: (WebCore::SQLiteDatabase::close): (WebCore::SQLiteDatabase::maximumSize): (WebCore::SQLiteDatabase::setMaximumSize): (WebCore::SQLiteDatabase::pageSize): (WebCore::SQLiteDatabase::freeSpaceSize): (WebCore::SQLiteDatabase::totalSize): (WebCore::SQLiteDatabase::runIncrementalVacuumCommand): (WebCore::SQLiteDatabase::setAuthorizer): * platform/sql/SQLiteDatabase.h: (WebCore::SQLiteDatabase::databaseMutex): * platform/sql/SQLiteStatement.cpp: (WebCore::SQLiteStatement::prepare): (WebCore::SQLiteStatement::step): * workers/WorkerThread.cpp: (WebCore::WorkerThread::start): (WebCore::WorkerThread::workerThread): (WebCore::WorkerThread::stop): * workers/WorkerThread.h: Source/WebKit: * Storage/StorageAreaSync.cpp: (WebCore::StorageAreaSync::syncTimerFired): (WebCore::StorageAreaSync::markImported): (WebCore::StorageAreaSync::blockUntilImportComplete): (WebCore::StorageAreaSync::performSync): * Storage/StorageAreaSync.h: * Storage/StorageTracker.cpp: (WebCore::StorageTracker::setDatabaseDirectoryPath): (WebCore::StorageTracker::finishedImportingOriginIdentifiers): (WebCore::StorageTracker::syncImportOriginIdentifiers): (WebCore::StorageTracker::syncFileSystemAndTrackerDatabase): (WebCore::StorageTracker::setOriginDetails): (WebCore::StorageTracker::syncSetOriginDetails): (WebCore::StorageTracker::origins): (WebCore::StorageTracker::deleteAllOrigins): (WebCore::StorageTracker::syncDeleteAllOrigins): (WebCore::StorageTracker::deleteOrigin): (WebCore::StorageTracker::syncDeleteOrigin): (WebCore::StorageTracker::canDeleteOrigin): (WebCore::StorageTracker::cancelDeletingOrigin): (WebCore::StorageTracker::diskUsageForOrigin): * Storage/StorageTracker.h: Source/WebKit/ios: * WebCoreSupport/WebFixedPositionContent.mm: (WebFixedPositionContentDataLock): (-[WebFixedPositionContent scrollOrZoomChanged:]): (-[WebFixedPositionContent overflowScrollPositionForLayer:changedTo:]): (-[WebFixedPositionContent setViewportConstrainedLayers:stickyContainerMap:]): (-[WebFixedPositionContent hasFixedOrStickyPositionLayers]): (-[WebFixedPositionContent minimumOffsetFromFixedPositionLayersToAnchorEdge:ofRect:inLayer:]): Source/WebKit/mac: * Storage/WebDatabaseManager.mm: (transactionBackgroundTaskIdentifierLock): (+[WebDatabaseManager startBackgroundTask]): (+[WebDatabaseManager endBackgroundTask]): * WebView/WebView.mm: (-[WebView _synchronizeCustomFixedPositionLayoutRect]): (-[WebView _setCustomFixedPositionLayoutRectInWebThread:synchronize:]): (-[WebView _setCustomFixedPositionLayoutRect:]): (-[WebView _fetchCustomFixedPositionLayoutRect:]): * WebView/WebViewData.h: Source/WebKit/win: * Plugins/PluginMainThreadScheduler.cpp: (WebCore::PluginMainThreadScheduler::scheduleCall): (WebCore::PluginMainThreadScheduler::registerPlugin): (WebCore::PluginMainThreadScheduler::unregisterPlugin): (WebCore::PluginMainThreadScheduler::dispatchCallsForPlugin): * Plugins/PluginMainThreadScheduler.h: * WebIconDatabase.cpp: (WebIconDatabase::didRemoveAllIcons): (WebIconDatabase::didImportIconURLForPageURL): (WebIconDatabase::deliverNotifications): * WebIconDatabase.h: * WebLocalizableStrings.cpp: (mainBundleLocStrings): (frameworkLocStringsMutex): (findCachedString): (cacheString): Source/WebKit2: * DatabaseProcess/DatabaseProcess.cpp: (WebKit::DatabaseProcess::postDatabaseTask): (WebKit::DatabaseProcess::performNextDatabaseTask): * DatabaseProcess/DatabaseProcess.h: * DatabaseProcess/IndexedDB/UniqueIDBDatabase.cpp: (WebKit::UniqueIDBDatabase::shutdown): (WebKit::UniqueIDBDatabase::postMainThreadTask): (WebKit::UniqueIDBDatabase::performNextMainThreadTask): (WebKit::UniqueIDBDatabase::postDatabaseTask): (WebKit::UniqueIDBDatabase::performNextDatabaseTask): * DatabaseProcess/IndexedDB/UniqueIDBDatabase.h: * Platform/IPC/Connection.cpp: (IPC::Connection::sendSyncMessage): (IPC::Connection::sendSyncMessageFromSecondaryThread): (IPC::Connection::waitForSyncReply): (IPC::Connection::processIncomingSyncReply): (IPC::Connection::connectionDidClose): * Platform/IPC/Connection.h: * Shared/CoordinatedGraphics/CoordinatedGraphicsScene.cpp: (WebKit::CoordinatedGraphicsScene::appendUpdate): * Shared/CoordinatedGraphics/CoordinatedGraphicsScene.h: * Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.cpp: (WebKit::ThreadedCompositor::createCompositingThread): (WebKit::ThreadedCompositor::runCompositingThread): (WebKit::ThreadedCompositor::terminateCompositingThread): * Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.h: * Shared/Network/CustomProtocols/Cocoa/CustomProtocolManagerCocoa.mm: (WebKit::CustomProtocolManager::addCustomProtocol): (WebKit::CustomProtocolManager::removeCustomProtocol): (WebKit::CustomProtocolManager::registerScheme): (WebKit::CustomProtocolManager::unregisterScheme): (WebKit::CustomProtocolManager::supportsScheme): (WebKit::CustomProtocolManager::protocolForID): * Shared/Network/CustomProtocols/CustomProtocolManager.h: * Shared/linux/SeccompFilters/SeccompBroker.cpp: * WebProcess/Plugins/PluginProcessConnectionManager.cpp: (WebKit::PluginProcessConnectionManager::getPluginProcessConnection): (WebKit::PluginProcessConnectionManager::removePluginProcessConnection): (WebKit::PluginProcessConnectionManager::pluginProcessCrashed): * WebProcess/Plugins/PluginProcessConnectionManager.h: * WebProcess/WebPage/EventDispatcher.cpp: (WebKit::EventDispatcher::addScrollingTreeForPage): (WebKit::EventDispatcher::removeScrollingTreeForPage): (WebKit::EventDispatcher::wheelEvent): * WebProcess/WebPage/EventDispatcher.h: * WebProcess/soup/WebKitSoupRequestInputStream.cpp: (webkitSoupRequestInputStreamReadAsync): (webkitSoupRequestInputStreamAddData): Source/WTF: This also beefs up and rationalizes the Condition API, so that it can deal with units of time other than just steady_clock. This makes it easier to port ThreadCondition and std::condition_variable code over to Condition. This patch does not take a position on what kind of time is best; from reading a lot of the uses in WebCore, it seems like our use of double to measure seconds is often nicer than the many different classes in std::chrono. Also added a Condition speed test, to make sure that all of this is a good idea. And indeed it is. The 1-producer/1-consumer scenario with a 100-element queue runs 36x faster using Lock/Condition than Mutex/ThreadCondition when you use the notifyOne()-per-enqueue style. It runs 58x faster with Lock/Condition when you use the notifyAll()-at-boundary style. Note that I have a bug open for making the notifyOne()-per-enqueue style even faster: https://bugs.webkit.org/show_bug.cgi?id=148090. Also, the 10-consumer/10-producer scenario with a 100-element queue runs 20x faster with Lock/Condition for notifyOne()-per-enqueue and 30x faster with notifyAll()-at-boundary. The only way to tweak the test to get Mutex/ThreadCondition to win is to have one producer, one consumer, a 1-element queue, and use the notifyOne()-per-enqueue style. In that case, one of the two threads is going to be waiting most of the time and the test basically measures wake-up latency and nothing else. Because Condition::wait() does a little bit more work than ThreadCondition::wait(), Mutex/ThreadCondition end up running 3% faster in this test case. But if you vary any of the parameters of the test, Mutex/ThreadCondition ends up losing - all it takes is more threads or a queue size of 5 or more. To my knowledge, we never do producer/consumer with a queue bounded to one element precisely because that approach is the least efficient regardless of locking algorithm. For example, neither WTF::MessageQueue nor DFG::Worklist have any bounds on their queue size. So, it seems that replacing all uses of system mutexes and condition variables with our own thing is a great idea. * benchmarks/LockSpeedTest.cpp: * benchmarks/ConditionSpeedTest.cpp: Added. * wtf/Condition.h: (WTF::Condition::Condition): (WTF::Condition::waitUntil): (WTF::Condition::waitFor): (WTF::Condition::wait): (WTF::Condition::waitUntilWallClockSeconds): (WTF::Condition::waitUntilMonotonicClockSeconds): (WTF::Condition::notifyOne): (WTF::Condition::notifyAll): (WTF::Condition::waitForSecondsImpl): (WTF::Condition::waitForImpl): (WTF::Condition::absoluteFromRelative): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): (WTF::MessageQueue<DataType>::append): (WTF::MessageQueue<DataType>::appendAndKill): (WTF::MessageQueue<DataType>::appendAndCheckEmpty): (WTF::MessageQueue<DataType>::prepend): (WTF::MessageQueue<DataType>::removeIf): (WTF::MessageQueue<DataType>::isEmpty): (WTF::MessageQueue<DataType>::kill): (WTF::MessageQueue<DataType>::killed): * wtf/ParallelJobsGeneric.cpp: (WTF::ParallelEnvironment::ThreadPrivate::execute): (WTF::ParallelEnvironment::ThreadPrivate::waitForFinish): (WTF::ParallelEnvironment::ThreadPrivate::workerThread): * wtf/ParallelJobsGeneric.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionally): * wtf/ParkingLot.h: (WTF::ParkingLot::compareAndPark): * wtf/ThreadingPthreads.cpp: (WTF::initializeThreading): * wtf/ThreadingWin.cpp: (WTF::initializeThreading): * wtf/dtoa.cpp: (WTF::pow5mult): * wtf/dtoa.h: Tools: * DumpRenderTree/JavaScriptThreading.cpp: (javaScriptThreadsMutex): (runJavaScriptThread): (startJavaScriptThreads): (stopJavaScriptThreads): * TestWebKitAPI/Tests/WTF/Condition.cpp: Fixed a bug in the test that I found from turning the test into a benchmark. * TestWebKitAPI/Tests/WTF/WorkQueue.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/glib/WorkQueueGLib.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/166252@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188594 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-18 19:31:28 +00:00
// As well as many other possible interleavings that all have T1 before T2 and T3 before T4 but are
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
// otherwise unconstrained. This method is useful primarily for debugging. It's also used by unit
// tests.
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression https://bugs.webkit.org/show_bug.cgi?id=160125 Reviewed by Geoffrey Garen and Keith Miller. JSTests: Most of the things I did properly covered by existing tests, but I found some simple cases of unshifting that had sketchy coverage. * stress/array-storage-array-unshift.js: Added. * stress/contiguous-array-unshift.js: Added. * stress/double-array-unshift.js: Added. * stress/int32-array-unshift.js: Added. Source/bmalloc: I needed to tryMemalign, so I added such a thing. * bmalloc/Allocator.cpp: (bmalloc::Allocator::allocate): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): * bmalloc/Allocator.h: * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): * bmalloc/bmalloc.h: (bmalloc::api::tryMemalign): Source/JavaScriptCore: In order to make the GC concurrent (bug 149432), we would either need to enable concurrent copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead from the barriers alone. Considering that MarkedSpace does a decent job of avoiding fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want to get rid of copied space. This change moves copied space's biggest client over to marked space. Moving butterflies to marked space means having them use the new Auxiliary HeapCell allocation path. This is a fairly mechanical change, but it caused performance regressions everywhere, so this change also fixes MarkedSpace's performance issues. At a high level the mechanical changes are: - We use AuxiliaryBarrier instead of CopyBarrier. - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth. - The JITs have to emit inlined marked space allocations instead of inline copy space allocations. - Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't. That's about 1/3 of this change. But this led to performance problems, which I fixed with optimizations that amounted to a major MarkedSpace rewrite: - MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely. - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well. - Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being: 16, 32, 48, 64, 80, 112, 160, 224, 320 The next size class would have been 432, but that's above the 400 byte cutoff. All of this is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what size classes you end up with by doing --dumpSizeClasses=true. - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: https://bugs.webkit.org/show_bug.cgi?id=161581. - Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree. - FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky. - It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch! - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes. All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1% regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a best-case scenario for bump allocation. I think that we should fix bmalloc to make up the difference, but take the hit for now because it's a crazy corner case. By comparison, the alternative approach of using a copy barrier would have cost us 1-2%. That's the real apples-to-apples comparison if your premise is that we should have a concurrent GC. After we finish removing copied space, we will be barrier-ready for concurrent GC: we already have a marking barrier and we simply won't need a copying barrier. This change gets us there for the purposes of our benchmarks, since the remaining clients of copied space are not very important. On the other hand, if we keep copying, then getting barrier-ready would mean adding back the copy barrier, which costs more perf. We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving typed arrays and a few other weird things over to Aux MarkedSpace. This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell, and CellContainer meant that I had to include those files from everywhere. Fortunately, just including JSCInlines.h (instead of manually including the files that includes) is usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something that we were already basically doing. In places where JSCInlines.h would be too much, I just included HeapInlines.h. This got weird, because we previously included HeapInlines.h from JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that meant having to manually include HeapInlines.h from the places that previously got it implicitly via JSObject.h. But that led to more problems for some reason: I started getting build errors because non-JSC files were having trouble including Opcode.h. That's just silly, since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and made it impossible to include it from outside JSC. This was a lot of work, but it was necessary to get the patch to build on all ports. It's also a net win. There were many places in WebCore that were transitively including a *ton* of JSC headers just because of the JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons. * API/JSManagedValue.mm: (-[JSManagedValue initWithValue:]): * API/JSTypedArray.cpp: * API/ObjCCallbackFunction.mm: * API/tests/testapi.mm: (testObjectiveCAPI): (testWeakValue): Deleted. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * Scripts/builtins/builtins_generate_combined_implementation.py: (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py: (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_separate_implementation.py: (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes): * assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::JumpList::link): (JSC::AbstractMacroAssembler::JumpList::linkTo): * assembler/MacroAssembler.h: * assembler/MacroAssemblerARM64.h: (JSC::MacroAssemblerARM64::add32): * assembler/MacroAssemblerCodeRef.cpp: Added. (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): (JSC::MacroAssemblerCodePtr::dumpWithName): (JSC::MacroAssemblerCodePtr::dump): (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): (JSC::MacroAssemblerCodeRef::dump): * assembler/MacroAssemblerCodeRef.h: (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted. (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted. (JSC::MacroAssemblerCodePtr::dump): Deleted. (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted. (JSC::MacroAssemblerCodeRef::dump): Deleted. * b3/B3BasicBlock.cpp: (JSC::B3::BasicBlock::appendBoolConstant): * b3/B3BasicBlock.h: * b3/B3DuplicateTails.cpp: * b3/B3StackmapGenerationParams.h: * b3/testb3.cpp: (JSC::B3::testPatchpointTerminalReturnValue): (JSC::B3::run): * bindings/ScriptValue.cpp: * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeLivenessAnalysis.cpp: * bytecode/BytecodeUseDef.h: * bytecode/CallLinkInfo.cpp: (JSC::CallLinkInfo::callTypeFor): * bytecode/CallLinkInfo.h: (JSC::CallLinkInfo::callTypeFor): Deleted. * bytecode/CallLinkStatus.cpp: * bytecode/CodeBlock.cpp: (JSC::CodeBlock::finishCreation): (JSC::CodeBlock::clearLLIntGetByIdCache): (JSC::CodeBlock::predictedMachineCodeSize): * bytecode/CodeBlock.h: (JSC::CodeBlock::jitCodeMap): Deleted. (JSC::clearLLIntGetByIdCache): Deleted. * bytecode/ExecutionCounter.h: * bytecode/Instruction.h: * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp: (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal): * bytecode/ObjectAllocationProfile.h: (JSC::ObjectAllocationProfile::isNull): (JSC::ObjectAllocationProfile::initialize): * bytecode/Opcode.h: (JSC::padOpcodeName): * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): (JSC::PolymorphicAccess::regenerate): * bytecode/PolymorphicAccess.h: * bytecode/PreciseJumpTargets.cpp: * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecode/UnlinkedCodeBlock.cpp: (JSC::UnlinkedCodeBlock::vm): Deleted. * bytecode/UnlinkedCodeBlock.h: * bytecode/UnlinkedInstructionStream.cpp: * bytecode/UnlinkedInstructionStream.h: * dfg/DFGOperations.cpp: * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT.h: (JSC::DFG::SpeculativeJIT::emitAllocateJSCell): (JSC::DFG::SpeculativeJIT::emitAllocateJSObject): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGStrengthReductionPhase.cpp: (JSC::DFG::StrengthReductionPhase::handleNode): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCompile.cpp: * ftl/FTLJITFinalizer.cpp: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments): (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest): (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell): (JSC::FTL::DFG::LowerDFGToB3::allocateCell): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted. * ftl/FTLOutput.cpp: (JSC::FTL::Output::constBool): (JSC::FTL::Output::add): (JSC::FTL::Output::shl): (JSC::FTL::Output::aShr): (JSC::FTL::Output::lShr): (JSC::FTL::Output::zeroExt): (JSC::FTL::Output::equal): (JSC::FTL::Output::notEqual): (JSC::FTL::Output::above): (JSC::FTL::Output::aboveOrEqual): (JSC::FTL::Output::below): (JSC::FTL::Output::belowOrEqual): (JSC::FTL::Output::greaterThan): (JSC::FTL::Output::greaterThanOrEqual): (JSC::FTL::Output::lessThan): (JSC::FTL::Output::lessThanOrEqual): (JSC::FTL::Output::select): (JSC::FTL::Output::appendSuccessor): (JSC::FTL::Output::addIncomingToPhi): * ftl/FTLOutput.h: * ftl/FTLValueFromBlock.h: (JSC::FTL::ValueFromBlock::operator bool): (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted. * ftl/FTLWeightedTarget.h: (JSC::FTL::WeightedTarget::frequentedBlock): * heap/CellContainer.h: Added. (JSC::CellContainer::CellContainer): (JSC::CellContainer::operator bool): (JSC::CellContainer::isMarkedBlock): (JSC::CellContainer::isLargeAllocation): (JSC::CellContainer::markedBlock): (JSC::CellContainer::largeAllocation): * heap/CellContainerInlines.h: Added. (JSC::CellContainer::isMarked): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::noteMarked): (JSC::CellContainer::cellSize): (JSC::CellContainer::weakSet): (JSC::CellContainer::flipIfNecessary): * heap/ConservativeRoots.cpp: (JSC::ConservativeRoots::ConservativeRoots): (JSC::ConservativeRoots::~ConservativeRoots): (JSC::ConservativeRoots::grow): (JSC::ConservativeRoots::genericAddPointer): (JSC::ConservativeRoots::genericAddSpan): * heap/ConservativeRoots.h: (JSC::ConservativeRoots::roots): * heap/CopyToken.h: * heap/FreeList.cpp: Added. (JSC::FreeList::dump): * heap/FreeList.h: Added. (JSC::FreeList::FreeList): (JSC::FreeList::list): (JSC::FreeList::bump): (JSC::FreeList::operator==): (JSC::FreeList::operator!=): (JSC::FreeList::operator bool): (JSC::FreeList::allocationWillFail): (JSC::FreeList::allocationWillSucceed): * heap/GCTypeMap.h: Added. (JSC::GCTypeMap::operator[]): * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalizeUnconditionalFinalizers): (JSC::Heap::markRoots): (JSC::Heap::copyBackingStores): (JSC::Heap::gatherStackRoots): (JSC::Heap::gatherJSStackRoots): (JSC::Heap::gatherScratchBufferRoots): (JSC::Heap::clearLivenessData): (JSC::Heap::visitSmallStrings): (JSC::Heap::visitConservativeRoots): (JSC::Heap::removeDeadCompilerWorklistEntries): (JSC::Heap::gatherExtraHeapSnapshotData): (JSC::Heap::removeDeadHeapSnapshotNodes): (JSC::Heap::visitProtectedObjects): (JSC::Heap::visitArgumentBuffers): (JSC::Heap::visitException): (JSC::Heap::visitStrongHandles): (JSC::Heap::visitHandleStack): (JSC::Heap::visitSamplingProfiler): (JSC::Heap::traceCodeBlocksAndJITStubRoutines): (JSC::Heap::converge): (JSC::Heap::visitWeakHandles): (JSC::Heap::updateObjectCounts): (JSC::Heap::clearUnmarkedExecutables): (JSC::Heap::deleteUnmarkedCompiledCode): (JSC::Heap::collectAllGarbage): (JSC::Heap::collect): (JSC::Heap::collectWithoutAnySweep): (JSC::Heap::collectImpl): (JSC::Heap::suspendCompilerThreads): (JSC::Heap::willStartCollection): (JSC::Heap::flushOldStructureIDTables): (JSC::Heap::flushWriteBarrierBuffer): (JSC::Heap::stopAllocation): (JSC::Heap::prepareForMarking): (JSC::Heap::reapWeakHandles): (JSC::Heap::pruneStaleEntriesFromWeakGCMaps): (JSC::Heap::sweepArrayBuffers): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): (JSC::MarkedBlockSnapshotFunctor::operator()): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::deleteSourceProviderCaches): (JSC::Heap::notifyIncrementalSweeper): (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks): (JSC::Heap::resetAllocators): (JSC::Heap::updateAllocationLimits): (JSC::Heap::didFinishCollection): (JSC::Heap::resumeCompilerThreads): (JSC::Zombify::visit): (JSC::Heap::forEachCodeBlockImpl): * heap/Heap.h: (JSC::Heap::allocatorForObjectWithoutDestructor): (JSC::Heap::allocatorForObjectWithDestructor): (JSC::Heap::allocatorForAuxiliaryData): (JSC::Heap::jitStubRoutines): (JSC::Heap::codeBlockSet): (JSC::Heap::storageAllocator): Deleted. * heap/HeapCell.h: (JSC::HeapCell::isZapped): Deleted. * heap/HeapCellInlines.h: Added. (JSC::HeapCell::isLargeAllocation): (JSC::HeapCell::cellContainer): (JSC::HeapCell::markedBlock): (JSC::HeapCell::largeAllocation): (JSC::HeapCell::heap): (JSC::HeapCell::vm): (JSC::HeapCell::cellSize): (JSC::HeapCell::allocatorAttributes): (JSC::HeapCell::destructionMode): (JSC::HeapCell::cellKind): * heap/HeapInlines.h: (JSC::Heap::heap): (JSC::Heap::isLive): (JSC::Heap::isMarked): (JSC::Heap::testAndSetMarked): (JSC::Heap::setMarked): (JSC::Heap::cellSize): (JSC::Heap::forEachCodeBlock): (JSC::Heap::allocateObjectOfType): (JSC::Heap::subspaceForObjectOfType): (JSC::Heap::allocatorForObjectOfType): (JSC::Heap::allocateAuxiliary): (JSC::Heap::tryAllocateAuxiliary): (JSC::Heap::tryReallocateAuxiliary): (JSC::Heap::isPointerGCObject): Deleted. (JSC::Heap::isValueGCObject): Deleted. * heap/HeapOperation.cpp: Added. (WTF::printInternal): * heap/HeapOperation.h: * heap/HeapUtil.h: Added. (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): (JSC::HeapUtil::isValueGCObject): * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::sweepNextBlock): * heap/IncrementalSweeper.h: * heap/LargeAllocation.cpp: Added. (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::lastChanceToFinalize): (JSC::LargeAllocation::shrink): (JSC::LargeAllocation::visitWeakSet): (JSC::LargeAllocation::reapWeakSet): (JSC::LargeAllocation::flip): (JSC::LargeAllocation::isEmpty): (JSC::LargeAllocation::sweep): (JSC::LargeAllocation::destroy): (JSC::LargeAllocation::dump): * heap/LargeAllocation.h: Added. (JSC::LargeAllocation::fromCell): (JSC::LargeAllocation::cell): (JSC::LargeAllocation::isLargeAllocation): (JSC::LargeAllocation::heap): (JSC::LargeAllocation::vm): (JSC::LargeAllocation::weakSet): (JSC::LargeAllocation::clearNewlyAllocated): (JSC::LargeAllocation::isNewlyAllocated): (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::isLive): (JSC::LargeAllocation::hasValidCell): (JSC::LargeAllocation::cellSize): (JSC::LargeAllocation::aboveLowerBound): (JSC::LargeAllocation::belowUpperBound): (JSC::LargeAllocation::contains): (JSC::LargeAllocation::attributes): (JSC::LargeAllocation::flipIfNecessary): (JSC::LargeAllocation::flipIfNecessaryConcurrently): (JSC::LargeAllocation::testAndSetMarked): (JSC::LargeAllocation::setMarked): (JSC::LargeAllocation::clearMarked): (JSC::LargeAllocation::noteMarked): (JSC::LargeAllocation::headerSize): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::retire): (JSC::MarkedAllocator::filterNextBlock): (JSC::MarkedAllocator::setNextBlockToSweep): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::blockHeaderSize): (JSC::MarkedAllocator::blockSizeForBytes): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::reset): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::setFreeList): (JSC::isListPagedOut): Deleted. (JSC::MarkedAllocator::tryAllocateHelper): Deleted. (JSC::MarkedAllocator::tryPopFreeList): Deleted. (JSC::MarkedAllocator::tryAllocate): Deleted. (JSC::MarkedAllocator::allocateBlock): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::takeLastActiveBlock): (JSC::MarkedAllocator::offsetOfFreeList): (JSC::MarkedAllocator::offsetOfCellSize): (JSC::MarkedAllocator::tryAllocate): (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted. (JSC::MarkedAllocator::MarkedAllocator): Deleted. (JSC::MarkedAllocator::init): Deleted. (JSC::MarkedAllocator::stopAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): (JSC::SetNewlyAllocatedFunctor::operator()): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::Handle::zap): (JSC::MarkedBlock::Handle::forEachFreeCell): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessarySlow): (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::needsFlip): (JSC::MarkedBlock::Handle::needsFlip): (JSC::MarkedBlock::Handle::willRemoveBlock): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::clearHasAnyMarked): (JSC::MarkedBlock::noteMarkedSlow): (WTF::printInternal): (JSC::MarkedBlock::create): Deleted. (JSC::MarkedBlock::destroy): Deleted. (JSC::MarkedBlock::callDestructor): Deleted. (JSC::MarkedBlock::specializedSweep): Deleted. (JSC::MarkedBlock::sweep): Deleted. (JSC::MarkedBlock::sweepHelper): Deleted. (JSC::MarkedBlock::stopAllocating): Deleted. (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted. (JSC::MarkedBlock::lastChanceToFinalize): Deleted. (JSC::MarkedBlock::resumeAllocating): Deleted. (JSC::MarkedBlock::didRetireBlock): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::VoidFunctor::returnValue): (JSC::MarkedBlock::CountFunctor::CountFunctor): (JSC::MarkedBlock::CountFunctor::count): (JSC::MarkedBlock::CountFunctor::returnValue): (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): (JSC::MarkedBlock::Handle::state): (JSC::MarkedBlock::needsDestruction): (JSC::MarkedBlock::handle): (JSC::MarkedBlock::Handle::block): (JSC::MarkedBlock::firstAtom): (JSC::MarkedBlock::atoms): (JSC::MarkedBlock::isAtomAligned): (JSC::MarkedBlock::Handle::cellAlign): (JSC::MarkedBlock::blockFor): (JSC::MarkedBlock::Handle::allocator): (JSC::MarkedBlock::Handle::heap): (JSC::MarkedBlock::Handle::vm): (JSC::MarkedBlock::vm): (JSC::MarkedBlock::Handle::weakSet): (JSC::MarkedBlock::weakSet): (JSC::MarkedBlock::Handle::shrink): (JSC::MarkedBlock::Handle::visitWeakSet): (JSC::MarkedBlock::Handle::reapWeakSet): (JSC::MarkedBlock::Handle::cellSize): (JSC::MarkedBlock::cellSize): (JSC::MarkedBlock::Handle::attributes): (JSC::MarkedBlock::attributes): (JSC::MarkedBlock::Handle::needsDestruction): (JSC::MarkedBlock::Handle::destruction): (JSC::MarkedBlock::Handle::cellKind): (JSC::MarkedBlock::Handle::markCount): (JSC::MarkedBlock::Handle::size): (JSC::MarkedBlock::atomNumber): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipForEdenCollection): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::Handle::assertFlipped): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::testAndSetMarked): (JSC::MarkedBlock::Handle::isNewlyAllocated): (JSC::MarkedBlock::Handle::setNewlyAllocated): (JSC::MarkedBlock::Handle::clearNewlyAllocated): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::isAtom): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::Handle::needsSweeping): (JSC::MarkedBlock::Handle::isAllocated): (JSC::MarkedBlock::Handle::isMarked): (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::hasAnyMarked): (JSC::MarkedBlock::noteMarked): (WTF::MarkedBlockHash::hash): (JSC::MarkedBlock::FreeList::FreeList): Deleted. (JSC::MarkedBlock::allocator): Deleted. (JSC::MarkedBlock::heap): Deleted. (JSC::MarkedBlock::shrink): Deleted. (JSC::MarkedBlock::visitWeakSet): Deleted. (JSC::MarkedBlock::reapWeakSet): Deleted. (JSC::MarkedBlock::willRemoveBlock): Deleted. (JSC::MarkedBlock::didConsumeFreeList): Deleted. (JSC::MarkedBlock::markCount): Deleted. (JSC::MarkedBlock::isEmpty): Deleted. (JSC::MarkedBlock::destruction): Deleted. (JSC::MarkedBlock::cellKind): Deleted. (JSC::MarkedBlock::size): Deleted. (JSC::MarkedBlock::capacity): Deleted. (JSC::MarkedBlock::setMarked): Deleted. (JSC::MarkedBlock::clearMarked): Deleted. (JSC::MarkedBlock::isNewlyAllocated): Deleted. (JSC::MarkedBlock::setNewlyAllocated): Deleted. (JSC::MarkedBlock::clearNewlyAllocated): Deleted. (JSC::MarkedBlock::isLive): Deleted. (JSC::MarkedBlock::isLiveCell): Deleted. (JSC::MarkedBlock::forEachCell): Deleted. (JSC::MarkedBlock::forEachLiveCell): Deleted. (JSC::MarkedBlock::forEachDeadCell): Deleted. (JSC::MarkedBlock::needsSweeping): Deleted. (JSC::MarkedBlock::isAllocated): Deleted. (JSC::MarkedBlock::isMarkedOrRetired): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::initializeSizeClassForStepSize): (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::lastChanceToFinalize): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::allocateLarge): (JSC::MarkedSpace::tryAllocateLarge): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::zombifySweep): (JSC::MarkedSpace::resetAllocators): (JSC::MarkedSpace::visitWeakSets): (JSC::MarkedSpace::reapWeakSets): (JSC::MarkedSpace::stopAllocating): (JSC::MarkedSpace::prepareForMarking): (JSC::MarkedSpace::resumeAllocating): (JSC::MarkedSpace::isPagedOut): (JSC::MarkedSpace::freeBlock): (JSC::MarkedSpace::freeOrShrinkBlock): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::VerifyMarked::operator()): (JSC::MarkedSpace::flip): (JSC::MarkedSpace::objectCount): (JSC::MarkedSpace::size): (JSC::MarkedSpace::capacity): (JSC::MarkedSpace::addActiveWeakSet): (JSC::MarkedSpace::didAddBlock): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::forEachAllocator): Deleted. (JSC::VerifyMarkedOrRetired::operator()): Deleted. (JSC::MarkedSpace::clearMarks): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::sizeClassToIndex): (JSC::MarkedSpace::indexToSizeClass): (JSC::MarkedSpace::version): (JSC::MarkedSpace::blocksWithNewObjects): (JSC::MarkedSpace::largeAllocations): (JSC::MarkedSpace::largeAllocationsNurseryOffset): (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection): (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin): (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd): (JSC::MarkedSpace::largeAllocationsForThisCollectionSize): (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): (JSC::MarkedSpace::allocatorFor): (JSC::MarkedSpace::destructorAllocatorFor): (JSC::MarkedSpace::auxiliaryAllocatorFor): (JSC::MarkedSpace::allocateWithoutDestructor): (JSC::MarkedSpace::allocateWithDestructor): (JSC::MarkedSpace::allocateAuxiliary): (JSC::MarkedSpace::tryAllocateAuxiliary): (JSC::MarkedSpace::forEachBlock): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::optimalSizeFor): (JSC::MarkedSpace::didAddBlock): Deleted. (JSC::MarkedSpace::didAllocateInBlock): Deleted. (JSC::MarkedSpace::objectCount): Deleted. (JSC::MarkedSpace::size): Deleted. (JSC::MarkedSpace::capacity): Deleted. * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::didStartMarking): (JSC::SlotVisitor::reset): (JSC::SlotVisitor::append): (JSC::SlotVisitor::appendJSCellOrAuxiliary): (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::appendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::noteLiveAuxiliaryCell): (JSC::SlotVisitor::visitChildren): * heap/SlotVisitor.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::WeakBlock): (JSC::WeakBlock::visit): (JSC::WeakBlock::reap): * heap/WeakBlock.h: (JSC::WeakBlock::disconnectContainer): (JSC::WeakBlock::disconnectMarkedBlock): Deleted. * heap/WeakSet.cpp: (JSC::WeakSet::~WeakSet): (JSC::WeakSet::sweep): (JSC::WeakSet::shrink): (JSC::WeakSet::addAllocator): * heap/WeakSet.h: (JSC::WeakSet::container): (JSC::WeakSet::setContainer): (JSC::WeakSet::WeakSet): (JSC::WeakSet::visit): (JSC::WeakSet::shrink): Deleted. * heap/WeakSetInlines.h: (JSC::WeakSet::allocate): * inspector/InjectedScriptManager.cpp: * inspector/JSGlobalObjectInspectorController.cpp: * inspector/JSJavaScriptCallFrame.cpp: * inspector/ScriptDebugServer.cpp: * inspector/agents/InspectorDebuggerAgent.cpp: * interpreter/CachedCall.h: (JSC::CachedCall::CachedCall): * interpreter/Interpreter.cpp: (JSC::loadVarargs): (JSC::StackFrame::sourceID): Deleted. (JSC::StackFrame::sourceURL): Deleted. (JSC::StackFrame::functionName): Deleted. (JSC::StackFrame::computeLineAndColumn): Deleted. (JSC::StackFrame::toString): Deleted. * interpreter/Interpreter.h: (JSC::StackFrame::isNative): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): (JSC::AssemblyHelpers::emitAllocate): (JSC::AssemblyHelpers::emitAllocateJSCell): (JSC::AssemblyHelpers::emitAllocateJSObject): (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): * jit/GCAwareJITStubRoutine.cpp: (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine): * jit/JIT.cpp: (JSC::JIT::compileCTINativeCall): (JSC::JIT::link): * jit/JIT.h: (JSC::JIT::compileCTINativeCall): Deleted. * jit/JITExceptions.cpp: (JSC::genericUnwind): * jit/JITExceptions.h: * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOpcodes32_64.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emitWriteBarrier): * jit/JITThunks.cpp: * jit/JITThunks.h: * jsc.cpp: (functionDescribeArray): (main): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntExceptions.cpp: * llint/LLIntThunks.cpp: * llint/LLIntThunks.h: * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter.cpp: * llint/LowLevelInterpreter32_64.asm: * llint/LowLevelInterpreter64.asm: * parser/ModuleAnalyzer.cpp: * parser/NodeConstructors.h: * parser/Nodes.h: * profiler/ProfilerBytecode.cpp: * profiler/ProfilerBytecode.h: * profiler/ProfilerBytecodeSequence.cpp: * runtime/ArrayConventions.h: (JSC::indexingHeaderForArrayStorage): (JSC::baseIndexingHeaderForArrayStorage): (JSC::indexingHeaderForArray): Deleted. (JSC::baseIndexingHeaderForArray): Deleted. * runtime/ArrayPrototype.cpp: (JSC::arrayProtoFuncSplice): (JSC::concatAppendOne): (JSC::arrayProtoPrivateFuncConcatMemcpy): * runtime/ArrayStorage.h: (JSC::ArrayStorage::vectorLength): (JSC::ArrayStorage::totalSizeFor): (JSC::ArrayStorage::totalSize): (JSC::ArrayStorage::availableVectorLength): (JSC::ArrayStorage::optimalVectorLength): (JSC::ArrayStorage::sizeFor): Deleted. * runtime/AuxiliaryBarrier.h: Added. (JSC::AuxiliaryBarrier::AuxiliaryBarrier): (JSC::AuxiliaryBarrier::clear): (JSC::AuxiliaryBarrier::get): (JSC::AuxiliaryBarrier::slot): (JSC::AuxiliaryBarrier::operator bool): (JSC::AuxiliaryBarrier::setWithoutBarrier): * runtime/AuxiliaryBarrierInlines.h: Added. (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier): (JSC::AuxiliaryBarrier<T>::set): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::availableContiguousVectorLength): (JSC::Butterfly::optimalContiguousVectorLength): (JSC::Butterfly::createUninitialized): (JSC::Butterfly::growArrayRight): * runtime/ClonedArguments.cpp: (JSC::ClonedArguments::createEmpty): * runtime/CommonSlowPathsExceptions.cpp: * runtime/CommonSlowPathsExceptions.h: * runtime/DataView.cpp: * runtime/DirectArguments.h: * runtime/ECMAScriptSpecInternalFunctions.cpp: * runtime/Error.cpp: * runtime/Error.h: * runtime/ErrorInstance.cpp: * runtime/ErrorInstance.h: * runtime/Exception.cpp: * runtime/Exception.h: * runtime/GeneratorFrame.cpp: * runtime/GeneratorPrototype.cpp: * runtime/InternalFunction.cpp: (JSC::InternalFunction::InternalFunction): * runtime/IntlCollator.cpp: * runtime/IntlCollatorConstructor.cpp: * runtime/IntlCollatorPrototype.cpp: * runtime/IntlDateTimeFormat.cpp: * runtime/IntlDateTimeFormatConstructor.cpp: * runtime/IntlDateTimeFormatPrototype.cpp: * runtime/IntlNumberFormat.cpp: * runtime/IntlNumberFormatConstructor.cpp: * runtime/IntlNumberFormatPrototype.cpp: * runtime/IntlObject.cpp: * runtime/IteratorPrototype.cpp: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitialized): (JSC::JSArray::setLengthWritable): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLengthWithArrayStorage): (JSC::JSArray::appendMemcpy): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithArrayStorage): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::createContiguousArrayButterfly): (JSC::createArrayButterfly): (JSC::JSArray::create): (JSC::JSArray::tryCreateUninitialized): Deleted. * runtime/JSArrayBufferView.h: * runtime/JSCInlines.h: * runtime/JSCJSValue.cpp: (JSC::JSValue::dumpInContextAssumingStructure): * runtime/JSCallee.cpp: (JSC::JSCallee::JSCallee): * runtime/JSCell.cpp: (JSC::JSCell::estimatedSize): * runtime/JSCell.h: (JSC::JSCell::cellStateOffset): Deleted. * runtime/JSCellInlines.h: (JSC::ExecState::vm): (JSC::JSCell::classInfo): (JSC::JSCell::callDestructor): (JSC::JSCell::vm): Deleted. * runtime/JSFunction.cpp: (JSC::JSFunction::create): (JSC::JSFunction::allocateAndInitializeRareData): (JSC::JSFunction::initializeRareData): (JSC::JSFunction::getOwnPropertySlot): (JSC::JSFunction::put): (JSC::JSFunction::deleteProperty): (JSC::JSFunction::defineOwnProperty): (JSC::JSFunction::setFunctionName): (JSC::JSFunction::reifyLength): (JSC::JSFunction::reifyName): (JSC::JSFunction::reifyLazyPropertyIfNeeded): (JSC::JSFunction::reifyBoundNameIfNeeded): * runtime/JSFunction.h: * runtime/JSFunctionInlines.h: (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint): (JSC::JSFunction::JSFunction): * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory): * runtime/JSInternalPromise.cpp: * runtime/JSInternalPromiseConstructor.cpp: * runtime/JSInternalPromiseDeferred.cpp: * runtime/JSInternalPromisePrototype.cpp: * runtime/JSJob.cpp: * runtime/JSMapIterator.cpp: * runtime/JSModuleNamespaceObject.cpp: * runtime/JSModuleRecord.cpp: * runtime/JSObject.cpp: (JSC::JSObject::visitButterfly): (JSC::JSObject::notifyPresenceOfIndexedAccessors): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createInitialUndecided): (JSC::JSObject::createInitialInt32): (JSC::JSObject::createInitialDouble): (JSC::JSObject::createInitialContiguous): (JSC::JSObject::createArrayStorage): (JSC::JSObject::createInitialArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::putByIndexBeyondVectorLength): (JSC::JSObject::putDirectIndexBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::increaseVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::growOutOfLineStorage): (JSC::JSObject::copyButterfly): Deleted. (JSC::JSObject::copyBackingStore): Deleted. * runtime/JSObject.h: (JSC::JSObject::globalObject): (JSC::JSObject::putDirectInternal): (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted. * runtime/JSObjectInlines.h: * runtime/JSPromise.cpp: * runtime/JSPromiseConstructor.cpp: * runtime/JSPromiseDeferred.cpp: * runtime/JSPromisePrototype.cpp: * runtime/JSPropertyNameIterator.cpp: * runtime/JSScope.cpp: (JSC::JSScope::resolve): * runtime/JSScope.h: (JSC::JSScope::globalObject): (JSC::JSScope::vm): Deleted. * runtime/JSSetIterator.cpp: * runtime/JSStringIterator.cpp: * runtime/JSTemplateRegistryKey.cpp: * runtime/JSTypedArrayViewConstructor.cpp: * runtime/JSTypedArrayViewPrototype.cpp: * runtime/JSWeakMap.cpp: * runtime/JSWeakSet.cpp: * runtime/MapConstructor.cpp: * runtime/MapIteratorPrototype.cpp: * runtime/MapPrototype.cpp: * runtime/NativeErrorConstructor.cpp: * runtime/NativeStdFunctionCell.cpp: * runtime/Operations.h: (JSC::scribbleFreeCells): (JSC::scribble): * runtime/Options.h: * runtime/PropertyTable.cpp: * runtime/ProxyConstructor.cpp: * runtime/ProxyObject.cpp: * runtime/ProxyRevoke.cpp: * runtime/RegExp.cpp: (JSC::RegExp::match): (JSC::RegExp::matchConcurrently): (JSC::RegExp::matchCompareWithInterpreter): * runtime/RegExp.h: * runtime/RegExpConstructor.h: * runtime/RegExpInlines.h: (JSC::RegExp::matchInline): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): (JSC::createRegExpMatchesArray): * runtime/RegExpPrototype.cpp: (JSC::genericSplit): * runtime/RuntimeType.cpp: * runtime/SamplingProfiler.cpp: (JSC::SamplingProfiler::processUnverifiedStackTraces): * runtime/SetConstructor.cpp: * runtime/SetIteratorPrototype.cpp: * runtime/SetPrototype.cpp: * runtime/StackFrame.cpp: Added. (JSC::StackFrame::sourceID): (JSC::StackFrame::sourceURL): (JSC::StackFrame::functionName): (JSC::StackFrame::computeLineAndColumn): (JSC::StackFrame::toString): * runtime/StackFrame.h: Added. (JSC::StackFrame::isNative): * runtime/StringConstructor.cpp: * runtime/StringIteratorPrototype.cpp: * runtime/StructureInlines.h: (JSC::Structure::propertyTable): * runtime/TemplateRegistry.cpp: * runtime/TestRunnerUtils.cpp: (JSC::finalizeStatsAtEndOfTesting): * runtime/TestRunnerUtils.h: * runtime/TypeProfilerLog.cpp: * runtime/TypeSet.cpp: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::ensureStackCapacityForCLoop): (JSC::VM::isSafeToRecurseSoftCLoop): * runtime/VM.h: * runtime/VMEntryScope.h: * runtime/VMInlines.h: (JSC::VM::ensureStackCapacityFor): (JSC::VM::isSafeToRecurseSoft): * runtime/WeakMapConstructor.cpp: * runtime/WeakMapData.cpp: * runtime/WeakMapPrototype.cpp: * runtime/WeakSetConstructor.cpp: * runtime/WeakSetPrototype.cpp: * testRegExp.cpp: (testOneRegExp): * tools/JSDollarVM.cpp: * tools/JSDollarVMPrototype.cpp: (JSC::JSDollarVMPrototype::isInObjectSpace): Source/WebCore: No new tests because no new WebCore behavior. Just rewiring #includes. * ForwardingHeaders/heap/HeapInlines.h: Added. * ForwardingHeaders/interpreter/Interpreter.h: Removed. * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added. * Modules/indexeddb/IDBCursorWithValue.cpp: * Modules/indexeddb/client/TransactionOperation.cpp: * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp: * Modules/indexeddb/server/UniqueIDBDatabase.cpp: * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp: * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp: * bindings/js/JSClientRectCustom.cpp: * bindings/js/JSDOMBinding.cpp: * bindings/js/JSDOMBinding.h: * bindings/js/JSDeviceMotionEventCustom.cpp: * bindings/js/JSDeviceOrientationEventCustom.cpp: * bindings/js/JSErrorEventCustom.cpp: * bindings/js/JSIDBCursorWithValueCustom.cpp: * bindings/js/JSIDBIndexCustom.cpp: * bindings/js/JSPopStateEventCustom.cpp: * bindings/js/JSWebGL2RenderingContextCustom.cpp: * bindings/js/JSWorkerGlobalScopeCustom.cpp: * bindings/js/WorkerScriptController.cpp: * contentextensions/ContentExtensionParser.cpp: * dom/ErrorEvent.cpp: * html/HTMLCanvasElement.cpp: * html/MediaDocument.cpp: * inspector/CommandLineAPIModule.cpp: * loader/EmptyClients.cpp: * page/CaptionUserPreferences.cpp: * page/Frame.cpp: * page/PageGroup.cpp: * page/UserContentController.cpp: * platform/mock/mediasource/MockBox.cpp: * testing/GCObservation.cpp: Source/WebKit2: Just rewiring some #includes. * UIProcess/ViewGestureController.cpp: * UIProcess/WebPageProxy.cpp: * UIProcess/WebProcessPool.cpp: * UIProcess/WebProcessProxy.cpp: * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp: * WebProcess/Plugins/Netscape/JSNPObject.cpp: Source/WTF: I needed tryFastAlignedMalloc() so I added it. * wtf/FastMalloc.cpp: (WTF::tryFastAlignedMalloc): * wtf/FastMalloc.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): (WTF::ParkingLot::forEach): Deleted. * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::forEach): * wtf/ScopedLambda.h: (WTF::scopedLambdaRef): * wtf/SentinelLinkedList.h: (WTF::SentinelLinkedList::forEach): (WTF::RawNode>::takeFrom): * wtf/SimpleStats.h: (WTF::SimpleStats::operator bool): (WTF::SimpleStats::operator!): Deleted. Tools: * DumpRenderTree/TestRunner.cpp: * DumpRenderTree/mac/DumpRenderTree.mm: (DumpRenderTreeMain): * Scripts/run-jsc-stress-tests: * TestWebKitAPI/Tests/WTF/Vector.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/179778@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
template<typename Func>
static void forEach(const Func& func)
{
[WTF] Introduce Thread class and use RefPtr<Thread> and align Windows Threading implementation semantics to Pthread one https://bugs.webkit.org/show_bug.cgi?id=170502 Reviewed by Mark Lam. Source/JavaScriptCore: * API/tests/CompareAndSwapTest.cpp: (testCompareAndSwap): * JavaScriptCore.xcodeproj/project.pbxproj: * b3/air/testair.cpp: * b3/testb3.cpp: (JSC::B3::run): * bytecode/SuperSampler.cpp: (JSC::initializeSuperSampler): * dfg/DFGWorklist.cpp: * disassembler/Disassembler.cpp: * heap/Heap.cpp: (JSC::Heap::lastChanceToFinalize): (JSC::Heap::notifyIsSafeToCollect): * heap/Heap.h: * heap/MachineStackMarker.cpp: (JSC::MachineThreads::~MachineThreads): (JSC::MachineThreads::addCurrentThread): (JSC::MachineThreads::removeThread): (JSC::MachineThreads::removeThreadIfFound): (JSC::MachineThreads::MachineThread::MachineThread): (JSC::MachineThreads::MachineThread::getRegisters): (JSC::MachineThreads::MachineThread::Registers::stackPointer): (JSC::MachineThreads::MachineThread::Registers::framePointer): (JSC::MachineThreads::MachineThread::Registers::instructionPointer): (JSC::MachineThreads::MachineThread::Registers::llintPC): (JSC::MachineThreads::MachineThread::captureStack): (JSC::MachineThreads::tryCopyOtherThreadStack): (JSC::MachineThreads::tryCopyOtherThreadStacks): (pthreadSignalHandlerSuspendResume): Deleted. (JSC::threadData): Deleted. (JSC::MachineThreads::Thread::Thread): Deleted. (JSC::MachineThreads::Thread::createForCurrentThread): Deleted. (JSC::MachineThreads::Thread::operator==): Deleted. (JSC::MachineThreads::machineThreadForCurrentThread): Deleted. (JSC::MachineThreads::ThreadData::ThreadData): Deleted. (JSC::MachineThreads::ThreadData::~ThreadData): Deleted. (JSC::MachineThreads::ThreadData::suspend): Deleted. (JSC::MachineThreads::ThreadData::resume): Deleted. (JSC::MachineThreads::ThreadData::getRegisters): Deleted. (JSC::MachineThreads::ThreadData::Registers::stackPointer): Deleted. (JSC::MachineThreads::ThreadData::Registers::framePointer): Deleted. (JSC::MachineThreads::ThreadData::Registers::instructionPointer): Deleted. (JSC::MachineThreads::ThreadData::Registers::llintPC): Deleted. (JSC::MachineThreads::ThreadData::freeRegisters): Deleted. (JSC::MachineThreads::ThreadData::captureStack): Deleted. * heap/MachineStackMarker.h: (JSC::MachineThreads::MachineThread::suspend): (JSC::MachineThreads::MachineThread::resume): (JSC::MachineThreads::MachineThread::threadID): (JSC::MachineThreads::MachineThread::stackBase): (JSC::MachineThreads::MachineThread::stackEnd): (JSC::MachineThreads::threadsListHead): (JSC::MachineThreads::Thread::operator!=): Deleted. (JSC::MachineThreads::Thread::suspend): Deleted. (JSC::MachineThreads::Thread::resume): Deleted. (JSC::MachineThreads::Thread::getRegisters): Deleted. (JSC::MachineThreads::Thread::freeRegisters): Deleted. (JSC::MachineThreads::Thread::captureStack): Deleted. (JSC::MachineThreads::Thread::platformThread): Deleted. (JSC::MachineThreads::Thread::stackBase): Deleted. (JSC::MachineThreads::Thread::stackEnd): Deleted. * jit/ICStats.cpp: (JSC::ICStats::ICStats): (JSC::ICStats::~ICStats): * jit/ICStats.h: * jsc.cpp: (functionDollarAgentStart): (startTimeoutThreadIfNeeded): * runtime/JSLock.cpp: (JSC::JSLock::lock): * runtime/JSLock.h: (JSC::JSLock::ownerThread): (JSC::JSLock::currentThreadIsHoldingLock): * runtime/SamplingProfiler.cpp: (JSC::FrameWalker::isValidFramePointer): (JSC::SamplingProfiler::SamplingProfiler): (JSC::SamplingProfiler::createThreadIfNecessary): (JSC::SamplingProfiler::takeSample): * runtime/SamplingProfiler.h: * runtime/VM.h: (JSC::VM::ownerThread): * runtime/VMTraps.cpp: (JSC::findActiveVMAndStackBounds): (JSC::VMTraps::SignalSender::send): (JSC::VMTraps::fireTrap): Source/WebCore: Mechanical change. Use Thread:: APIs. * Modules/indexeddb/server/IDBServer.cpp: (WebCore::IDBServer::IDBServer::IDBServer): * Modules/indexeddb/server/IDBServer.h: * Modules/webaudio/AsyncAudioDecoder.cpp: (WebCore::AsyncAudioDecoder::AsyncAudioDecoder): (WebCore::AsyncAudioDecoder::~AsyncAudioDecoder): (WebCore::AsyncAudioDecoder::runLoop): * Modules/webaudio/AsyncAudioDecoder.h: * Modules/webaudio/OfflineAudioDestinationNode.cpp: (WebCore::OfflineAudioDestinationNode::OfflineAudioDestinationNode): (WebCore::OfflineAudioDestinationNode::uninitialize): (WebCore::OfflineAudioDestinationNode::startRendering): * Modules/webaudio/OfflineAudioDestinationNode.h: * Modules/webdatabase/Database.cpp: (WebCore::Database::securityOrigin): * Modules/webdatabase/DatabaseThread.cpp: (WebCore::DatabaseThread::start): (WebCore::DatabaseThread::databaseThread): (WebCore::DatabaseThread::recordDatabaseOpen): (WebCore::DatabaseThread::recordDatabaseClosed): * Modules/webdatabase/DatabaseThread.h: (WebCore::DatabaseThread::getThreadID): * bindings/js/GCController.cpp: (WebCore::GCController::garbageCollectOnAlternateThreadForDebugging): * fileapi/AsyncFileStream.cpp: (WebCore::callOnFileThread): * loader/icon/IconDatabase.cpp: (WebCore::IconDatabase::open): (WebCore::IconDatabase::close): * loader/icon/IconDatabase.h: * page/ResourceUsageThread.cpp: (WebCore::ResourceUsageThread::createThreadIfNeeded): * page/ResourceUsageThread.h: * page/scrolling/ScrollingThread.cpp: (WebCore::ScrollingThread::ScrollingThread): (WebCore::ScrollingThread::isCurrentThread): (WebCore::ScrollingThread::createThreadIfNeeded): (WebCore::ScrollingThread::threadCallback): * page/scrolling/ScrollingThread.h: * platform/audio/HRTFDatabaseLoader.cpp: (WebCore::HRTFDatabaseLoader::HRTFDatabaseLoader): (WebCore::HRTFDatabaseLoader::loadAsynchronously): (WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion): * platform/audio/HRTFDatabaseLoader.h: * platform/audio/ReverbConvolver.cpp: (WebCore::ReverbConvolver::ReverbConvolver): (WebCore::ReverbConvolver::~ReverbConvolver): * platform/audio/ReverbConvolver.h: * platform/audio/gstreamer/AudioFileReaderGStreamer.cpp: (WebCore::createBusFromAudioFile): (WebCore::createBusFromInMemoryAudioFile): * platform/graphics/gstreamer/WebKitWebSourceGStreamer.cpp: (ResourceHandleStreamingClient::ResourceHandleStreamingClient): (ResourceHandleStreamingClient::~ResourceHandleStreamingClient): * platform/network/cf/LoaderRunLoopCF.cpp: (WebCore::loaderRunLoop): * platform/network/curl/CurlDownload.cpp: (WebCore::CurlDownloadManager::startThreadIfNeeded): (WebCore::CurlDownloadManager::stopThread): * platform/network/curl/CurlDownload.h: * platform/network/curl/SocketStreamHandleImpl.h: * platform/network/curl/SocketStreamHandleImplCurl.cpp: (WebCore::SocketStreamHandleImpl::startThread): (WebCore::SocketStreamHandleImpl::stopThread): * workers/WorkerThread.cpp: (WebCore::WorkerThread::WorkerThread): (WebCore::WorkerThread::start): (WebCore::WorkerThread::workerThread): * workers/WorkerThread.h: (WebCore::WorkerThread::threadID): Source/WebKit: Mechanical change. Use Thread:: APIs. * Storage/StorageThread.cpp: (WebCore::StorageThread::StorageThread): (WebCore::StorageThread::~StorageThread): (WebCore::StorageThread::start): (WebCore::StorageThread::dispatch): (WebCore::StorageThread::terminate): * Storage/StorageThread.h: Source/WebKit2: Mechanical change. Use Thread:: APIs. * NetworkProcess/NetworkProcess.cpp: (WebKit::NetworkProcess::initializeNetworkProcess): * NetworkProcess/cache/NetworkCacheIOChannelSoup.cpp: (WebKit::NetworkCache::IOChannel::readSyncInThread): * Platform/IPC/Connection.cpp: (IPC::Connection::processIncomingMessage): * Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.h: (WebKit::XPCServiceInitializer): * UIProcess/linux/MemoryPressureMonitor.cpp: (WebKit::MemoryPressureMonitor::MemoryPressureMonitor): * WebProcess/WebProcess.cpp: (WebKit::WebProcess::initializeWebProcess): Source/WTF: This patch is refactoring of WTF Threading mechanism to merge JavaScriptCore's threading extension to WTF Threading. Previously, JavaScriptCore requires richer threading features (such as suspending and resuming threads), and they are implemented for PlatformThread in JavaScriptCore. But these features should be implemented in WTF Threading side instead of maintaining JSC's own threading features too. This patch removes these features from JSC and move it to WTF Threading. However, current WTF Threading has one problem: Windows version of WTF Threading has different semantics from Pthreads one. In Windows WTF Threading, we cannot perform any operation after the target thread is detached: WTF Threading stop tracking the state of the thread once the thread is detached. But this is not the same to Pthreads one. In Pthreads, pthread_detach just means that the resource of the thread will be destroyed automatically. While some operations like pthread_join will be rejected, some operations like pthread_kill will be accepted. The problem is that detached thread can be suspended and resumed in JSC. For example, in jsc.cpp, we start the worker thread and detach it immediately. In worker thread, we will create VM and thus concurrent GC will suspend and resume the detached thread. However, in Windows WTF Threading, we have no reference to the detached thread. Thus we cannot perform suspend and resume operations onto the detached thread. To solve the problem, we change Windows Threading mechanism drastically to align it to the Pthread semantics. In the new Threading, we have RefPtr<Thread> class. It holds a handle to a platform thread. We can perform threading operations with this class. For example, Thread::suspend is offered. And we use destructor of the thread local variable to release the resources held by RefPtr<Thread>. In Windows, Thread::detach does nothing because the resource will be destroyed automatically by RefPtr<Thread>. To do so, we introduce ThreadHolder for Windows. This is similar to the previous ThreadIdentifierData for Pthreads. It holds RefPtr<Thread> in the thread local storage (technically, it is Fiber Local Storage in Windows). Thread::current() will return this reference. The problematic situation is that the order of the deallocation of the thread local storage is not defined. So we should not touch thread local storage in the destructor of the thread local storage. To avoid such edge cases, we have currentThread() / Thread::currentID() APIs. They are safe to be called even in the destructor of the other thread local storage. And in Windows, in the FLS destructor, we will create the thread_local variable to defer the destruction of the ThreadHolder. We ensure that this destructor is called after the other FLS destructors are called in Windows 10. This patch is performance neutral. * WTF.xcodeproj/project.pbxproj: * benchmarks/ConditionSpeedTest.cpp: * benchmarks/LockFairnessTest.cpp: * benchmarks/LockSpeedTest.cpp: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/MainThread.h: * wtf/MemoryPressureHandler.h: * wtf/ParallelJobsGeneric.cpp: (WTF::ParallelEnvironment::ThreadPrivate::tryLockFor): (WTF::ParallelEnvironment::ThreadPrivate::workerThread): * wtf/ParallelJobsGeneric.h: (WTF::ParallelEnvironment::ThreadPrivate::ThreadPrivate): Deleted. * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::forEach): * wtf/PlatformRegisters.h: Renamed from Source/JavaScriptCore/runtime/PlatformThread.h. * wtf/RefPtr.h: (WTF::RefPtr::RefPtr): * wtf/ThreadFunctionInvocation.h: (WTF::ThreadFunctionInvocation::ThreadFunctionInvocation): * wtf/ThreadHolder.cpp: Added. (WTF::ThreadHolder::~ThreadHolder): (WTF::ThreadHolder::initialize): * wtf/ThreadHolder.h: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.h. (WTF::ThreadHolder::thread): (WTF::ThreadHolder::ThreadHolder): * wtf/ThreadHolderPthreads.cpp: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.cpp. (WTF::ThreadHolder::initializeOnce): (WTF::ThreadHolder::current): (WTF::ThreadHolder::destruct): * wtf/ThreadHolderWin.cpp: Added. (WTF::threadMapMutex): (WTF::threadMap): (WTF::ThreadHolder::initializeOnce): (WTF::ThreadHolder::current): (WTF::ThreadHolder::destruct): * wtf/ThreadSpecific.h: * wtf/Threading.cpp: (WTF::Thread::normalizeThreadName): (WTF::threadEntryPoint): (WTF::Thread::create): (WTF::Thread::setCurrentThreadIsUserInteractive): (WTF::Thread::setCurrentThreadIsUserInitiated): (WTF::Thread::setGlobalMaxQOSClass): (WTF::Thread::adjustedQOSClass): (WTF::Thread::dump): (WTF::initializeThreading): (WTF::normalizeThreadName): Deleted. (WTF::createThread): Deleted. (WTF::setCurrentThreadIsUserInteractive): Deleted. (WTF::setCurrentThreadIsUserInitiated): Deleted. (WTF::setGlobalMaxQOSClass): Deleted. (WTF::adjustedQOSClass): Deleted. * wtf/Threading.h: (WTF::Thread::id): (WTF::Thread::operator==): (WTF::Thread::operator!=): (WTF::Thread::joinableState): (WTF::Thread::didBecomeDetached): (WTF::Thread::didJoin): (WTF::Thread::hasExited): (WTF::currentThread): * wtf/ThreadingPthreads.cpp: (WTF::Thread::Thread): (WTF::Thread::~Thread): (WTF::Thread::signalHandlerSuspendResume): (WTF::Thread::initializePlatformThreading): (WTF::initializeCurrentThreadEvenIfNonWTFCreated): (WTF::wtfThreadEntryPoint): (WTF::Thread::createInternal): (WTF::Thread::initializeCurrentThreadInternal): (WTF::Thread::changePriority): (WTF::Thread::waitForCompletion): (WTF::Thread::detach): (WTF::Thread::current): (WTF::Thread::currentID): (WTF::Thread::signal): (WTF::Thread::resume): (WTF::Thread::getRegisters): (WTF::Thread::didExit): (WTF::Thread::establish): (WTF::PthreadState::PthreadState): Deleted. (WTF::PthreadState::joinableState): Deleted. (WTF::PthreadState::pthreadHandle): Deleted. (WTF::PthreadState::didBecomeDetached): Deleted. (WTF::PthreadState::didExit): Deleted. (WTF::PthreadState::didJoin): Deleted. (WTF::PthreadState::hasExited): Deleted. (WTF::threadMapMutex): Deleted. (WTF::initializeThreading): Deleted. (WTF::threadMap): Deleted. (WTF::identifierByPthreadHandle): Deleted. (WTF::establishIdentifierForPthreadHandle): Deleted. (WTF::pthreadHandleForIdentifierWithLockAlreadyHeld): Deleted. (WTF::createThreadInternal): Deleted. (WTF::initializeCurrentThreadInternal): Deleted. (WTF::changeThreadPriority): Deleted. (WTF::waitForThreadCompletion): Deleted. (WTF::detachThread): Deleted. (WTF::threadDidExit): Deleted. (WTF::currentThread): Deleted. (WTF::signalThread): Deleted. * wtf/ThreadingWin.cpp: (WTF::Thread::Thread): (WTF::Thread::~Thread): (WTF::Thread::initializeCurrentThreadInternal): (WTF::Thread::initializePlatformThreading): (WTF::wtfThreadEntryPoint): (WTF::Thread::createInternal): (WTF::Thread::changePriority): (WTF::Thread::waitForCompletion): (WTF::Thread::detach): (WTF::Thread::resume): (WTF::Thread::getRegisters): (WTF::Thread::current): (WTF::Thread::currentID): (WTF::Thread::didExit): (WTF::Thread::establish): (WTF::initializeCurrentThreadInternal): Deleted. (WTF::threadMapMutex): Deleted. (WTF::initializeThreading): Deleted. (WTF::threadMap): Deleted. (WTF::storeThreadHandleByIdentifier): Deleted. (WTF::threadHandleForIdentifier): Deleted. (WTF::clearThreadHandleForIdentifier): Deleted. (WTF::createThreadInternal): Deleted. (WTF::changeThreadPriority): Deleted. (WTF::waitForThreadCompletion): Deleted. (WTF::detachThread): Deleted. (WTF::currentThread): Deleted. * wtf/WorkQueue.cpp: (WTF::WorkQueue::concurrentApply): * wtf/WorkQueue.h: * wtf/cocoa/WorkQueueCocoa.cpp: (WTF::dispatchQOSClass): * wtf/generic/WorkQueueGeneric.cpp: (WorkQueue::platformInitialize): (WorkQueue::platformInvalidate): * wtf/linux/MemoryPressureHandlerLinux.cpp: (WTF::MemoryPressureHandler::EventFDPoller::EventFDPoller): (WTF::MemoryPressureHandler::EventFDPoller::~EventFDPoller): * wtf/win/MainThreadWin.cpp: (WTF::initializeMainThreadPlatform): Tools: Mechanical change. Use Thread:: APIs. * DumpRenderTree/JavaScriptThreading.cpp: (runJavaScriptThread): (startJavaScriptThreads): (stopJavaScriptThreads): * DumpRenderTree/mac/DumpRenderTree.mm: (testThreadIdentifierMap): * TestWebKitAPI/Tests/WTF/Condition.cpp: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/187690@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@215265 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-04-12 12:08:29 +00:00
forEachImpl(scopedLambdaRef<void(Thread&, const void*)>(func));
Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression https://bugs.webkit.org/show_bug.cgi?id=160125 Reviewed by Geoffrey Garen and Keith Miller. JSTests: Most of the things I did properly covered by existing tests, but I found some simple cases of unshifting that had sketchy coverage. * stress/array-storage-array-unshift.js: Added. * stress/contiguous-array-unshift.js: Added. * stress/double-array-unshift.js: Added. * stress/int32-array-unshift.js: Added. Source/bmalloc: I needed to tryMemalign, so I added such a thing. * bmalloc/Allocator.cpp: (bmalloc::Allocator::allocate): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): * bmalloc/Allocator.h: * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): * bmalloc/bmalloc.h: (bmalloc::api::tryMemalign): Source/JavaScriptCore: In order to make the GC concurrent (bug 149432), we would either need to enable concurrent copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead from the barriers alone. Considering that MarkedSpace does a decent job of avoiding fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want to get rid of copied space. This change moves copied space's biggest client over to marked space. Moving butterflies to marked space means having them use the new Auxiliary HeapCell allocation path. This is a fairly mechanical change, but it caused performance regressions everywhere, so this change also fixes MarkedSpace's performance issues. At a high level the mechanical changes are: - We use AuxiliaryBarrier instead of CopyBarrier. - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly CheckedBoolean stuff, since it's so much more trouble than it's worth. - The JITs have to emit inlined marked space allocations instead of inline copy space allocations. - Everyone has to get used to zeroing their butterflies after allocation instead of relying on them being pre-zeroed by the GC. Copied space would zero things for you, while marked space doesn't. That's about 1/3 of this change. But this led to performance problems, which I fixed with optimizations that amounted to a major MarkedSpace rewrite: - MarkedSpace always causes internal fragmentation for array allocations because the vector length we choose when we resize usually leads to a cell size that doesn't correspond to any size class. I got around this by making array allocations usually round up vectorLength to the maximum allowed by the size class that we would have allocated in. Also, ensureLengthSlow() and friends first make sure that the requested length can't just be fulfilled with the current allocation size. This safeguard means that not every array allocation has to do size class queries. For example, the fast path of new Array(length) never does any size class queries, under the assumption that (1) the speed gained from avoiding an ensureLengthSlow() call, which then just changes the vectorLength by doing the size class query, is too small to offset the speed lost by doing the query on every allocation and (2) new Array(length) is a pretty good hint that resizing is not very likely. - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This changes MarkedSpace size classes to use a linear progression for very small sizes followed by a geometric progression that naturally transitions to a hyperbolic progression. We want hyperbolic sizes when we get close to blockSize: for example the largest size we want is payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The next size down should be payloadSize / 3 rounded down, and so on. After the last precise size (80 bytes), we proceed using a geometric progression, but round up each size to minimize slop at the end of the block. This naturally causes the geometric progression to turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so it can be controlled with runtime options. I found that a base of 1.4 works pretty well. - Large allocations caused massive internal fragmentation, since the smallest large allocation had to use exactly blockSize, and the largest small allocation used blockSize / 2. The next size up - the first large allocation size to require two blocks - also had 50% internal fragmentation. This is because we required large allocations to be blockSize aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that. Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large allocations are deliberately misaligned by 8 bytes. This actually works out great since (1) typed arrays won't use large allocations anyway since they have their own malloc fallback and (2) large array butterflies already have a 8 byte header, which means that the 8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took extreme care to make sure that the isLargeAllocation bit checks are as rare as possible; for example, ExecState::vm() skips the check because we know that callees must be small allocations. It's also possible to use template tricks to do one check for cell container kind, and then invoke a function specialized for MarkedBlock or a function specialized for LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used from functions that are template-specialized like this. That's mostly to speed up the GC marking code. Most other code can use CellContainer API or HeapCell API directly. That's another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(), HeapCell::isLargeAllocation(), and HeapCell::cellContainer(). The size cutoff for large allocations is runtime-configurable, so long as you don't choose something so small that callees end up large. I found that 400 bytes is roughly optimal. This means that the MarkedBlock size classes end up being: 16, 32, 48, 64, 80, 112, 160, 224, 320 The next size class would have been 432, but that's above the 400 byte cutoff. All of this is configurable with --sizeClassProgression and --largeAllocationCutoff. You can see what size classes you end up with by doing --dumpSizeClasses=true. - Copied space uses 64KB blocks, while marked space used to use 16KB blocks. Allocating a lot of stuff in 16KB blocks was slower than allocating it in 64KB blocks because the GC had a lot of per-block overhead. I removed this overhead: It's now 2x faster to scan all MarkedBlocks because the list that contains the interesting meta-data is allocated on the side, for better locality during a sequential walk. It's no longer necessary to scan MarkedBlocks to find WeakSets, since the sets of WeakSets for eden scan and full scan are maintained on-the-fly. It's no longer necessary to scan all MarkedBlocks to clear mark bits because we now use versioned mark bits: to clear then, just increment the 64-bit heap version. It's no longer necessary to scan retired MarkedBlocks while allocating because marking retires them on-the-fly. It's no longer necessary to sort all blocks in the IncrementalSweeper's snapshot because blocks now know if they are in the snapshot. Put together, these optimizations allowed me to reduce block size to 16KB without losing much performance. There is some small perf loss on JetStream/splay, but not enough to hurt JetStream overall. I tried reducing block sizes further, to 4KB, since that is a progression on membuster. That's not possible yet, since there is still enough per-block overhead yet that such a reduction hurts JetStream too much. I filed a bug about improving this further: https://bugs.webkit.org/show_bug.cgi?id=161581. - Even after all of that, copying butterflies was still faster because it allowed us to skip sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them, so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much larger than O(live), especially in an eden collection. Copying satisfies this premise while mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires that we walk the block to build a free list. This means walking dead space. The new allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead. The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump pointer always bumps by cellSize, so the result of filling a block with bumping looks as if we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to quickly tell if the block is entirely free. This makes sweeping O(1) whenever a MarkedBlock is completely empty, which is the common case because of the generational hypothesis: the number of objects that survive an eden collection is a tiny fraction of the number of objects that had been allocated, and this fraction is so small that there are typically fewer than one survivors per MarkedBlock. This change was enough to make this change a net win over tip-of-tree. - FTL now shares the same allocation fast paths as everything else, which is great, because bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that control flow, since it won't be able to improve the machine code we write ourselves. GC fast paths are best written in assembly. So, I've empowered B3 to have even better support for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void. So, the new FTL allocation fast paths are just Patchpoint terminals that call through to AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the size class calculation and constant-hoisting the allocator. Also, I gave the FTL the ability to constant-fold some allocator logic (in case we first assume that we're doing a variable-length allocation but then realize that the length is known). I think it makes sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is, since this makes lowering easier (you can constant fold during lowering more easily) and it reduces the amount of malloc traffic. In the future, we could teach B3 how to better constant-fold this code. That would require allowing loads to be constant-folded, which is doable but hella tricky. - It used to be that if a logical object allocation required two physical allocations (first the butterfly and then the cell), then the JIT would emit the code in such a way that a failure in the second fast path would cause us to forget the successful first physical allocation. This was pointlessly wasteful. It turns out that it's very cheap to devote a register to storing either the butterfly or null, because the butterfly register is anyway going to be free inside the first allocation. The only overhead here is zeroing the butterfly register. With that in place, we can just pass the butterfly-or-null to the slow path, which can then either allocate a butterfly or not. So now we never waste a successful allocation. This patch implements such a solution both in DFG (where it's easy to do this since we control registers already) and in FTL (where it's annoying, because mutable "butterfly-or-null" variables are hard to say in SSA; also I realized that we had code duplicated the JSArray allocation utility, so I deduplicated it). This came up because in one version of this patch, this wastage would resonate with some Kraken benchmark: the benchmark would always allocate N small things followed by one bigger thing. The problem was I accidentally adjusted the various fixed overheads in MarkedBlock in such a way that the JSObject size class, which both the small and big thing shared for their cell, could hold exactly N cells per MarkedBlock. Then the benchmark would always call slow path when it allocated the big thing. So, it would end up having to allocate the big thing's large butterfly twice, every single time! Ouch! - It used to be that we zeroed CopiedBlocks using memset, and so array allocations enjoyed amortization of the cost of zeroing. This doesn't work anymore - it's now up to the client of the allocator to initialize the object to whatever state they need. It used to be that we would just use a dumb loop. I initially changed this so that we would end up in memset for large allocations, but this didn't actually help performance that much. I got a much better result by playing with different memsets written in assembly. First I wrote one using non-temporal stores. That was a small speed-up over memset. Then I tried the classic "rep stos" approach, and holy cow that version was fast. It's a ~20% speed-up on array allocation microbenchmarks. So, this patch adds code paths to do "rep stos" on x86_64, or memset, or use a loop, as appropriate, for both "contiguous" arrays (holes are zero) and double arrays (holes are PNaN). Note that the JIT always emits either a loop or a flat slab of stores (if the size is known), but those paths in the JIT won't trigger for NewArrayWithSize() if the size is large, since that takes us to the operationNewArrayWithSize() slow path, which calls into JSArray::create(). That's why the optimizations here are all in JSArray::create() - that's the hot place for large arrays that need to be filled with holes. All of this put together gives us neutral perf on JetStream, membuster, and PLT3, a ~1% regression on Speedometer, and up to a 4% regression Kraken. The Kraken regression is because Kraken was allocating exactly 1024 element arrays at a rate of 400MB/sec. This is a best-case scenario for bump allocation. I think that we should fix bmalloc to make up the difference, but take the hit for now because it's a crazy corner case. By comparison, the alternative approach of using a copy barrier would have cost us 1-2%. That's the real apples-to-apples comparison if your premise is that we should have a concurrent GC. After we finish removing copied space, we will be barrier-ready for concurrent GC: we already have a marking barrier and we simply won't need a copying barrier. This change gets us there for the purposes of our benchmarks, since the remaining clients of copied space are not very important. On the other hand, if we keep copying, then getting barrier-ready would mean adding back the copy barrier, which costs more perf. We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving typed arrays and a few other weird things over to Aux MarkedSpace. This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell, and CellContainer meant that I had to include those files from everywhere. Fortunately, just including JSCInlines.h (instead of manually including the files that includes) is usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something that we were already basically doing. In places where JSCInlines.h would be too much, I just included HeapInlines.h. This got weird, because we previously included HeapInlines.h from JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that meant having to manually include HeapInlines.h from the places that previously got it implicitly via JSObject.h. But that led to more problems for some reason: I started getting build errors because non-JSC files were having trouble including Opcode.h. That's just silly, since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and made it impossible to include it from outside JSC. This was a lot of work, but it was necessary to get the patch to build on all ports. It's also a net win. There were many places in WebCore that were transitively including a *ton* of JSC headers just because of the JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons. * API/JSManagedValue.mm: (-[JSManagedValue initWithValue:]): * API/JSTypedArray.cpp: * API/ObjCCallbackFunction.mm: * API/tests/testapi.mm: (testObjectiveCAPI): (testWeakValue): Deleted. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * Scripts/builtins/builtins_generate_combined_implementation.py: (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py: (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes): * Scripts/builtins/builtins_generate_separate_implementation.py: (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes): * assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::JumpList::link): (JSC::AbstractMacroAssembler::JumpList::linkTo): * assembler/MacroAssembler.h: * assembler/MacroAssemblerARM64.h: (JSC::MacroAssemblerARM64::add32): * assembler/MacroAssemblerCodeRef.cpp: Added. (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): (JSC::MacroAssemblerCodePtr::dumpWithName): (JSC::MacroAssemblerCodePtr::dump): (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): (JSC::MacroAssemblerCodeRef::dump): * assembler/MacroAssemblerCodeRef.h: (JSC::MacroAssemblerCodePtr::createLLIntCodePtr): Deleted. (JSC::MacroAssemblerCodePtr::dumpWithName): Deleted. (JSC::MacroAssemblerCodePtr::dump): Deleted. (JSC::MacroAssemblerCodeRef::createLLIntCodeRef): Deleted. (JSC::MacroAssemblerCodeRef::dump): Deleted. * b3/B3BasicBlock.cpp: (JSC::B3::BasicBlock::appendBoolConstant): * b3/B3BasicBlock.h: * b3/B3DuplicateTails.cpp: * b3/B3StackmapGenerationParams.h: * b3/testb3.cpp: (JSC::B3::testPatchpointTerminalReturnValue): (JSC::B3::run): * bindings/ScriptValue.cpp: * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeLivenessAnalysis.cpp: * bytecode/BytecodeUseDef.h: * bytecode/CallLinkInfo.cpp: (JSC::CallLinkInfo::callTypeFor): * bytecode/CallLinkInfo.h: (JSC::CallLinkInfo::callTypeFor): Deleted. * bytecode/CallLinkStatus.cpp: * bytecode/CodeBlock.cpp: (JSC::CodeBlock::finishCreation): (JSC::CodeBlock::clearLLIntGetByIdCache): (JSC::CodeBlock::predictedMachineCodeSize): * bytecode/CodeBlock.h: (JSC::CodeBlock::jitCodeMap): Deleted. (JSC::clearLLIntGetByIdCache): Deleted. * bytecode/ExecutionCounter.h: * bytecode/Instruction.h: * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp: (JSC::LLIntPrototypeLoadAdaptiveStructureWatchpoint::fireInternal): * bytecode/ObjectAllocationProfile.h: (JSC::ObjectAllocationProfile::isNull): (JSC::ObjectAllocationProfile::initialize): * bytecode/Opcode.h: (JSC::padOpcodeName): * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): (JSC::PolymorphicAccess::regenerate): * bytecode/PolymorphicAccess.h: * bytecode/PreciseJumpTargets.cpp: * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecode/UnlinkedCodeBlock.cpp: (JSC::UnlinkedCodeBlock::vm): Deleted. * bytecode/UnlinkedCodeBlock.h: * bytecode/UnlinkedInstructionStream.cpp: * bytecode/UnlinkedInstructionStream.h: * dfg/DFGOperations.cpp: * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT.h: (JSC::DFG::SpeculativeJIT::emitAllocateJSCell): (JSC::DFG::SpeculativeJIT::emitAllocateJSObject): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): * dfg/DFGStrengthReductionPhase.cpp: (JSC::DFG::StrengthReductionPhase::handleNode): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCompile.cpp: * ftl/FTLJITFinalizer.cpp: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments): (JSC::FTL::DFG::LowerDFGToB3::compileCreateRest): (JSC::FTL::DFG::LowerDFGToB3::allocateArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize): (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell): (JSC::FTL::DFG::LowerDFGToB3::allocateCell): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::compileAllocateArrayWithSize): Deleted. * ftl/FTLOutput.cpp: (JSC::FTL::Output::constBool): (JSC::FTL::Output::add): (JSC::FTL::Output::shl): (JSC::FTL::Output::aShr): (JSC::FTL::Output::lShr): (JSC::FTL::Output::zeroExt): (JSC::FTL::Output::equal): (JSC::FTL::Output::notEqual): (JSC::FTL::Output::above): (JSC::FTL::Output::aboveOrEqual): (JSC::FTL::Output::below): (JSC::FTL::Output::belowOrEqual): (JSC::FTL::Output::greaterThan): (JSC::FTL::Output::greaterThanOrEqual): (JSC::FTL::Output::lessThan): (JSC::FTL::Output::lessThanOrEqual): (JSC::FTL::Output::select): (JSC::FTL::Output::appendSuccessor): (JSC::FTL::Output::addIncomingToPhi): * ftl/FTLOutput.h: * ftl/FTLValueFromBlock.h: (JSC::FTL::ValueFromBlock::operator bool): (JSC::FTL::ValueFromBlock::ValueFromBlock): Deleted. * ftl/FTLWeightedTarget.h: (JSC::FTL::WeightedTarget::frequentedBlock): * heap/CellContainer.h: Added. (JSC::CellContainer::CellContainer): (JSC::CellContainer::operator bool): (JSC::CellContainer::isMarkedBlock): (JSC::CellContainer::isLargeAllocation): (JSC::CellContainer::markedBlock): (JSC::CellContainer::largeAllocation): * heap/CellContainerInlines.h: Added. (JSC::CellContainer::isMarked): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::noteMarked): (JSC::CellContainer::cellSize): (JSC::CellContainer::weakSet): (JSC::CellContainer::flipIfNecessary): * heap/ConservativeRoots.cpp: (JSC::ConservativeRoots::ConservativeRoots): (JSC::ConservativeRoots::~ConservativeRoots): (JSC::ConservativeRoots::grow): (JSC::ConservativeRoots::genericAddPointer): (JSC::ConservativeRoots::genericAddSpan): * heap/ConservativeRoots.h: (JSC::ConservativeRoots::roots): * heap/CopyToken.h: * heap/FreeList.cpp: Added. (JSC::FreeList::dump): * heap/FreeList.h: Added. (JSC::FreeList::FreeList): (JSC::FreeList::list): (JSC::FreeList::bump): (JSC::FreeList::operator==): (JSC::FreeList::operator!=): (JSC::FreeList::operator bool): (JSC::FreeList::allocationWillFail): (JSC::FreeList::allocationWillSucceed): * heap/GCTypeMap.h: Added. (JSC::GCTypeMap::operator[]): * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalizeUnconditionalFinalizers): (JSC::Heap::markRoots): (JSC::Heap::copyBackingStores): (JSC::Heap::gatherStackRoots): (JSC::Heap::gatherJSStackRoots): (JSC::Heap::gatherScratchBufferRoots): (JSC::Heap::clearLivenessData): (JSC::Heap::visitSmallStrings): (JSC::Heap::visitConservativeRoots): (JSC::Heap::removeDeadCompilerWorklistEntries): (JSC::Heap::gatherExtraHeapSnapshotData): (JSC::Heap::removeDeadHeapSnapshotNodes): (JSC::Heap::visitProtectedObjects): (JSC::Heap::visitArgumentBuffers): (JSC::Heap::visitException): (JSC::Heap::visitStrongHandles): (JSC::Heap::visitHandleStack): (JSC::Heap::visitSamplingProfiler): (JSC::Heap::traceCodeBlocksAndJITStubRoutines): (JSC::Heap::converge): (JSC::Heap::visitWeakHandles): (JSC::Heap::updateObjectCounts): (JSC::Heap::clearUnmarkedExecutables): (JSC::Heap::deleteUnmarkedCompiledCode): (JSC::Heap::collectAllGarbage): (JSC::Heap::collect): (JSC::Heap::collectWithoutAnySweep): (JSC::Heap::collectImpl): (JSC::Heap::suspendCompilerThreads): (JSC::Heap::willStartCollection): (JSC::Heap::flushOldStructureIDTables): (JSC::Heap::flushWriteBarrierBuffer): (JSC::Heap::stopAllocation): (JSC::Heap::prepareForMarking): (JSC::Heap::reapWeakHandles): (JSC::Heap::pruneStaleEntriesFromWeakGCMaps): (JSC::Heap::sweepArrayBuffers): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): (JSC::MarkedBlockSnapshotFunctor::operator()): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::deleteSourceProviderCaches): (JSC::Heap::notifyIncrementalSweeper): (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks): (JSC::Heap::resetAllocators): (JSC::Heap::updateAllocationLimits): (JSC::Heap::didFinishCollection): (JSC::Heap::resumeCompilerThreads): (JSC::Zombify::visit): (JSC::Heap::forEachCodeBlockImpl): * heap/Heap.h: (JSC::Heap::allocatorForObjectWithoutDestructor): (JSC::Heap::allocatorForObjectWithDestructor): (JSC::Heap::allocatorForAuxiliaryData): (JSC::Heap::jitStubRoutines): (JSC::Heap::codeBlockSet): (JSC::Heap::storageAllocator): Deleted. * heap/HeapCell.h: (JSC::HeapCell::isZapped): Deleted. * heap/HeapCellInlines.h: Added. (JSC::HeapCell::isLargeAllocation): (JSC::HeapCell::cellContainer): (JSC::HeapCell::markedBlock): (JSC::HeapCell::largeAllocation): (JSC::HeapCell::heap): (JSC::HeapCell::vm): (JSC::HeapCell::cellSize): (JSC::HeapCell::allocatorAttributes): (JSC::HeapCell::destructionMode): (JSC::HeapCell::cellKind): * heap/HeapInlines.h: (JSC::Heap::heap): (JSC::Heap::isLive): (JSC::Heap::isMarked): (JSC::Heap::testAndSetMarked): (JSC::Heap::setMarked): (JSC::Heap::cellSize): (JSC::Heap::forEachCodeBlock): (JSC::Heap::allocateObjectOfType): (JSC::Heap::subspaceForObjectOfType): (JSC::Heap::allocatorForObjectOfType): (JSC::Heap::allocateAuxiliary): (JSC::Heap::tryAllocateAuxiliary): (JSC::Heap::tryReallocateAuxiliary): (JSC::Heap::isPointerGCObject): Deleted. (JSC::Heap::isValueGCObject): Deleted. * heap/HeapOperation.cpp: Added. (WTF::printInternal): * heap/HeapOperation.h: * heap/HeapUtil.h: Added. (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): (JSC::HeapUtil::isValueGCObject): * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::sweepNextBlock): * heap/IncrementalSweeper.h: * heap/LargeAllocation.cpp: Added. (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::lastChanceToFinalize): (JSC::LargeAllocation::shrink): (JSC::LargeAllocation::visitWeakSet): (JSC::LargeAllocation::reapWeakSet): (JSC::LargeAllocation::flip): (JSC::LargeAllocation::isEmpty): (JSC::LargeAllocation::sweep): (JSC::LargeAllocation::destroy): (JSC::LargeAllocation::dump): * heap/LargeAllocation.h: Added. (JSC::LargeAllocation::fromCell): (JSC::LargeAllocation::cell): (JSC::LargeAllocation::isLargeAllocation): (JSC::LargeAllocation::heap): (JSC::LargeAllocation::vm): (JSC::LargeAllocation::weakSet): (JSC::LargeAllocation::clearNewlyAllocated): (JSC::LargeAllocation::isNewlyAllocated): (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::isLive): (JSC::LargeAllocation::hasValidCell): (JSC::LargeAllocation::cellSize): (JSC::LargeAllocation::aboveLowerBound): (JSC::LargeAllocation::belowUpperBound): (JSC::LargeAllocation::contains): (JSC::LargeAllocation::attributes): (JSC::LargeAllocation::flipIfNecessary): (JSC::LargeAllocation::flipIfNecessaryConcurrently): (JSC::LargeAllocation::testAndSetMarked): (JSC::LargeAllocation::setMarked): (JSC::LargeAllocation::clearMarked): (JSC::LargeAllocation::noteMarked): (JSC::LargeAllocation::headerSize): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::retire): (JSC::MarkedAllocator::filterNextBlock): (JSC::MarkedAllocator::setNextBlockToSweep): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::blockHeaderSize): (JSC::MarkedAllocator::blockSizeForBytes): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::reset): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::setFreeList): (JSC::isListPagedOut): Deleted. (JSC::MarkedAllocator::tryAllocateHelper): Deleted. (JSC::MarkedAllocator::tryPopFreeList): Deleted. (JSC::MarkedAllocator::tryAllocate): Deleted. (JSC::MarkedAllocator::allocateBlock): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::takeLastActiveBlock): (JSC::MarkedAllocator::offsetOfFreeList): (JSC::MarkedAllocator::offsetOfCellSize): (JSC::MarkedAllocator::tryAllocate): (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted. (JSC::MarkedAllocator::MarkedAllocator): Deleted. (JSC::MarkedAllocator::init): Deleted. (JSC::MarkedAllocator::stopAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::SetNewlyAllocatedFunctor::SetNewlyAllocatedFunctor): (JSC::SetNewlyAllocatedFunctor::operator()): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::Handle::zap): (JSC::MarkedBlock::Handle::forEachFreeCell): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessarySlow): (JSC::MarkedBlock::flipIfNecessaryConcurrentlySlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::needsFlip): (JSC::MarkedBlock::Handle::needsFlip): (JSC::MarkedBlock::Handle::willRemoveBlock): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::clearHasAnyMarked): (JSC::MarkedBlock::noteMarkedSlow): (WTF::printInternal): (JSC::MarkedBlock::create): Deleted. (JSC::MarkedBlock::destroy): Deleted. (JSC::MarkedBlock::callDestructor): Deleted. (JSC::MarkedBlock::specializedSweep): Deleted. (JSC::MarkedBlock::sweep): Deleted. (JSC::MarkedBlock::sweepHelper): Deleted. (JSC::MarkedBlock::stopAllocating): Deleted. (JSC::MarkedBlock::clearMarksWithCollectionType): Deleted. (JSC::MarkedBlock::lastChanceToFinalize): Deleted. (JSC::MarkedBlock::resumeAllocating): Deleted. (JSC::MarkedBlock::didRetireBlock): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::VoidFunctor::returnValue): (JSC::MarkedBlock::CountFunctor::CountFunctor): (JSC::MarkedBlock::CountFunctor::count): (JSC::MarkedBlock::CountFunctor::returnValue): (JSC::MarkedBlock::Handle::hasAnyNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): (JSC::MarkedBlock::Handle::state): (JSC::MarkedBlock::needsDestruction): (JSC::MarkedBlock::handle): (JSC::MarkedBlock::Handle::block): (JSC::MarkedBlock::firstAtom): (JSC::MarkedBlock::atoms): (JSC::MarkedBlock::isAtomAligned): (JSC::MarkedBlock::Handle::cellAlign): (JSC::MarkedBlock::blockFor): (JSC::MarkedBlock::Handle::allocator): (JSC::MarkedBlock::Handle::heap): (JSC::MarkedBlock::Handle::vm): (JSC::MarkedBlock::vm): (JSC::MarkedBlock::Handle::weakSet): (JSC::MarkedBlock::weakSet): (JSC::MarkedBlock::Handle::shrink): (JSC::MarkedBlock::Handle::visitWeakSet): (JSC::MarkedBlock::Handle::reapWeakSet): (JSC::MarkedBlock::Handle::cellSize): (JSC::MarkedBlock::cellSize): (JSC::MarkedBlock::Handle::attributes): (JSC::MarkedBlock::attributes): (JSC::MarkedBlock::Handle::needsDestruction): (JSC::MarkedBlock::Handle::destruction): (JSC::MarkedBlock::Handle::cellKind): (JSC::MarkedBlock::Handle::markCount): (JSC::MarkedBlock::Handle::size): (JSC::MarkedBlock::atomNumber): (JSC::MarkedBlock::flipIfNecessary): (JSC::MarkedBlock::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipIfNecessary): (JSC::MarkedBlock::Handle::flipIfNecessaryConcurrently): (JSC::MarkedBlock::Handle::flipForEdenCollection): (JSC::MarkedBlock::assertFlipped): (JSC::MarkedBlock::Handle::assertFlipped): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::testAndSetMarked): (JSC::MarkedBlock::Handle::isNewlyAllocated): (JSC::MarkedBlock::Handle::setNewlyAllocated): (JSC::MarkedBlock::Handle::clearNewlyAllocated): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::isAtom): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::Handle::needsSweeping): (JSC::MarkedBlock::Handle::isAllocated): (JSC::MarkedBlock::Handle::isMarked): (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::hasAnyMarked): (JSC::MarkedBlock::noteMarked): (WTF::MarkedBlockHash::hash): (JSC::MarkedBlock::FreeList::FreeList): Deleted. (JSC::MarkedBlock::allocator): Deleted. (JSC::MarkedBlock::heap): Deleted. (JSC::MarkedBlock::shrink): Deleted. (JSC::MarkedBlock::visitWeakSet): Deleted. (JSC::MarkedBlock::reapWeakSet): Deleted. (JSC::MarkedBlock::willRemoveBlock): Deleted. (JSC::MarkedBlock::didConsumeFreeList): Deleted. (JSC::MarkedBlock::markCount): Deleted. (JSC::MarkedBlock::isEmpty): Deleted. (JSC::MarkedBlock::destruction): Deleted. (JSC::MarkedBlock::cellKind): Deleted. (JSC::MarkedBlock::size): Deleted. (JSC::MarkedBlock::capacity): Deleted. (JSC::MarkedBlock::setMarked): Deleted. (JSC::MarkedBlock::clearMarked): Deleted. (JSC::MarkedBlock::isNewlyAllocated): Deleted. (JSC::MarkedBlock::setNewlyAllocated): Deleted. (JSC::MarkedBlock::clearNewlyAllocated): Deleted. (JSC::MarkedBlock::isLive): Deleted. (JSC::MarkedBlock::isLiveCell): Deleted. (JSC::MarkedBlock::forEachCell): Deleted. (JSC::MarkedBlock::forEachLiveCell): Deleted. (JSC::MarkedBlock::forEachDeadCell): Deleted. (JSC::MarkedBlock::needsSweeping): Deleted. (JSC::MarkedBlock::isAllocated): Deleted. (JSC::MarkedBlock::isMarkedOrRetired): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::initializeSizeClassForStepSize): (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::lastChanceToFinalize): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::allocateLarge): (JSC::MarkedSpace::tryAllocateLarge): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::zombifySweep): (JSC::MarkedSpace::resetAllocators): (JSC::MarkedSpace::visitWeakSets): (JSC::MarkedSpace::reapWeakSets): (JSC::MarkedSpace::stopAllocating): (JSC::MarkedSpace::prepareForMarking): (JSC::MarkedSpace::resumeAllocating): (JSC::MarkedSpace::isPagedOut): (JSC::MarkedSpace::freeBlock): (JSC::MarkedSpace::freeOrShrinkBlock): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::VerifyMarked::operator()): (JSC::MarkedSpace::flip): (JSC::MarkedSpace::objectCount): (JSC::MarkedSpace::size): (JSC::MarkedSpace::capacity): (JSC::MarkedSpace::addActiveWeakSet): (JSC::MarkedSpace::didAddBlock): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::forEachAllocator): Deleted. (JSC::VerifyMarkedOrRetired::operator()): Deleted. (JSC::MarkedSpace::clearMarks): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::sizeClassToIndex): (JSC::MarkedSpace::indexToSizeClass): (JSC::MarkedSpace::version): (JSC::MarkedSpace::blocksWithNewObjects): (JSC::MarkedSpace::largeAllocations): (JSC::MarkedSpace::largeAllocationsNurseryOffset): (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection): (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin): (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd): (JSC::MarkedSpace::largeAllocationsForThisCollectionSize): (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): (JSC::MarkedSpace::allocatorFor): (JSC::MarkedSpace::destructorAllocatorFor): (JSC::MarkedSpace::auxiliaryAllocatorFor): (JSC::MarkedSpace::allocateWithoutDestructor): (JSC::MarkedSpace::allocateWithDestructor): (JSC::MarkedSpace::allocateAuxiliary): (JSC::MarkedSpace::tryAllocateAuxiliary): (JSC::MarkedSpace::forEachBlock): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::optimalSizeFor): (JSC::MarkedSpace::didAddBlock): Deleted. (JSC::MarkedSpace::didAllocateInBlock): Deleted. (JSC::MarkedSpace::objectCount): Deleted. (JSC::MarkedSpace::size): Deleted. (JSC::MarkedSpace::capacity): Deleted. * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::didStartMarking): (JSC::SlotVisitor::reset): (JSC::SlotVisitor::append): (JSC::SlotVisitor::appendJSCellOrAuxiliary): (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::appendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::noteLiveAuxiliaryCell): (JSC::SlotVisitor::visitChildren): * heap/SlotVisitor.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::WeakBlock): (JSC::WeakBlock::visit): (JSC::WeakBlock::reap): * heap/WeakBlock.h: (JSC::WeakBlock::disconnectContainer): (JSC::WeakBlock::disconnectMarkedBlock): Deleted. * heap/WeakSet.cpp: (JSC::WeakSet::~WeakSet): (JSC::WeakSet::sweep): (JSC::WeakSet::shrink): (JSC::WeakSet::addAllocator): * heap/WeakSet.h: (JSC::WeakSet::container): (JSC::WeakSet::setContainer): (JSC::WeakSet::WeakSet): (JSC::WeakSet::visit): (JSC::WeakSet::shrink): Deleted. * heap/WeakSetInlines.h: (JSC::WeakSet::allocate): * inspector/InjectedScriptManager.cpp: * inspector/JSGlobalObjectInspectorController.cpp: * inspector/JSJavaScriptCallFrame.cpp: * inspector/ScriptDebugServer.cpp: * inspector/agents/InspectorDebuggerAgent.cpp: * interpreter/CachedCall.h: (JSC::CachedCall::CachedCall): * interpreter/Interpreter.cpp: (JSC::loadVarargs): (JSC::StackFrame::sourceID): Deleted. (JSC::StackFrame::sourceURL): Deleted. (JSC::StackFrame::functionName): Deleted. (JSC::StackFrame::computeLineAndColumn): Deleted. (JSC::StackFrame::toString): Deleted. * interpreter/Interpreter.h: (JSC::StackFrame::isNative): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): (JSC::AssemblyHelpers::emitAllocate): (JSC::AssemblyHelpers::emitAllocateJSCell): (JSC::AssemblyHelpers::emitAllocateJSObject): (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): * jit/GCAwareJITStubRoutine.cpp: (JSC::GCAwareJITStubRoutine::GCAwareJITStubRoutine): * jit/JIT.cpp: (JSC::JIT::compileCTINativeCall): (JSC::JIT::link): * jit/JIT.h: (JSC::JIT::compileCTINativeCall): Deleted. * jit/JITExceptions.cpp: (JSC::genericUnwind): * jit/JITExceptions.h: * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOpcodes32_64.cpp: (JSC::JIT::emit_op_new_object): (JSC::JIT::emitSlow_op_new_object): (JSC::JIT::emit_op_create_this): (JSC::JIT::emitSlow_op_create_this): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emitWriteBarrier): * jit/JITThunks.cpp: * jit/JITThunks.h: * jsc.cpp: (functionDescribeArray): (main): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntExceptions.cpp: * llint/LLIntThunks.cpp: * llint/LLIntThunks.h: * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter.cpp: * llint/LowLevelInterpreter32_64.asm: * llint/LowLevelInterpreter64.asm: * parser/ModuleAnalyzer.cpp: * parser/NodeConstructors.h: * parser/Nodes.h: * profiler/ProfilerBytecode.cpp: * profiler/ProfilerBytecode.h: * profiler/ProfilerBytecodeSequence.cpp: * runtime/ArrayConventions.h: (JSC::indexingHeaderForArrayStorage): (JSC::baseIndexingHeaderForArrayStorage): (JSC::indexingHeaderForArray): Deleted. (JSC::baseIndexingHeaderForArray): Deleted. * runtime/ArrayPrototype.cpp: (JSC::arrayProtoFuncSplice): (JSC::concatAppendOne): (JSC::arrayProtoPrivateFuncConcatMemcpy): * runtime/ArrayStorage.h: (JSC::ArrayStorage::vectorLength): (JSC::ArrayStorage::totalSizeFor): (JSC::ArrayStorage::totalSize): (JSC::ArrayStorage::availableVectorLength): (JSC::ArrayStorage::optimalVectorLength): (JSC::ArrayStorage::sizeFor): Deleted. * runtime/AuxiliaryBarrier.h: Added. (JSC::AuxiliaryBarrier::AuxiliaryBarrier): (JSC::AuxiliaryBarrier::clear): (JSC::AuxiliaryBarrier::get): (JSC::AuxiliaryBarrier::slot): (JSC::AuxiliaryBarrier::operator bool): (JSC::AuxiliaryBarrier::setWithoutBarrier): * runtime/AuxiliaryBarrierInlines.h: Added. (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier): (JSC::AuxiliaryBarrier<T>::set): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::availableContiguousVectorLength): (JSC::Butterfly::optimalContiguousVectorLength): (JSC::Butterfly::createUninitialized): (JSC::Butterfly::growArrayRight): * runtime/ClonedArguments.cpp: (JSC::ClonedArguments::createEmpty): * runtime/CommonSlowPathsExceptions.cpp: * runtime/CommonSlowPathsExceptions.h: * runtime/DataView.cpp: * runtime/DirectArguments.h: * runtime/ECMAScriptSpecInternalFunctions.cpp: * runtime/Error.cpp: * runtime/Error.h: * runtime/ErrorInstance.cpp: * runtime/ErrorInstance.h: * runtime/Exception.cpp: * runtime/Exception.h: * runtime/GeneratorFrame.cpp: * runtime/GeneratorPrototype.cpp: * runtime/InternalFunction.cpp: (JSC::InternalFunction::InternalFunction): * runtime/IntlCollator.cpp: * runtime/IntlCollatorConstructor.cpp: * runtime/IntlCollatorPrototype.cpp: * runtime/IntlDateTimeFormat.cpp: * runtime/IntlDateTimeFormatConstructor.cpp: * runtime/IntlDateTimeFormatPrototype.cpp: * runtime/IntlNumberFormat.cpp: * runtime/IntlNumberFormatConstructor.cpp: * runtime/IntlNumberFormatPrototype.cpp: * runtime/IntlObject.cpp: * runtime/IteratorPrototype.cpp: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitialized): (JSC::JSArray::setLengthWritable): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLengthWithArrayStorage): (JSC::JSArray::appendMemcpy): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithArrayStorage): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::createContiguousArrayButterfly): (JSC::createArrayButterfly): (JSC::JSArray::create): (JSC::JSArray::tryCreateUninitialized): Deleted. * runtime/JSArrayBufferView.h: * runtime/JSCInlines.h: * runtime/JSCJSValue.cpp: (JSC::JSValue::dumpInContextAssumingStructure): * runtime/JSCallee.cpp: (JSC::JSCallee::JSCallee): * runtime/JSCell.cpp: (JSC::JSCell::estimatedSize): * runtime/JSCell.h: (JSC::JSCell::cellStateOffset): Deleted. * runtime/JSCellInlines.h: (JSC::ExecState::vm): (JSC::JSCell::classInfo): (JSC::JSCell::callDestructor): (JSC::JSCell::vm): Deleted. * runtime/JSFunction.cpp: (JSC::JSFunction::create): (JSC::JSFunction::allocateAndInitializeRareData): (JSC::JSFunction::initializeRareData): (JSC::JSFunction::getOwnPropertySlot): (JSC::JSFunction::put): (JSC::JSFunction::deleteProperty): (JSC::JSFunction::defineOwnProperty): (JSC::JSFunction::setFunctionName): (JSC::JSFunction::reifyLength): (JSC::JSFunction::reifyName): (JSC::JSFunction::reifyLazyPropertyIfNeeded): (JSC::JSFunction::reifyBoundNameIfNeeded): * runtime/JSFunction.h: * runtime/JSFunctionInlines.h: (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint): (JSC::JSFunction::JSFunction): * runtime/JSGenericTypedArrayViewInlines.h: (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory): * runtime/JSInternalPromise.cpp: * runtime/JSInternalPromiseConstructor.cpp: * runtime/JSInternalPromiseDeferred.cpp: * runtime/JSInternalPromisePrototype.cpp: * runtime/JSJob.cpp: * runtime/JSMapIterator.cpp: * runtime/JSModuleNamespaceObject.cpp: * runtime/JSModuleRecord.cpp: * runtime/JSObject.cpp: (JSC::JSObject::visitButterfly): (JSC::JSObject::notifyPresenceOfIndexedAccessors): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createInitialUndecided): (JSC::JSObject::createInitialInt32): (JSC::JSObject::createInitialDouble): (JSC::JSObject::createInitialContiguous): (JSC::JSObject::createArrayStorage): (JSC::JSObject::createInitialArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::putByIndexBeyondVectorLength): (JSC::JSObject::putDirectIndexBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::increaseVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::growOutOfLineStorage): (JSC::JSObject::copyButterfly): Deleted. (JSC::JSObject::copyBackingStore): Deleted. * runtime/JSObject.h: (JSC::JSObject::globalObject): (JSC::JSObject::putDirectInternal): (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted. * runtime/JSObjectInlines.h: * runtime/JSPromise.cpp: * runtime/JSPromiseConstructor.cpp: * runtime/JSPromiseDeferred.cpp: * runtime/JSPromisePrototype.cpp: * runtime/JSPropertyNameIterator.cpp: * runtime/JSScope.cpp: (JSC::JSScope::resolve): * runtime/JSScope.h: (JSC::JSScope::globalObject): (JSC::JSScope::vm): Deleted. * runtime/JSSetIterator.cpp: * runtime/JSStringIterator.cpp: * runtime/JSTemplateRegistryKey.cpp: * runtime/JSTypedArrayViewConstructor.cpp: * runtime/JSTypedArrayViewPrototype.cpp: * runtime/JSWeakMap.cpp: * runtime/JSWeakSet.cpp: * runtime/MapConstructor.cpp: * runtime/MapIteratorPrototype.cpp: * runtime/MapPrototype.cpp: * runtime/NativeErrorConstructor.cpp: * runtime/NativeStdFunctionCell.cpp: * runtime/Operations.h: (JSC::scribbleFreeCells): (JSC::scribble): * runtime/Options.h: * runtime/PropertyTable.cpp: * runtime/ProxyConstructor.cpp: * runtime/ProxyObject.cpp: * runtime/ProxyRevoke.cpp: * runtime/RegExp.cpp: (JSC::RegExp::match): (JSC::RegExp::matchConcurrently): (JSC::RegExp::matchCompareWithInterpreter): * runtime/RegExp.h: * runtime/RegExpConstructor.h: * runtime/RegExpInlines.h: (JSC::RegExp::matchInline): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): (JSC::createRegExpMatchesArray): * runtime/RegExpPrototype.cpp: (JSC::genericSplit): * runtime/RuntimeType.cpp: * runtime/SamplingProfiler.cpp: (JSC::SamplingProfiler::processUnverifiedStackTraces): * runtime/SetConstructor.cpp: * runtime/SetIteratorPrototype.cpp: * runtime/SetPrototype.cpp: * runtime/StackFrame.cpp: Added. (JSC::StackFrame::sourceID): (JSC::StackFrame::sourceURL): (JSC::StackFrame::functionName): (JSC::StackFrame::computeLineAndColumn): (JSC::StackFrame::toString): * runtime/StackFrame.h: Added. (JSC::StackFrame::isNative): * runtime/StringConstructor.cpp: * runtime/StringIteratorPrototype.cpp: * runtime/StructureInlines.h: (JSC::Structure::propertyTable): * runtime/TemplateRegistry.cpp: * runtime/TestRunnerUtils.cpp: (JSC::finalizeStatsAtEndOfTesting): * runtime/TestRunnerUtils.h: * runtime/TypeProfilerLog.cpp: * runtime/TypeSet.cpp: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::ensureStackCapacityForCLoop): (JSC::VM::isSafeToRecurseSoftCLoop): * runtime/VM.h: * runtime/VMEntryScope.h: * runtime/VMInlines.h: (JSC::VM::ensureStackCapacityFor): (JSC::VM::isSafeToRecurseSoft): * runtime/WeakMapConstructor.cpp: * runtime/WeakMapData.cpp: * runtime/WeakMapPrototype.cpp: * runtime/WeakSetConstructor.cpp: * runtime/WeakSetPrototype.cpp: * testRegExp.cpp: (testOneRegExp): * tools/JSDollarVM.cpp: * tools/JSDollarVMPrototype.cpp: (JSC::JSDollarVMPrototype::isInObjectSpace): Source/WebCore: No new tests because no new WebCore behavior. Just rewiring #includes. * ForwardingHeaders/heap/HeapInlines.h: Added. * ForwardingHeaders/interpreter/Interpreter.h: Removed. * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added. * Modules/indexeddb/IDBCursorWithValue.cpp: * Modules/indexeddb/client/TransactionOperation.cpp: * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp: * Modules/indexeddb/server/UniqueIDBDatabase.cpp: * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp: * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp: * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp: * bindings/js/JSClientRectCustom.cpp: * bindings/js/JSDOMBinding.cpp: * bindings/js/JSDOMBinding.h: * bindings/js/JSDeviceMotionEventCustom.cpp: * bindings/js/JSDeviceOrientationEventCustom.cpp: * bindings/js/JSErrorEventCustom.cpp: * bindings/js/JSIDBCursorWithValueCustom.cpp: * bindings/js/JSIDBIndexCustom.cpp: * bindings/js/JSPopStateEventCustom.cpp: * bindings/js/JSWebGL2RenderingContextCustom.cpp: * bindings/js/JSWorkerGlobalScopeCustom.cpp: * bindings/js/WorkerScriptController.cpp: * contentextensions/ContentExtensionParser.cpp: * dom/ErrorEvent.cpp: * html/HTMLCanvasElement.cpp: * html/MediaDocument.cpp: * inspector/CommandLineAPIModule.cpp: * loader/EmptyClients.cpp: * page/CaptionUserPreferences.cpp: * page/Frame.cpp: * page/PageGroup.cpp: * page/UserContentController.cpp: * platform/mock/mediasource/MockBox.cpp: * testing/GCObservation.cpp: Source/WebKit2: Just rewiring some #includes. * UIProcess/ViewGestureController.cpp: * UIProcess/WebPageProxy.cpp: * UIProcess/WebProcessPool.cpp: * UIProcess/WebProcessProxy.cpp: * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp: * WebProcess/Plugins/Netscape/JSNPObject.cpp: Source/WTF: I needed tryFastAlignedMalloc() so I added it. * wtf/FastMalloc.cpp: (WTF::tryFastAlignedMalloc): * wtf/FastMalloc.h: * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): (WTF::ParkingLot::forEach): Deleted. * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::forEach): * wtf/ScopedLambda.h: (WTF::scopedLambdaRef): * wtf/SentinelLinkedList.h: (WTF::SentinelLinkedList::forEach): (WTF::RawNode>::takeFrom): * wtf/SimpleStats.h: (WTF::SimpleStats::operator bool): (WTF::SimpleStats::operator!): Deleted. Tools: * DumpRenderTree/TestRunner.cpp: * DumpRenderTree/mac/DumpRenderTree.mm: (DumpRenderTreeMain): * Scripts/run-jsc-stress-tests: * TestWebKitAPI/Tests/WTF/Vector.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/179778@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@205462 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-06 01:02:22 +00:00
}
private:
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
WTF_EXPORT_PRIVATE static ParkResult parkConditionallyImpl(
const void* address,
ScopedLambda should have a lifetime story that makes sense to the compiler https://bugs.webkit.org/show_bug.cgi?id=158118 Reviewed by Mark Lam. Source/WTF: Prior to this change, there were two lifetime bugs in ScopedLambda: - scopedLambda(Functor&&) would bind Functor to const lambda&, so the resulting ScopedLambdaFunctor would hold a reference to the original lambda. This would have surprising behavior; for example it meant that this code was wrong: auto l = scopedLambda<things>([&] ...); The solution is to have explicit copy/move versions of scopedLambda() rather than rely on perfect forwarding. - ScopedLambdaFunctor did not override its copy or move operations, so if the compiler did not RVO scopedLambda(), it would return a ScopedLambdaFunctor whose m_arg points to a dead temporary ScopedLambdaFunctor instance. The solution is to have explicit copy/move constructors and operators, which preserve the invariant that ScopedLambda::m_arg points to this. One nice side-effect of all of these constructors and operators being explicit is that we can rely on WTFMove's excellent assertions, which helped catch the first issue. This reverts ParkingLot to use ScopedLambda again. * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): * wtf/ScopedLambda.h: (WTF::scopedLambda): Tools: Added a test case. This test crashes before the fix and now it passes. * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/ScopedLambda.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/176236@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@201433 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-05-26 21:58:42 +00:00
const ScopedLambda<bool()>& validation,
const ScopedLambda<void()>& beforeSleep,
WTF::ParkingLot should stop using std::chrono because std::chrono::duration casts are prone to overflows https://bugs.webkit.org/show_bug.cgi?id=152045 Reviewed by Andy Estes. Source/JavaScriptCore: Probably the nicest example of why this patch is a good idea is the change in AtomicsObject.cpp. * jit/ICStats.cpp: (JSC::ICStats::ICStats): * runtime/AtomicsObject.cpp: (JSC::atomicsFuncWait): Source/WebCore: No new layout tests because no new behavior. The new WTF time classes have some unit tests in TestWebKitAPI. * fileapi/ThreadableBlobRegistry.cpp: (WebCore::ThreadableBlobRegistry::blobSize): * platform/MainThreadSharedTimer.h: * platform/SharedTimer.h: * platform/ThreadTimers.cpp: (WebCore::ThreadTimers::updateSharedTimer): * platform/cf/MainThreadSharedTimerCF.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/efl/MainThreadSharedTimerEfl.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/glib/MainThreadSharedTimerGLib.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * platform/win/MainThreadSharedTimerWin.cpp: (WebCore::MainThreadSharedTimer::setFireInterval): * workers/WorkerRunLoop.cpp: (WebCore::WorkerRunLoop::runInMode): Source/WebKit2: * Platform/IPC/Connection.cpp: (IPC::Connection::SyncMessageState::wait): (IPC::Connection::sendMessage): (IPC::Connection::timeoutRespectingIgnoreTimeoutsForTesting): (IPC::Connection::waitForMessage): (IPC::Connection::sendSyncMessage): (IPC::Connection::waitForSyncReply): * Platform/IPC/Connection.h: (IPC::Connection::sendSync): (IPC::Connection::waitForAndDispatchImmediately): * Platform/IPC/MessageSender.h: (IPC::MessageSender::sendSync): * UIProcess/ChildProcessProxy.h: (WebKit::ChildProcessProxy::sendSync): * UIProcess/Network/NetworkProcessProxy.cpp: (WebKit::NetworkProcessProxy::sendProcessWillSuspendImminently): * UIProcess/Storage/StorageManager.cpp: (WebKit::StorageManager::applicationWillTerminate): * UIProcess/WebProcessProxy.cpp: (WebKit::WebProcessProxy::sendProcessWillSuspendImminently): * UIProcess/WebResourceLoadStatisticsStore.cpp: (WebKit::WebResourceLoadStatisticsStore::applicationWillTerminate): * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.h: * UIProcess/mac/RemoteLayerTreeDrawingAreaProxy.mm: (-[WKOneShotDisplayLinkHandler displayLinkFired:]): (WebKit::RemoteLayerTreeDrawingAreaProxy::commitLayerTree): (WebKit::RemoteLayerTreeDrawingAreaProxy::didRefreshDisplay): (WebKit::RemoteLayerTreeDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/TiledCoreAnimationDrawingAreaProxy.mm: (WebKit::TiledCoreAnimationDrawingAreaProxy::waitForDidUpdateActivityState): * UIProcess/mac/WKImmediateActionController.mm: (-[WKImmediateActionController immediateActionRecognizerWillBeginAnimation:]): * UIProcess/mac/WebPageProxyMac.mm: (WebKit::WebPageProxy::stringSelectionForPasteboard): (WebKit::WebPageProxy::dataSelectionForPasteboard): (WebKit::WebPageProxy::readSelectionFromPasteboard): (WebKit::WebPageProxy::shouldDelayWindowOrderingForEvent): (WebKit::WebPageProxy::acceptsFirstMouse): * WebProcess/WebCoreSupport/WebChromeClient.cpp: (WebKit::WebChromeClient::runBeforeUnloadConfirmPanel): (WebKit::WebChromeClient::runJavaScriptAlert): (WebKit::WebChromeClient::runJavaScriptConfirm): (WebKit::WebChromeClient::runJavaScriptPrompt): (WebKit::WebChromeClient::print): (WebKit::WebChromeClient::exceededDatabaseQuota): (WebKit::WebChromeClient::reachedApplicationCacheOriginQuota): * WebProcess/WebCoreSupport/WebFrameLoaderClient.cpp: (WebKit::WebFrameLoaderClient::dispatchDecidePolicyForResponse): * WebProcess/WebPage/WebPage.cpp: (WebKit::WebPage::postSynchronousMessageForTesting): Source/WTF: We used to use 'double' for all time measurements. Sometimes it was milliseconds, sometimes it was seconds. Sometimes we measured a span of time, sometimes we spoke of time since some epoch. When we spoke of time since epoch, we either used a monotonic clock or a wall clock. The type - always 'double' - never told us what kind of time we had, even though there were roughly six of them (sec interval, ms interval, sec since epoch on wall, ms since epoch on wall, sec since epoch monotonic, ms since epoch monotonic). At some point, we thought that it would be a good idea to replace these doubles with std::chrono. But since replacing some things with std::chrono, we found it to be terribly inconvenient: - Outrageous API. I never want to say std::chrono::milliseconds(blah). I never want to say std::chrono::steady_clock::timepoint. The syntax for duration_cast is ugly, and ideally duration_cast would not even be a thing. - No overflow protection. std::chrono uses integers by default and using anything else is clumsy. But the integer math is done without regard for the rough edges of integer math, so any cast between std::chrono types risks overflow. Any comparison risks overflow because it may do conversions silently. We have even found bugs where some C++ implementations had more overflows than others, which ends up being a special kind of hell. In many cases, the overflow also has nasal demons. It's an error to represent time using integers. It would have been excusable back when floating point math was not guaranteed to be supported on all platforms, but that would have been a long time ago. Time is a continuous, infinite concept and it's a perfect fit for floating point: - Floating point preserves precision under multiplication in all but extreme cases, so using floating point for time means that unit conversions are almost completely lossless. This means that we don't have to think very hard about what units to use. In this patch, we use seconds almost everywhere. We only convert at boundaries, like an API boundary that wants something other than seconds. - Floating point makes it easy to reason about infinity, which is something that time code wants to do a lot. Example: when would you like to timeout? Infinity please! This is the most elegant way of having an API support both a timeout variant and a no-timeout variant. - Floating point does well-understood things when math goes wrong, and these things are pretty well optimized to match what a mathematician would do when computing with real numbers represented using scientific notation with a finite number of significant digits. This means that time math under floating point looks like normal math. On the other hand, std::chrono time math looks like garbage because you have to always check for multiple possible UB corners whenever you touch large integers. Integers that represent time are very likely to be large and you don't have to do much to overflow them. At this time, based on the number of bugs we have already seen due to chrono overflows, I am not certain that we even understand what are all of the corner cases that we should even check for. This patch introduces a new set of timekeeping classes that are all based on double, and all internally use seconds. These classes support algebraic typing. The classes are: - Seconds: this is for measuring a duration. - WallTime: time since epoch according to a wall clock (aka real time clock). - MonotonicTime: time since epoch according to a monotonic clock. - ClockType: enum that says either Wall or Monotonic. - TimeWithDynamicClockType: a tuple of double and ClockType, which represents either a wall time or a monotonic time. All of these classes behave like C++ values and are cheap to copy around since they are very nearly POD. This supports comprehensive conversions between the various time types. Most of this is by way of algebra. Here are just some of the rules we recognize: WallTime = WallTime + Seconds Seconds = WallTime - WallTime MonotonicTime = MonotonicTime + Seconds etc... We support negative, infinite, and NaN times because math. We support conversions between MonotonicTime and WallTime, like: WallTime wt = mt.approximateWallTime() This is called this "approximate" because the only way to do it is to get the current time on both clocks and convert relative to that. Many of our APIs would be happy using whatever notion of time the user wanted to use. For those APIs, which includes Condition and ParkingLot, we have TimeWithDynamicClockType. You can automatically convert WallTime or MonotonicTime to TimeWithDynamicClockType. This means that if you use a WallTime with Condition::waitUntil, then Condition's internal logic for when it should wake up makes its decision based on the current WallTime - but if you use MonotonicTime then waitUntil will make its decision based on current MonotonicTime. This is a greater level of flexibility than chrono allowed, since chrono did not have the concept of a dynamic clock type. This patch does not include conversions between std::chrono and these new time classes, because past experience shows that we're quite bad at getting conversions between std::chrono and anything else right. Also, I didn't need such conversion code because this patch only converts code that transitively touches ParkingLot and Condition. It was easy to get all of that code onto the new time classes. * WTF.xcodeproj/project.pbxproj: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/ClockType.cpp: Added. (WTF::printInternal): * wtf/ClockType.h: Added. * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::waitFor): (WTF::ConditionBase::wait): (WTF::ConditionBase::waitUntilWallClockSeconds): Deleted. (WTF::ConditionBase::waitUntilMonotonicClockSeconds): Deleted. (WTF::ConditionBase::waitForSeconds): Deleted. (WTF::ConditionBase::waitForSecondsImpl): Deleted. (WTF::ConditionBase::waitForImpl): Deleted. (WTF::ConditionBase::absoluteFromRelative): Deleted. * wtf/CrossThreadQueue.h: (WTF::CrossThreadQueue<DataType>::waitForMessage): * wtf/CurrentTime.cpp: (WTF::sleep): * wtf/MessageQueue.h: (WTF::MessageQueue::infiniteTime): Deleted. * wtf/MonotonicTime.cpp: Added. (WTF::MonotonicTime::now): (WTF::MonotonicTime::approximateWallTime): (WTF::MonotonicTime::dump): (WTF::MonotonicTime::sleep): * wtf/MonotonicTime.h: Added. (WTF::MonotonicTime::MonotonicTime): (WTF::MonotonicTime::fromRawDouble): (WTF::MonotonicTime::infinity): (WTF::MonotonicTime::secondsSinceEpoch): (WTF::MonotonicTime::approximateMonotonicTime): (WTF::MonotonicTime::operator bool): (WTF::MonotonicTime::operator+): (WTF::MonotonicTime::operator-): (WTF::MonotonicTime::operator+=): (WTF::MonotonicTime::operator-=): (WTF::MonotonicTime::operator==): (WTF::MonotonicTime::operator!=): (WTF::MonotonicTime::operator<): (WTF::MonotonicTime::operator>): (WTF::MonotonicTime::operator<=): (WTF::MonotonicTime::operator>=): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkCount): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): * wtf/Seconds.cpp: Added. (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::dump): (WTF::Seconds::sleep): * wtf/Seconds.h: Added. (WTF::Seconds::Seconds): (WTF::Seconds::value): (WTF::Seconds::seconds): (WTF::Seconds::milliseconds): (WTF::Seconds::microseconds): (WTF::Seconds::nanoseconds): (WTF::Seconds::fromMilliseconds): (WTF::Seconds::fromMicroseconds): (WTF::Seconds::fromNanoseconds): (WTF::Seconds::infinity): (WTF::Seconds::operator bool): (WTF::Seconds::operator+): (WTF::Seconds::operator-): (WTF::Seconds::operator*): (WTF::Seconds::operator/): (WTF::Seconds::operator+=): (WTF::Seconds::operator-=): (WTF::Seconds::operator*=): (WTF::Seconds::operator/=): (WTF::Seconds::operator==): (WTF::Seconds::operator!=): (WTF::Seconds::operator<): (WTF::Seconds::operator>): (WTF::Seconds::operator<=): (WTF::Seconds::operator>=): * wtf/TimeWithDynamicClockType.cpp: Added. (WTF::TimeWithDynamicClockType::now): (WTF::TimeWithDynamicClockType::nowWithSameClock): (WTF::TimeWithDynamicClockType::wallTime): (WTF::TimeWithDynamicClockType::monotonicTime): (WTF::TimeWithDynamicClockType::approximateWallTime): (WTF::TimeWithDynamicClockType::approximateMonotonicTime): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator<): (WTF::TimeWithDynamicClockType::operator>): (WTF::TimeWithDynamicClockType::operator<=): (WTF::TimeWithDynamicClockType::operator>=): (WTF::TimeWithDynamicClockType::dump): (WTF::TimeWithDynamicClockType::sleep): * wtf/TimeWithDynamicClockType.h: Added. (WTF::TimeWithDynamicClockType::TimeWithDynamicClockType): (WTF::TimeWithDynamicClockType::fromRawDouble): (WTF::TimeWithDynamicClockType::secondsSinceEpoch): (WTF::TimeWithDynamicClockType::clockType): (WTF::TimeWithDynamicClockType::withSameClockAndRawDouble): (WTF::TimeWithDynamicClockType::operator bool): (WTF::TimeWithDynamicClockType::operator+): (WTF::TimeWithDynamicClockType::operator-): (WTF::TimeWithDynamicClockType::operator+=): (WTF::TimeWithDynamicClockType::operator-=): (WTF::TimeWithDynamicClockType::operator==): (WTF::TimeWithDynamicClockType::operator!=): * wtf/WallTime.cpp: Added. (WTF::WallTime::now): (WTF::WallTime::approximateMonotonicTime): (WTF::WallTime::dump): (WTF::WallTime::sleep): * wtf/WallTime.h: Added. (WTF::WallTime::WallTime): (WTF::WallTime::fromRawDouble): (WTF::WallTime::infinity): (WTF::WallTime::secondsSinceEpoch): (WTF::WallTime::approximateWallTime): (WTF::WallTime::operator bool): (WTF::WallTime::operator+): (WTF::WallTime::operator-): (WTF::WallTime::operator+=): (WTF::WallTime::operator-=): (WTF::WallTime::operator==): (WTF::WallTime::operator!=): (WTF::WallTime::operator<): (WTF::WallTime::operator>): (WTF::WallTime::operator<=): (WTF::WallTime::operator>=): * wtf/threads/BinarySemaphore.cpp: (WTF::BinarySemaphore::wait): * wtf/threads/BinarySemaphore.h: Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Condition.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp: (TestWebKitAPI::ToUpperConverter::stopProducing): (TestWebKitAPI::ToUpperConverter::stopConsuming): * TestWebKitAPI/Tests/WTF/Time.cpp: Added. (WTF::operator<<): (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/182152@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@208415 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-11-05 03:02:39 +00:00
const TimeWithDynamicClockType& timeout);
WTF_EXPORT_PRIVATE static void unparkOneImpl(
WTF::Lock should be fair eventually https://bugs.webkit.org/show_bug.cgi?id=159384 Reviewed by Geoffrey Garen. Source/WTF: In https://webkit.org/blog/6161/locking-in-webkit/ we showed how relaxing the fairness of locks makes them fast. That post presented lock fairness as a trade-off between two extremes: - Barging. A barging lock, like WTF::Lock, releases the lock in unlock() even if there was a thread on the queue. If there was a thread on the queue, the lock is released and that thread is made runnable. That thread may then grab the lock, or some other thread may grab the lock first (it may barge). Usually, the barging thread is the thread that released the lock in the first place. This maximizes throughput but hurts fairness. There is no good theoretical bound on how unfair the lock may become, but empirical data suggests that it's fair enough for the cases we previously measured. - FIFO. A FIFO lock, like HandoffLock in ToyLocks.h, does not release the lock in unlock() if there is a thread waiting. If there is a thread waiting, unlock() will make that thread runnable and inform it that it now holds the lock. This ensures perfect round-robin fairness and allows us to reason theoretically about how long it may take for a thread to grab the lock. For example, if we know that only N threads are running and each one may contend on a critical section, and each one may hold the lock for at most S seconds, then the time it takes to grab the lock is N * S. Unfortunately, FIFO locks perform very badly in most cases. This is because for the common case of short critical sections, they force a context switch after each critical section if the lock is contended. This change makes WTF::Lock almost as fair as FIFO while still being as fast as barging. Thanks to this new algorithm, you can now have both of these things at the same time. This change makes WTF::Lock eventually fair. We can almost (more on the caveats below) guarantee that the time it takes to grab a lock is N * max(1ms, S). In other words, critical sections that are longer than 1ms are always fair. For shorter critical sections, the amount of time that any thread waits is 1ms times the number of threads. There are some caveats that arise from our use of randomness, but even then, in the limit as the critical section length goes to infinity, the lock becomes fair. The corner cases are unlikely to happen; our experiments show that the lock becomes exactly as fair as a FIFO lock for any critical section that is 1ms or longer. The fairness mechanism is broken into two parts. WTF::Lock can now choose to unlock a lock fairly or unfairly thanks to the new ParkingLot token mechanism. WTF::Lock knows when to use fair unlocking based on a timeout mechanism in ParkingLot called timeToBeFair. ParkingLot::unparkOne() and ParkingLot::parkConditionally() can now communicate with each other via a token. unparkOne() can pass a token, which parkConditionally() will return. This change also makes parkConditionally() a lot more precise about when it was unparked due to a call to unparkOne(). If unparkOne() is told that a thread was unparked then this thread is guaranteed to report that it was unparked rather than timing out, and that thread is guaranteed to get the token that unparkOne() passed. The token is an intptr_t. We use it as a boolean variable in WTF::Lock, but you could use it to pass arbitrary data structures. By default, the token is zero. WTF::Lock's unlock() will pass 1 as the token if it is doing fair unlocking. In that case, unlock() will not release the lock, and lock() will know that it holds the lock as soon as parkConditionally() returns. Note that this algorithm relies on unparkOne() invoking WTF::Lock's callback while the queue lock is held, so that WTF::Lock can make a decision about unlock strategy and inject a token while it has complete knowledge over the state of the queue. As such, it's not immediately obvious how to implement this algorithm on top of futexes. You really need ParkingLot! WTF::Lock does not use fair unlocking every time. We expose a new API, Lock::unlockFairly(), which forces the fair unlocking behavior. Additionally, ParkingLot now maintains a per-bucket stochastic fairness timeout. When the timeout fires, the unparkOne() callback sees UnparkResult::timeToBeFair = true. This timeout is set to be anywhere from 0ms to 1ms at random. When a dequeue happens and there are threads that actually get dequeued, we check if the time since the last unfair unlock (the last time timeToBeFair was set to true) is more than the timeout amount. If so, then we set timeToBeFair to true and reset the timeout. This means that in the absence of ParkingLot collisions, unfair unlocking is guaranteed to happen at least once per millisecond. It will happen at 2 KHz on average. If there are collisions, then each collision adds one millisecond to the worst case (and 0.5 ms to the average case). The reason why we don't just use a fixed 1ms timeout is that we want to avoid resonance. Imagine a program in which some thread acquires a lock at 1 KHz in-phase with the timeToBeFair timeout. Then this thread would be the benefactor of fairness to the detriment of everyone else. Randomness ensures that we aren't too fair to any one thread. Empirically, this is neutral on our major benchmarks like JetStream but it's an enormous improvement in LockFairnessTest. It's common for an unfair lock (either our BargingLock, the old WTF::Lock, any of the other futex-based locks that barge, or new os_unfair_lock) to allow only one thread to hold the lock during a whole second in which each thread is holding the lock for 1ms at a time. This is because in a barging lock, releasing a lock after holding it for 1ms and then reacquiring it immediately virtually ensures that none of the other threads can wake up in time to grab it before it's relocked. But the new WTF::Lock handles this case like a champ: each thread gets equal turns. Here's some data. If we launch 10 threads and have each of them run for 1 second while repeatedly holding a critical section for 1ms, then here's how many times each thread gets to hold the lock using the old WTF::Lock algorithm: 799, 6, 1, 1, 1, 1, 1, 1, 1, 1 One thread hogged the lock for almost the whole time! With the new WTF::Lock, the lock becomes totally fair: 80, 79, 79, 79, 79, 79, 79, 80, 80, 79 I don't know of anyone creating such an automatically-fair adaptive lock before, so I think that this is a pretty awesome advancement to the state of the art! This change is good for three reasons: - We do have long critical sections in WebKit and we don't want to have to worry about starvation. This reduces the likelihood that we will see starvation due to our lock strategy. - I was talking to ggaren about bmalloc's locking needs, and he wanted unlockFairly() or lockFairly() or some moral equivalent for the scavenger thread. - If we use a WTF::Lock to manage heap access in a multithreaded GC, we'll need the ability to unlock and relock without barging. * benchmarks/LockFairnessTest.cpp: (main): * benchmarks/ToyLocks.h: * wtf/Condition.h: (WTF::ConditionBase::waitUntil): (WTF::ConditionBase::notifyOne): * wtf/Lock.cpp: (WTF::LockBase::lockSlow): (WTF::LockBase::unlockSlow): (WTF::LockBase::unlockFairlySlow): (WTF::LockBase::unlockSlowImpl): * wtf/Lock.h: (WTF::LockBase::try_lock): (WTF::LockBase::unlock): (WTF::LockBase::unlockFairly): (WTF::LockBase::isHeld): (WTF::LockBase::isFullyReset): * wtf/ParkingLot.cpp: (WTF::ParkingLot::parkConditionallyImpl): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkOneImpl): (WTF::ParkingLot::unparkAll): * wtf/ParkingLot.h: (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::compareAndPark): (WTF::ParkingLot::unparkOne): Tools: * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/178039@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@203350 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-07-18 18:32:52 +00:00
const void* address, const ScopedLambda<intptr_t(UnparkResult)>& callback);
[WTF] Introduce Thread class and use RefPtr<Thread> and align Windows Threading implementation semantics to Pthread one https://bugs.webkit.org/show_bug.cgi?id=170502 Reviewed by Mark Lam. Source/JavaScriptCore: * API/tests/CompareAndSwapTest.cpp: (testCompareAndSwap): * JavaScriptCore.xcodeproj/project.pbxproj: * b3/air/testair.cpp: * b3/testb3.cpp: (JSC::B3::run): * bytecode/SuperSampler.cpp: (JSC::initializeSuperSampler): * dfg/DFGWorklist.cpp: * disassembler/Disassembler.cpp: * heap/Heap.cpp: (JSC::Heap::lastChanceToFinalize): (JSC::Heap::notifyIsSafeToCollect): * heap/Heap.h: * heap/MachineStackMarker.cpp: (JSC::MachineThreads::~MachineThreads): (JSC::MachineThreads::addCurrentThread): (JSC::MachineThreads::removeThread): (JSC::MachineThreads::removeThreadIfFound): (JSC::MachineThreads::MachineThread::MachineThread): (JSC::MachineThreads::MachineThread::getRegisters): (JSC::MachineThreads::MachineThread::Registers::stackPointer): (JSC::MachineThreads::MachineThread::Registers::framePointer): (JSC::MachineThreads::MachineThread::Registers::instructionPointer): (JSC::MachineThreads::MachineThread::Registers::llintPC): (JSC::MachineThreads::MachineThread::captureStack): (JSC::MachineThreads::tryCopyOtherThreadStack): (JSC::MachineThreads::tryCopyOtherThreadStacks): (pthreadSignalHandlerSuspendResume): Deleted. (JSC::threadData): Deleted. (JSC::MachineThreads::Thread::Thread): Deleted. (JSC::MachineThreads::Thread::createForCurrentThread): Deleted. (JSC::MachineThreads::Thread::operator==): Deleted. (JSC::MachineThreads::machineThreadForCurrentThread): Deleted. (JSC::MachineThreads::ThreadData::ThreadData): Deleted. (JSC::MachineThreads::ThreadData::~ThreadData): Deleted. (JSC::MachineThreads::ThreadData::suspend): Deleted. (JSC::MachineThreads::ThreadData::resume): Deleted. (JSC::MachineThreads::ThreadData::getRegisters): Deleted. (JSC::MachineThreads::ThreadData::Registers::stackPointer): Deleted. (JSC::MachineThreads::ThreadData::Registers::framePointer): Deleted. (JSC::MachineThreads::ThreadData::Registers::instructionPointer): Deleted. (JSC::MachineThreads::ThreadData::Registers::llintPC): Deleted. (JSC::MachineThreads::ThreadData::freeRegisters): Deleted. (JSC::MachineThreads::ThreadData::captureStack): Deleted. * heap/MachineStackMarker.h: (JSC::MachineThreads::MachineThread::suspend): (JSC::MachineThreads::MachineThread::resume): (JSC::MachineThreads::MachineThread::threadID): (JSC::MachineThreads::MachineThread::stackBase): (JSC::MachineThreads::MachineThread::stackEnd): (JSC::MachineThreads::threadsListHead): (JSC::MachineThreads::Thread::operator!=): Deleted. (JSC::MachineThreads::Thread::suspend): Deleted. (JSC::MachineThreads::Thread::resume): Deleted. (JSC::MachineThreads::Thread::getRegisters): Deleted. (JSC::MachineThreads::Thread::freeRegisters): Deleted. (JSC::MachineThreads::Thread::captureStack): Deleted. (JSC::MachineThreads::Thread::platformThread): Deleted. (JSC::MachineThreads::Thread::stackBase): Deleted. (JSC::MachineThreads::Thread::stackEnd): Deleted. * jit/ICStats.cpp: (JSC::ICStats::ICStats): (JSC::ICStats::~ICStats): * jit/ICStats.h: * jsc.cpp: (functionDollarAgentStart): (startTimeoutThreadIfNeeded): * runtime/JSLock.cpp: (JSC::JSLock::lock): * runtime/JSLock.h: (JSC::JSLock::ownerThread): (JSC::JSLock::currentThreadIsHoldingLock): * runtime/SamplingProfiler.cpp: (JSC::FrameWalker::isValidFramePointer): (JSC::SamplingProfiler::SamplingProfiler): (JSC::SamplingProfiler::createThreadIfNecessary): (JSC::SamplingProfiler::takeSample): * runtime/SamplingProfiler.h: * runtime/VM.h: (JSC::VM::ownerThread): * runtime/VMTraps.cpp: (JSC::findActiveVMAndStackBounds): (JSC::VMTraps::SignalSender::send): (JSC::VMTraps::fireTrap): Source/WebCore: Mechanical change. Use Thread:: APIs. * Modules/indexeddb/server/IDBServer.cpp: (WebCore::IDBServer::IDBServer::IDBServer): * Modules/indexeddb/server/IDBServer.h: * Modules/webaudio/AsyncAudioDecoder.cpp: (WebCore::AsyncAudioDecoder::AsyncAudioDecoder): (WebCore::AsyncAudioDecoder::~AsyncAudioDecoder): (WebCore::AsyncAudioDecoder::runLoop): * Modules/webaudio/AsyncAudioDecoder.h: * Modules/webaudio/OfflineAudioDestinationNode.cpp: (WebCore::OfflineAudioDestinationNode::OfflineAudioDestinationNode): (WebCore::OfflineAudioDestinationNode::uninitialize): (WebCore::OfflineAudioDestinationNode::startRendering): * Modules/webaudio/OfflineAudioDestinationNode.h: * Modules/webdatabase/Database.cpp: (WebCore::Database::securityOrigin): * Modules/webdatabase/DatabaseThread.cpp: (WebCore::DatabaseThread::start): (WebCore::DatabaseThread::databaseThread): (WebCore::DatabaseThread::recordDatabaseOpen): (WebCore::DatabaseThread::recordDatabaseClosed): * Modules/webdatabase/DatabaseThread.h: (WebCore::DatabaseThread::getThreadID): * bindings/js/GCController.cpp: (WebCore::GCController::garbageCollectOnAlternateThreadForDebugging): * fileapi/AsyncFileStream.cpp: (WebCore::callOnFileThread): * loader/icon/IconDatabase.cpp: (WebCore::IconDatabase::open): (WebCore::IconDatabase::close): * loader/icon/IconDatabase.h: * page/ResourceUsageThread.cpp: (WebCore::ResourceUsageThread::createThreadIfNeeded): * page/ResourceUsageThread.h: * page/scrolling/ScrollingThread.cpp: (WebCore::ScrollingThread::ScrollingThread): (WebCore::ScrollingThread::isCurrentThread): (WebCore::ScrollingThread::createThreadIfNeeded): (WebCore::ScrollingThread::threadCallback): * page/scrolling/ScrollingThread.h: * platform/audio/HRTFDatabaseLoader.cpp: (WebCore::HRTFDatabaseLoader::HRTFDatabaseLoader): (WebCore::HRTFDatabaseLoader::loadAsynchronously): (WebCore::HRTFDatabaseLoader::waitForLoaderThreadCompletion): * platform/audio/HRTFDatabaseLoader.h: * platform/audio/ReverbConvolver.cpp: (WebCore::ReverbConvolver::ReverbConvolver): (WebCore::ReverbConvolver::~ReverbConvolver): * platform/audio/ReverbConvolver.h: * platform/audio/gstreamer/AudioFileReaderGStreamer.cpp: (WebCore::createBusFromAudioFile): (WebCore::createBusFromInMemoryAudioFile): * platform/graphics/gstreamer/WebKitWebSourceGStreamer.cpp: (ResourceHandleStreamingClient::ResourceHandleStreamingClient): (ResourceHandleStreamingClient::~ResourceHandleStreamingClient): * platform/network/cf/LoaderRunLoopCF.cpp: (WebCore::loaderRunLoop): * platform/network/curl/CurlDownload.cpp: (WebCore::CurlDownloadManager::startThreadIfNeeded): (WebCore::CurlDownloadManager::stopThread): * platform/network/curl/CurlDownload.h: * platform/network/curl/SocketStreamHandleImpl.h: * platform/network/curl/SocketStreamHandleImplCurl.cpp: (WebCore::SocketStreamHandleImpl::startThread): (WebCore::SocketStreamHandleImpl::stopThread): * workers/WorkerThread.cpp: (WebCore::WorkerThread::WorkerThread): (WebCore::WorkerThread::start): (WebCore::WorkerThread::workerThread): * workers/WorkerThread.h: (WebCore::WorkerThread::threadID): Source/WebKit: Mechanical change. Use Thread:: APIs. * Storage/StorageThread.cpp: (WebCore::StorageThread::StorageThread): (WebCore::StorageThread::~StorageThread): (WebCore::StorageThread::start): (WebCore::StorageThread::dispatch): (WebCore::StorageThread::terminate): * Storage/StorageThread.h: Source/WebKit2: Mechanical change. Use Thread:: APIs. * NetworkProcess/NetworkProcess.cpp: (WebKit::NetworkProcess::initializeNetworkProcess): * NetworkProcess/cache/NetworkCacheIOChannelSoup.cpp: (WebKit::NetworkCache::IOChannel::readSyncInThread): * Platform/IPC/Connection.cpp: (IPC::Connection::processIncomingMessage): * Shared/EntryPointUtilities/mac/XPCService/XPCServiceEntryPoint.h: (WebKit::XPCServiceInitializer): * UIProcess/linux/MemoryPressureMonitor.cpp: (WebKit::MemoryPressureMonitor::MemoryPressureMonitor): * WebProcess/WebProcess.cpp: (WebKit::WebProcess::initializeWebProcess): Source/WTF: This patch is refactoring of WTF Threading mechanism to merge JavaScriptCore's threading extension to WTF Threading. Previously, JavaScriptCore requires richer threading features (such as suspending and resuming threads), and they are implemented for PlatformThread in JavaScriptCore. But these features should be implemented in WTF Threading side instead of maintaining JSC's own threading features too. This patch removes these features from JSC and move it to WTF Threading. However, current WTF Threading has one problem: Windows version of WTF Threading has different semantics from Pthreads one. In Windows WTF Threading, we cannot perform any operation after the target thread is detached: WTF Threading stop tracking the state of the thread once the thread is detached. But this is not the same to Pthreads one. In Pthreads, pthread_detach just means that the resource of the thread will be destroyed automatically. While some operations like pthread_join will be rejected, some operations like pthread_kill will be accepted. The problem is that detached thread can be suspended and resumed in JSC. For example, in jsc.cpp, we start the worker thread and detach it immediately. In worker thread, we will create VM and thus concurrent GC will suspend and resume the detached thread. However, in Windows WTF Threading, we have no reference to the detached thread. Thus we cannot perform suspend and resume operations onto the detached thread. To solve the problem, we change Windows Threading mechanism drastically to align it to the Pthread semantics. In the new Threading, we have RefPtr<Thread> class. It holds a handle to a platform thread. We can perform threading operations with this class. For example, Thread::suspend is offered. And we use destructor of the thread local variable to release the resources held by RefPtr<Thread>. In Windows, Thread::detach does nothing because the resource will be destroyed automatically by RefPtr<Thread>. To do so, we introduce ThreadHolder for Windows. This is similar to the previous ThreadIdentifierData for Pthreads. It holds RefPtr<Thread> in the thread local storage (technically, it is Fiber Local Storage in Windows). Thread::current() will return this reference. The problematic situation is that the order of the deallocation of the thread local storage is not defined. So we should not touch thread local storage in the destructor of the thread local storage. To avoid such edge cases, we have currentThread() / Thread::currentID() APIs. They are safe to be called even in the destructor of the other thread local storage. And in Windows, in the FLS destructor, we will create the thread_local variable to defer the destruction of the ThreadHolder. We ensure that this destructor is called after the other FLS destructors are called in Windows 10. This patch is performance neutral. * WTF.xcodeproj/project.pbxproj: * benchmarks/ConditionSpeedTest.cpp: * benchmarks/LockFairnessTest.cpp: * benchmarks/LockSpeedTest.cpp: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/CMakeLists.txt: * wtf/MainThread.h: * wtf/MemoryPressureHandler.h: * wtf/ParallelJobsGeneric.cpp: (WTF::ParallelEnvironment::ThreadPrivate::tryLockFor): (WTF::ParallelEnvironment::ThreadPrivate::workerThread): * wtf/ParallelJobsGeneric.h: (WTF::ParallelEnvironment::ThreadPrivate::ThreadPrivate): Deleted. * wtf/ParkingLot.cpp: (WTF::ParkingLot::forEachImpl): * wtf/ParkingLot.h: (WTF::ParkingLot::forEach): * wtf/PlatformRegisters.h: Renamed from Source/JavaScriptCore/runtime/PlatformThread.h. * wtf/RefPtr.h: (WTF::RefPtr::RefPtr): * wtf/ThreadFunctionInvocation.h: (WTF::ThreadFunctionInvocation::ThreadFunctionInvocation): * wtf/ThreadHolder.cpp: Added. (WTF::ThreadHolder::~ThreadHolder): (WTF::ThreadHolder::initialize): * wtf/ThreadHolder.h: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.h. (WTF::ThreadHolder::thread): (WTF::ThreadHolder::ThreadHolder): * wtf/ThreadHolderPthreads.cpp: Renamed from Source/WTF/wtf/ThreadIdentifierDataPthreads.cpp. (WTF::ThreadHolder::initializeOnce): (WTF::ThreadHolder::current): (WTF::ThreadHolder::destruct): * wtf/ThreadHolderWin.cpp: Added. (WTF::threadMapMutex): (WTF::threadMap): (WTF::ThreadHolder::initializeOnce): (WTF::ThreadHolder::current): (WTF::ThreadHolder::destruct): * wtf/ThreadSpecific.h: * wtf/Threading.cpp: (WTF::Thread::normalizeThreadName): (WTF::threadEntryPoint): (WTF::Thread::create): (WTF::Thread::setCurrentThreadIsUserInteractive): (WTF::Thread::setCurrentThreadIsUserInitiated): (WTF::Thread::setGlobalMaxQOSClass): (WTF::Thread::adjustedQOSClass): (WTF::Thread::dump): (WTF::initializeThreading): (WTF::normalizeThreadName): Deleted. (WTF::createThread): Deleted. (WTF::setCurrentThreadIsUserInteractive): Deleted. (WTF::setCurrentThreadIsUserInitiated): Deleted. (WTF::setGlobalMaxQOSClass): Deleted. (WTF::adjustedQOSClass): Deleted. * wtf/Threading.h: (WTF::Thread::id): (WTF::Thread::operator==): (WTF::Thread::operator!=): (WTF::Thread::joinableState): (WTF::Thread::didBecomeDetached): (WTF::Thread::didJoin): (WTF::Thread::hasExited): (WTF::currentThread): * wtf/ThreadingPthreads.cpp: (WTF::Thread::Thread): (WTF::Thread::~Thread): (WTF::Thread::signalHandlerSuspendResume): (WTF::Thread::initializePlatformThreading): (WTF::initializeCurrentThreadEvenIfNonWTFCreated): (WTF::wtfThreadEntryPoint): (WTF::Thread::createInternal): (WTF::Thread::initializeCurrentThreadInternal): (WTF::Thread::changePriority): (WTF::Thread::waitForCompletion): (WTF::Thread::detach): (WTF::Thread::current): (WTF::Thread::currentID): (WTF::Thread::signal): (WTF::Thread::resume): (WTF::Thread::getRegisters): (WTF::Thread::didExit): (WTF::Thread::establish): (WTF::PthreadState::PthreadState): Deleted. (WTF::PthreadState::joinableState): Deleted. (WTF::PthreadState::pthreadHandle): Deleted. (WTF::PthreadState::didBecomeDetached): Deleted. (WTF::PthreadState::didExit): Deleted. (WTF::PthreadState::didJoin): Deleted. (WTF::PthreadState::hasExited): Deleted. (WTF::threadMapMutex): Deleted. (WTF::initializeThreading): Deleted. (WTF::threadMap): Deleted. (WTF::identifierByPthreadHandle): Deleted. (WTF::establishIdentifierForPthreadHandle): Deleted. (WTF::pthreadHandleForIdentifierWithLockAlreadyHeld): Deleted. (WTF::createThreadInternal): Deleted. (WTF::initializeCurrentThreadInternal): Deleted. (WTF::changeThreadPriority): Deleted. (WTF::waitForThreadCompletion): Deleted. (WTF::detachThread): Deleted. (WTF::threadDidExit): Deleted. (WTF::currentThread): Deleted. (WTF::signalThread): Deleted. * wtf/ThreadingWin.cpp: (WTF::Thread::Thread): (WTF::Thread::~Thread): (WTF::Thread::initializeCurrentThreadInternal): (WTF::Thread::initializePlatformThreading): (WTF::wtfThreadEntryPoint): (WTF::Thread::createInternal): (WTF::Thread::changePriority): (WTF::Thread::waitForCompletion): (WTF::Thread::detach): (WTF::Thread::resume): (WTF::Thread::getRegisters): (WTF::Thread::current): (WTF::Thread::currentID): (WTF::Thread::didExit): (WTF::Thread::establish): (WTF::initializeCurrentThreadInternal): Deleted. (WTF::threadMapMutex): Deleted. (WTF::initializeThreading): Deleted. (WTF::threadMap): Deleted. (WTF::storeThreadHandleByIdentifier): Deleted. (WTF::threadHandleForIdentifier): Deleted. (WTF::clearThreadHandleForIdentifier): Deleted. (WTF::createThreadInternal): Deleted. (WTF::changeThreadPriority): Deleted. (WTF::waitForThreadCompletion): Deleted. (WTF::detachThread): Deleted. (WTF::currentThread): Deleted. * wtf/WorkQueue.cpp: (WTF::WorkQueue::concurrentApply): * wtf/WorkQueue.h: * wtf/cocoa/WorkQueueCocoa.cpp: (WTF::dispatchQOSClass): * wtf/generic/WorkQueueGeneric.cpp: (WorkQueue::platformInitialize): (WorkQueue::platformInvalidate): * wtf/linux/MemoryPressureHandlerLinux.cpp: (WTF::MemoryPressureHandler::EventFDPoller::EventFDPoller): (WTF::MemoryPressureHandler::EventFDPoller::~EventFDPoller): * wtf/win/MainThreadWin.cpp: (WTF::initializeMainThreadPlatform): Tools: Mechanical change. Use Thread:: APIs. * DumpRenderTree/JavaScriptThreading.cpp: (runJavaScriptThread): (startJavaScriptThreads): (stopJavaScriptThreads): * DumpRenderTree/mac/DumpRenderTree.mm: (testThreadIdentifierMap): * TestWebKitAPI/Tests/WTF/Condition.cpp: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::runLockTest): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Canonical link: https://commits.webkit.org/187690@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@215265 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-04-12 12:08:29 +00:00
WTF_EXPORT_PRIVATE static void forEachImpl(const ScopedLambda<void(Thread&, const void*)>&);
WTF should have a ParkingLot for parking sleeping threads, so that locks can fit in 1.6 bits https://bugs.webkit.org/show_bug.cgi?id=147665 Reviewed by Mark Lam. Source/JavaScriptCore: Replace ByteSpinLock with ByteLock. * runtime/ConcurrentJITLock.h: Source/WTF: This change adds a major new abstraction for concurrency algorithms in WebKit. It's called a ParkingLot, and it makes available a thread parking queue for each virtual address in memory. The queues are maintained by a data-access-parallel concurrent hashtable implementation. The memory usage is bounded at around half a KB per thread. The ParkingLot makes it easy to turn any spinlock-based concurrency protocol into one that parks threads after a while. Because queue state management is up to the ParkingLot and not the user's data structure, this patch uses it to implement a full adaptive mutex in one byte. In fact, only three states of that byte are used (0 = available, 1 = locked, 2 = locked and there are parked threads). Hence the joke that ParkingLot allows locks that fit in 1.6 bits. ByteLock is used as a replacement for ByteSpinLock in JavaScriptCore. The API tests for this also demo how to create a completely fair (FIFO) binary semamphore. The comment in Lock.h shows how we could accelerate Lock performance using ParkingLot. After we are sure that this code works, we can expand the use of ParkingLot. That's covered by https://bugs.webkit.org/show_bug.cgi?id=147841. * WTF.vcxproj/WTF.vcxproj: * WTF.xcodeproj/project.pbxproj: * benchmarks/LockSpeedTest.cpp: (main): * wtf/Atomics.h: (WTF::Atomic::compareExchangeWeak): (WTF::Atomic::compareExchangeStrong): * wtf/ByteLock.cpp: Added. (WTF::ByteLock::lockSlow): (WTF::ByteLock::unlockSlow): * wtf/ByteLock.h: Added. (WTF::ByteLock::ByteLock): (WTF::ByteLock::lock): (WTF::ByteLock::unlock): (WTF::ByteLock::isHeld): (WTF::ByteLock::isLocked): * wtf/CMakeLists.txt: * wtf/Lock.h: * wtf/ParkingLot.cpp: Added. (WTF::ParkingLot::parkConditionally): (WTF::ParkingLot::unparkOne): (WTF::ParkingLot::unparkAll): (WTF::ParkingLot::forEach): * wtf/ParkingLot.h: Added. (WTF::ParkingLot::compareAndPark): Tools: * TestWebKitAPI/CMakeLists.txt: * TestWebKitAPI/TestWebKitAPI.vcxproj/TestWebKitAPI.vcxproj: * TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj: * TestWebKitAPI/Tests/WTF/Lock.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WTF/ParkingLot.cpp: Added. (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/165996@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@188280 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2015-08-11 19:51:35 +00:00
};
} // namespace WTF
using WTF::ParkingLot;