haikuwebkit/Source/WTF/wtf/FastBitVector.cpp

83 lines
3.0 KiB
C++
Raw Permalink Normal View History

Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
/*
* Copyright (C) 2016 Apple Inc. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include "config.h"
Use pragma once in WTF https://bugs.webkit.org/show_bug.cgi?id=190527 Reviewed by Chris Dumez. Source/WTF: We also need to consistently include wtf headers from within wtf so we can build wtf without symbol redefinition errors from including the copy in Source and the copy in the build directory. * wtf/ASCIICType.h: * wtf/Assertions.cpp: * wtf/Assertions.h: * wtf/Atomics.h: * wtf/AutomaticThread.cpp: * wtf/AutomaticThread.h: * wtf/BackwardsGraph.h: * wtf/Bag.h: * wtf/BagToHashMap.h: * wtf/BitVector.cpp: * wtf/BitVector.h: * wtf/Bitmap.h: * wtf/BloomFilter.h: * wtf/Box.h: * wtf/BubbleSort.h: * wtf/BumpPointerAllocator.h: * wtf/ByteOrder.h: * wtf/CPUTime.cpp: * wtf/CallbackAggregator.h: * wtf/CheckedArithmetic.h: * wtf/CheckedBoolean.h: * wtf/ClockType.cpp: * wtf/ClockType.h: * wtf/CommaPrinter.h: * wtf/CompilationThread.cpp: * wtf/CompilationThread.h: * wtf/Compiler.h: * wtf/ConcurrentPtrHashSet.cpp: * wtf/ConcurrentVector.h: * wtf/Condition.h: * wtf/CountingLock.cpp: * wtf/CrossThreadTaskHandler.cpp: * wtf/CryptographicUtilities.cpp: * wtf/CryptographicUtilities.h: * wtf/CryptographicallyRandomNumber.cpp: * wtf/CryptographicallyRandomNumber.h: * wtf/CurrentTime.cpp: * wtf/DataLog.cpp: * wtf/DataLog.h: * wtf/DateMath.cpp: * wtf/DateMath.h: * wtf/DecimalNumber.cpp: * wtf/DecimalNumber.h: * wtf/Deque.h: * wtf/DisallowCType.h: * wtf/Dominators.h: * wtf/DoublyLinkedList.h: * wtf/FastBitVector.cpp: * wtf/FastMalloc.cpp: * wtf/FastMalloc.h: * wtf/FeatureDefines.h: * wtf/FilePrintStream.cpp: * wtf/FilePrintStream.h: * wtf/FlipBytes.h: * wtf/FunctionDispatcher.cpp: * wtf/FunctionDispatcher.h: * wtf/GetPtr.h: * wtf/Gigacage.cpp: * wtf/GlobalVersion.cpp: * wtf/GraphNodeWorklist.h: * wtf/GregorianDateTime.cpp: * wtf/GregorianDateTime.h: * wtf/HashFunctions.h: * wtf/HashMap.h: * wtf/HashMethod.h: * wtf/HashSet.h: * wtf/HashTable.cpp: * wtf/HashTraits.h: * wtf/Indenter.h: * wtf/IndexSparseSet.h: * wtf/InlineASM.h: * wtf/Insertion.h: * wtf/IteratorAdaptors.h: * wtf/IteratorRange.h: * wtf/JSONValues.cpp: * wtf/JSValueMalloc.cpp: * wtf/LEBDecoder.h: * wtf/Language.cpp: * wtf/ListDump.h: * wtf/Lock.cpp: * wtf/Lock.h: * wtf/LockAlgorithm.h: * wtf/LockedPrintStream.cpp: * wtf/Locker.h: * wtf/MD5.cpp: * wtf/MD5.h: * wtf/MainThread.cpp: * wtf/MainThread.h: * wtf/MallocPtr.h: * wtf/MathExtras.h: * wtf/MediaTime.cpp: * wtf/MediaTime.h: * wtf/MemoryPressureHandler.cpp: * wtf/MessageQueue.h: * wtf/MetaAllocator.cpp: * wtf/MetaAllocator.h: * wtf/MetaAllocatorHandle.h: * wtf/MonotonicTime.cpp: * wtf/MonotonicTime.h: * wtf/NakedPtr.h: * wtf/NoLock.h: * wtf/NoTailCalls.h: * wtf/Noncopyable.h: * wtf/NumberOfCores.cpp: * wtf/NumberOfCores.h: * wtf/OSAllocator.h: * wtf/OSAllocatorPosix.cpp: * wtf/OSRandomSource.cpp: * wtf/OSRandomSource.h: * wtf/ObjcRuntimeExtras.h: * wtf/OrderMaker.h: * wtf/PackedIntVector.h: * wtf/PageAllocation.h: * wtf/PageBlock.cpp: * wtf/PageBlock.h: * wtf/PageReservation.h: * wtf/ParallelHelperPool.cpp: * wtf/ParallelHelperPool.h: * wtf/ParallelJobs.h: * wtf/ParallelJobsLibdispatch.h: * wtf/ParallelVectorIterator.h: * wtf/ParkingLot.cpp: * wtf/ParkingLot.h: * wtf/Platform.h: * wtf/PointerComparison.h: * wtf/Poisoned.cpp: * wtf/PrintStream.cpp: * wtf/PrintStream.h: * wtf/ProcessID.h: * wtf/ProcessPrivilege.cpp: * wtf/RAMSize.cpp: * wtf/RAMSize.h: * wtf/RandomDevice.cpp: * wtf/RandomNumber.cpp: * wtf/RandomNumber.h: * wtf/RandomNumberSeed.h: * wtf/RangeSet.h: * wtf/RawPointer.h: * wtf/ReadWriteLock.cpp: * wtf/RedBlackTree.h: * wtf/Ref.h: * wtf/RefCountedArray.h: * wtf/RefCountedLeakCounter.cpp: * wtf/RefCountedLeakCounter.h: * wtf/RefCounter.h: * wtf/RefPtr.h: * wtf/RetainPtr.h: * wtf/RunLoop.cpp: * wtf/RunLoop.h: * wtf/RunLoopTimer.h: * wtf/RunLoopTimerCF.cpp: * wtf/SHA1.cpp: * wtf/SHA1.h: * wtf/SaturatedArithmetic.h: (saturatedSubtraction): * wtf/SchedulePair.h: * wtf/SchedulePairCF.cpp: * wtf/SchedulePairMac.mm: * wtf/ScopedLambda.h: * wtf/Seconds.cpp: * wtf/Seconds.h: * wtf/SegmentedVector.h: * wtf/SentinelLinkedList.h: * wtf/SharedTask.h: * wtf/SimpleStats.h: * wtf/SingleRootGraph.h: * wtf/SinglyLinkedList.h: * wtf/SixCharacterHash.cpp: * wtf/SixCharacterHash.h: * wtf/SmallPtrSet.h: * wtf/Spectrum.h: * wtf/StackBounds.cpp: * wtf/StackBounds.h: * wtf/StackStats.cpp: * wtf/StackStats.h: * wtf/StackTrace.cpp: * wtf/StdLibExtras.h: * wtf/StreamBuffer.h: * wtf/StringHashDumpContext.h: * wtf/StringPrintStream.cpp: * wtf/StringPrintStream.h: * wtf/ThreadGroup.cpp: * wtf/ThreadMessage.cpp: * wtf/ThreadSpecific.h: * wtf/Threading.cpp: * wtf/Threading.h: * wtf/ThreadingPrimitives.h: * wtf/ThreadingPthreads.cpp: * wtf/TimeWithDynamicClockType.cpp: * wtf/TimeWithDynamicClockType.h: * wtf/TimingScope.cpp: * wtf/TinyLRUCache.h: * wtf/TinyPtrSet.h: * wtf/TriState.h: * wtf/TypeCasts.h: * wtf/UUID.cpp: * wtf/UnionFind.h: * wtf/VMTags.h: * wtf/ValueCheck.h: * wtf/Vector.h: * wtf/VectorTraits.h: * wtf/WallTime.cpp: * wtf/WallTime.h: * wtf/WeakPtr.h: * wtf/WeakRandom.h: * wtf/WordLock.cpp: * wtf/WordLock.h: * wtf/WorkQueue.cpp: * wtf/WorkQueue.h: * wtf/WorkerPool.cpp: * wtf/cf/LanguageCF.cpp: * wtf/cf/RunLoopCF.cpp: * wtf/cocoa/Entitlements.mm: * wtf/cocoa/MachSendRight.cpp: * wtf/cocoa/MainThreadCocoa.mm: * wtf/cocoa/MemoryFootprintCocoa.cpp: * wtf/cocoa/WorkQueueCocoa.cpp: * wtf/dtoa.cpp: * wtf/dtoa.h: * wtf/ios/WebCoreThread.cpp: * wtf/ios/WebCoreThread.h: * wtf/mac/AppKitCompatibilityDeclarations.h: * wtf/mac/DeprecatedSymbolsUsedBySafari.mm: * wtf/mbmalloc.cpp: * wtf/persistence/PersistentCoders.cpp: * wtf/persistence/PersistentDecoder.cpp: * wtf/persistence/PersistentEncoder.cpp: * wtf/spi/cf/CFBundleSPI.h: * wtf/spi/darwin/CommonCryptoSPI.h: * wtf/text/ASCIIFastPath.h: * wtf/text/ASCIILiteral.cpp: * wtf/text/AtomicString.cpp: * wtf/text/AtomicString.h: * wtf/text/AtomicStringHash.h: * wtf/text/AtomicStringImpl.cpp: * wtf/text/AtomicStringImpl.h: * wtf/text/AtomicStringTable.cpp: * wtf/text/AtomicStringTable.h: * wtf/text/Base64.cpp: * wtf/text/CString.cpp: * wtf/text/CString.h: * wtf/text/ConversionMode.h: * wtf/text/ExternalStringImpl.cpp: * wtf/text/IntegerToStringConversion.h: * wtf/text/LChar.h: * wtf/text/LineEnding.cpp: * wtf/text/StringBuffer.h: * wtf/text/StringBuilder.cpp: * wtf/text/StringBuilder.h: * wtf/text/StringBuilderJSON.cpp: * wtf/text/StringCommon.h: * wtf/text/StringConcatenate.h: * wtf/text/StringHash.h: * wtf/text/StringImpl.cpp: * wtf/text/StringImpl.h: * wtf/text/StringOperators.h: * wtf/text/StringView.cpp: * wtf/text/StringView.h: * wtf/text/SymbolImpl.cpp: * wtf/text/SymbolRegistry.cpp: * wtf/text/SymbolRegistry.h: * wtf/text/TextBreakIterator.cpp: * wtf/text/TextBreakIterator.h: * wtf/text/TextBreakIteratorInternalICU.h: * wtf/text/TextPosition.h: * wtf/text/TextStream.cpp: * wtf/text/UniquedStringImpl.h: * wtf/text/WTFString.cpp: * wtf/text/WTFString.h: * wtf/text/cocoa/StringCocoa.mm: * wtf/text/cocoa/StringViewCocoa.mm: * wtf/text/cocoa/TextBreakIteratorInternalICUCocoa.cpp: * wtf/text/icu/UTextProvider.cpp: * wtf/text/icu/UTextProvider.h: * wtf/text/icu/UTextProviderLatin1.cpp: * wtf/text/icu/UTextProviderLatin1.h: * wtf/text/icu/UTextProviderUTF16.cpp: * wtf/text/icu/UTextProviderUTF16.h: * wtf/threads/BinarySemaphore.cpp: * wtf/threads/BinarySemaphore.h: * wtf/threads/Signals.cpp: * wtf/unicode/CharacterNames.h: * wtf/unicode/Collator.h: * wtf/unicode/CollatorDefault.cpp: * wtf/unicode/UTF8.cpp: * wtf/unicode/UTF8.h: Tools: Put WorkQueue in namespace DRT so it does not conflict with WTF::WorkQueue. * DumpRenderTree/TestRunner.cpp: (TestRunner::queueLoadHTMLString): (TestRunner::queueLoadAlternateHTMLString): (TestRunner::queueBackNavigation): (TestRunner::queueForwardNavigation): (TestRunner::queueLoadingScript): (TestRunner::queueNonLoadingScript): (TestRunner::queueReload): * DumpRenderTree/WorkQueue.cpp: (WorkQueue::singleton): Deleted. (WorkQueue::WorkQueue): Deleted. (WorkQueue::queue): Deleted. (WorkQueue::dequeue): Deleted. (WorkQueue::count): Deleted. (WorkQueue::clear): Deleted. (WorkQueue::processWork): Deleted. * DumpRenderTree/WorkQueue.h: (WorkQueue::setFrozen): Deleted. * DumpRenderTree/WorkQueueItem.h: * DumpRenderTree/mac/DumpRenderTree.mm: (runTest): * DumpRenderTree/mac/FrameLoadDelegate.mm: (-[FrameLoadDelegate processWork:]): (-[FrameLoadDelegate webView:locationChangeDone:forDataSource:]): * DumpRenderTree/mac/TestRunnerMac.mm: (TestRunner::notifyDone): (TestRunner::forceImmediateCompletion): (TestRunner::queueLoad): * DumpRenderTree/win/DumpRenderTree.cpp: (runTest): * DumpRenderTree/win/FrameLoadDelegate.cpp: (FrameLoadDelegate::processWork): (FrameLoadDelegate::locationChangeDone): * DumpRenderTree/win/TestRunnerWin.cpp: (TestRunner::notifyDone): (TestRunner::forceImmediateCompletion): (TestRunner::queueLoad): Canonical link: https://commits.webkit.org/205473@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@237099 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-10-15 14:24:49 +00:00
#include <wtf/FastBitVector.h>
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
#include <wtf/NeverDestroyed.h>
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
namespace WTF {
Experiment: create lots of different malloc zones for easier accounting of memory use https://bugs.webkit.org/show_bug.cgi?id=186422 Patch by Yusuke Suzuki <ysuzuki@apple.com> and Simon Fraser <simon.fraser@apple.com> on 2020-01-02 Reviewed by Saam Barati. Source/bmalloc: * bmalloc/BPlatform.h: * bmalloc/Environment.cpp: (bmalloc::Environment::computeIsDebugHeapEnabled): * bmalloc/IsoHeap.h: (bmalloc::api::IsoHeap::IsoHeap): * bmalloc/IsoHeapInlines.h: (bmalloc::api::IsoHeap<Type>::IsoHeap): * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::allocateSlow): (bmalloc::IsoTLS::deallocateSlow): Source/JavaScriptCore: * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * assembler/AssemblerBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * assembler/AssemblerBuffer.h: (JSC::AssemblerData::AssemblerData): (JSC::AssemblerData::operator=): (JSC::AssemblerData::~AssemblerData): (JSC::AssemblerData::grow): * bytecode/AccessCase.cpp: * bytecode/AccessCase.h: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeBasicBlock.h: * bytecode/CodeBlock.cpp: * bytecode/CodeBlock.h: * bytecode/InstructionStream.cpp: * bytecode/InstructionStream.h: * bytecode/PolymorphicAccess.cpp: * bytecode/PolymorphicAccess.h: * bytecode/UnlinkedMetadataTable.cpp: (JSC::UnlinkedMetadataTable::finalize): * bytecode/UnlinkedMetadataTable.h: * bytecode/UnlinkedMetadataTableInlines.h: (JSC::UnlinkedMetadataTable::UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::~UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::link): (JSC::UnlinkedMetadataTable::unlink): * bytecode/ValueProfile.h: (JSC::ValueProfileAndVirtualRegisterBuffer::ValueProfileAndVirtualRegisterBuffer): * bytecode/Watchpoint.cpp: * bytecode/Watchpoint.h: * dfg/DFGBasicBlock.cpp: * dfg/DFGBasicBlock.h: * dfg/DFGNode.cpp: * dfg/DFGNode.h: * dfg/DFGSpeculativeJIT.cpp: * dfg/DFGSpeculativeJIT.h: * heap/BlockDirectory.cpp: * heap/BlockDirectory.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::freeAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GCSegmentedArray.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * heap/GCSegmentedArray.h: * heap/GCSegmentedArrayInlines.h: (JSC::GCArraySegment<T>::create): (JSC::GCArraySegment<T>::destroy): * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/IsoSubspace.cpp: (JSC::IsoSubspace::IsoSubspace): * heap/MarkedBlock.cpp: * heap/MarkedBlock.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::destroy): * heap/WeakBlock.h: * jit/JITCode.cpp: * jit/JITCode.h: * jit/RegisterAtOffsetList.cpp: * jit/RegisterAtOffsetList.h: * parser/Nodes.cpp: * parser/Nodes.h: * parser/ParserArena.cpp: (JSC::ParserArena::deallocateObjects): (JSC::ParserArena::allocateFreeablePool): * parser/ParserArena.h: * parser/SourceProvider.cpp: * parser/SourceProvider.h: * parser/SourceProviderCache.cpp: * parser/SourceProviderCache.h: * parser/SourceProviderCacheItem.h: (JSC::SourceProviderCacheItem::create): * runtime/CachePayload.cpp: (JSC::CachePayload::makeMallocPayload): * runtime/CachePayload.h: * runtime/CachedBytecode.h: (JSC::CachedBytecode::create): * runtime/CachedTypes.cpp: (JSC::Encoder::release): (JSC::Encoder::Page::Page): (JSC::CachedVector::encode): (JSC::CachedVector::decode const): (JSC::CachedInstructionStream::decode const): * runtime/PropertyMapHashTable.h: (JSC::PropertyTable::rehash): * runtime/PropertyTable.cpp: (JSC::PropertyTable::PropertyTable): (JSC::PropertyTable::~PropertyTable): * runtime/SymbolTable.cpp: * runtime/SymbolTable.h: * runtime/VM.cpp: (JSC::VM::~VM): * runtime/VM.h: (JSC::ScratchBuffer::create): (JSC::VM::exceptionFuzzingBuffer): * wasm/WasmInstance.cpp: (JSC::Wasm::Instance::Instance): * wasm/WasmInstance.h: * wasm/WasmTable.cpp: (JSC::Wasm::Table::Table): (JSC::Wasm::FuncRefTable::FuncRefTable): * wasm/WasmTable.h: Source/WebCore: * Sources.txt: * WebCore.xcodeproj/project.pbxproj: * bindings/js/SerializedScriptValue.cpp: * bindings/js/SerializedScriptValue.h: * css/CSSFontFace.cpp: * css/CSSFontFace.h: * css/CSSSelector.cpp: * css/CSSSelector.h: * css/CSSValue.cpp: * css/CSSValue.h: * css/StyleProperties.cpp: (WebCore::ImmutableStyleProperties::create): * css/StyleProperties.h: * css/StyleRule.cpp: * css/StyleRule.h: * dom/ElementData.cpp: (WebCore::ShareableElementData::createWithAttributes): (WebCore::UniqueElementData::makeShareableCopy const): * dom/ElementData.h: * dom/NodeRareData.cpp: * dom/NodeRareData.h: * dom/QualifiedName.cpp: * dom/QualifiedName.h: * html/parser/HTMLDocumentParser.cpp: * html/parser/HTMLDocumentParser.h: * loader/DocumentLoader.cpp: * loader/DocumentLoader.h: * loader/ResourceLoader.cpp: * loader/ResourceLoader.h: * loader/cache/CachedResource.cpp: * loader/cache/CachedResource.h: * page/PerformanceEntry.cpp: * page/PerformanceEntry.h: * platform/graphics/Font.cpp: * platform/graphics/Font.h: * platform/graphics/FontCascadeFonts.cpp: * platform/graphics/FontCascadeFonts.h: * platform/graphics/Region.cpp: * platform/graphics/Region.h: * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::releaseUint8Vector): * platform/graphics/cg/ImageBufferCG.cpp: (WebCore::ImageBuffer::ImageBuffer): * platform/graphics/nicosia/NicosiaBuffer.cpp: (Nicosia::Buffer::Buffer): * platform/network/ResourceHandle.cpp: * platform/network/ResourceHandleInternal.h: * platform/network/cf/FormDataStreamCFNet.cpp: (WebCore::closeCurrentStream): (WebCore::advanceCurrentStream): * rendering/RenderLayer.cpp: * rendering/RenderLayer.h: * rendering/TableLayout.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * rendering/TableLayout.h: * rendering/style/RenderStyle.cpp: * rendering/style/RenderStyle.h: * rendering/style/SVGRenderStyle.cpp: * rendering/style/SVGRenderStyle.h: * rendering/style/SVGRenderStyleDefs.cpp: * rendering/style/SVGRenderStyleDefs.h: * rendering/style/StyleBoxData.cpp: * rendering/style/StyleBoxData.h: * rendering/style/StyleInheritedData.cpp: * rendering/style/StyleInheritedData.h: * rendering/style/StyleRareInheritedData.cpp: * rendering/style/StyleRareInheritedData.h: * rendering/style/StyleRareNonInheritedData.cpp: * rendering/style/StyleRareNonInheritedData.h: * rendering/style/StyleSurroundData.cpp: * rendering/style/StyleSurroundData.h: * rendering/style/StyleTransformData.cpp: * rendering/style/StyleTransformData.h: * style/StyleTreeResolver.cpp: * style/StyleTreeResolver.h: * svg/animation/SMILTimeContainer.cpp: * svg/animation/SMILTimeContainer.h: Source/WebKit: * Shared/ShareableBitmap.cpp: (WebKit::ShareableBitmap::create): (WebKit::ShareableBitmap::~ShareableBitmap): * UIProcess/mac/LegacySessionStateCoding.cpp: (WebKit::HistoryEntryDataEncoder::HistoryEntryDataEncoder): (WebKit::HistoryEntryDataEncoder::finishEncoding): (WebKit::encodeSessionHistoryEntryData): (WebKit::encodeLegacySessionState): Source/WTF: This patch introduces ENABLE(MALLOC_HEAP_BREAKDOWN). If this is enabled, we allocate malloc_zone per malloc kind. This offers the way to investigate the usage of memory per kind by using vmmap, like the following. VIRTUAL RESIDENT DIRTY SWAPPED ALLOCATION BYTES DIRTY+SWAP REGION MALLOC ZONE SIZE SIZE SIZE SIZE COUNT ALLOCATED FRAG SIZE % FRAG COUNT =========== ======= ========= ========= ========= ========= ========= ========= ====== ====== StringImpl_0x116efd000 188.0M 69.3M 30.9M 0K 139456 18.0M 12.9M 42% 34 DefaultMallocZone_0x10f487000 176.0M 53.9M 14.1M 0K 115956 9955K 4497K 32% 22 Vector_0x116eff000 162.0M 56.3M 55.3M 0K 140715 17.3M 37.9M 69% 36 MetadataTable_0x11843b000 152.0M 17.5M 17.5M 0K 14200 2353K 15.2M 87% 26 WebKit Using System Malloc_0x114cbe000 150.0M 31.6M 21.8M 0K 87422 16.7M 5278K 24% 23 InstructionStream_0x118469000 150.0M 5764K 5764K 0K 14470 4688K 1076K 19% 24 AssemblerData_0x117ee6000 150.0M 1928K 1928K 0K 1 16 1928K 100% 24 To achieve this goal without making very large change, we put a template type in various containers. For example, Vector will take Malloc parameter (the default one is FastMalloc allocator). If ENABLE(MALLOC_HEAP_BREAKDOWN) is enabled, we change this to specific VectorMalloc allocator, and vmmap can show memory usage of this allocator. This patch also supports malloc_zone per IsoHeap. So we can see memory allocation per IsoHeap in vmmap. To use this feature, we need to flip two compile time flags, ENABLE(MALLOC_HEAP_BREAKDOWN) in WTF and BENABLE_MALLOC_HEAP_BREAKDOWN in bmalloc. And use `vmmap $PID` to dump malloc zones. To allocate objects of a class with a specific malloc-zone, use WTF_MAKE_FAST_ALLOCATED_WITH_HEAP_IDENTIFIER(HeapIdentifier) for the class, and define allocator by DECLARE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a header and DEFINE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a cpp file. This patch also introduce callstack collector for malloc. Vector, HashMap etc. are used to allocate various things, but the above malloc_zone feature only tells thing like "Vector takes XXX MB memory". But what we want to know in this case is what Vector is consuming memory. We collect StackShot for each malloc call, and combine these information to tell which callsite is consuming much memory, which tell us that what Vector is consuming memory. * WTF.xcodeproj/project.pbxproj: * wtf/Bag.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/Bag.h: (WTF::Private::BagNode::BagNode): Deleted. * wtf/BitVector.cpp: (WTF::BitVector::OutOfLineBits::create): (WTF::BitVector::OutOfLineBits::destroy): * wtf/CMakeLists.txt: * wtf/ConcurrentBuffer.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/ConcurrentBuffer.h: * wtf/DebugHeap.cpp: Copied from Source/JavaScriptCore/runtime/CachePayload.cpp. (WTF::DebugHeap::DebugHeap): (WTF::DebugHeap::malloc): (WTF::DebugHeap::calloc): (WTF::DebugHeap::memalign): (WTF::DebugHeap::realloc): (WTF::DebugHeap::free): * wtf/DebugHeap.h: Added. * wtf/FastBitVector.cpp: (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::~FastBitVectorWordOwner): * wtf/FastMalloc.cpp: (WTF::fastMallocDumpMallocStats): (WTF::AvoidRecordingScope::AvoidRecordingScope): (WTF::AvoidRecordingScope::~AvoidRecordingScope): (WTF::MallocCallTracker::MallocSiteData::MallocSiteData): (WTF::MallocCallTracker::singleton): (WTF::MallocCallTracker::MallocCallTracker): (WTF::MallocCallTracker::recordMalloc): (WTF::MallocCallTracker::recordRealloc): (WTF::MallocCallTracker::recordFree): (WTF::MallocCallTracker::dumpStats): (WTF::fastMalloc): (WTF::fastRealloc): (WTF::fastFree): (WTF::fastAlignedMalloc): (WTF::tryFastAlignedMalloc): (WTF::fastAlignedFree): * wtf/FastMalloc.h: (WTF::FastMalloc::zeroedMalloc): (WTF::FastMalloc::tryZeroedMalloc): * wtf/Forward.h: * wtf/HashTable.cpp: * wtf/HashTable.h: (WTF::KeyTraits>::allocateTable): (WTF::KeyTraits>::deallocateTable): (WTF::KeyTraits>::rehash): * wtf/MallocPtr.h: (WTF::MallocPtr::MallocPtr): (WTF::MallocPtr::malloc): (WTF::MallocPtr::zeroedMalloc): (WTF::MallocPtr::tryMalloc): (WTF::MallocPtr::tryZeroedMalloc): (WTF::adoptMallocPtr): * wtf/MetaAllocator.cpp: (WTF::MetaAllocator::allocFreeSpaceNode): (WTF::MetaAllocator::freeFreeSpaceNode): * wtf/MetaAllocatorHandle.h: * wtf/Platform.h: * wtf/RefCountedArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/RefCountedArray.h: (WTF::RefCountedArray::RefCountedArray): (WTF::RefCountedArray::~RefCountedArray): (WTF::RefCountedArray::assign): * wtf/SegmentedVector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SegmentedVector.h: * wtf/SmallPtrSet.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SmallPtrSet.h: (WTF::SmallPtrSet::~SmallPtrSet): (WTF::SmallPtrSet::grow): * wtf/UniqueArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/UniqueArray.h: (WTF::UniqueArrayFree::operator() const): (WTF::UniqueArrayFree<T::operator() const): * wtf/Vector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::VectorBufferBase::releaseBuffer): (WTF::VectorBuffer::releaseBuffer): (WTF::Vector::swap): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::uncheckedConstructAndAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::Malloc>::isolatedCopy const): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::uncheckedConstructAndAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. (WTF::minCapacity>::isolatedCopy const): Deleted. * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): (WTF::StringBuffer::release): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): * wtf/text/StringImpl.h: (WTF::StringImpl::StringImpl): (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): * wtf/text/cf/StringImplCF.cpp: (WTF::StringWrapperCFAllocator::allocate): (WTF::StringWrapperCFAllocator::reallocate): (WTF::StringWrapperCFAllocator::deallocate): Canonical link: https://commits.webkit.org/218863@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253987 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-01-03 02:36:43 +00:00
DEFINE_ALLOCATOR_WITH_HEAP_IDENTIFIER(FastBitVector);
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
void FastBitVectorWordOwner::setEqualsSlow(const FastBitVectorWordOwner& other)
{
uint32_t* newArray = static_cast<uint32_t*>(
Experiment: create lots of different malloc zones for easier accounting of memory use https://bugs.webkit.org/show_bug.cgi?id=186422 Patch by Yusuke Suzuki <ysuzuki@apple.com> and Simon Fraser <simon.fraser@apple.com> on 2020-01-02 Reviewed by Saam Barati. Source/bmalloc: * bmalloc/BPlatform.h: * bmalloc/Environment.cpp: (bmalloc::Environment::computeIsDebugHeapEnabled): * bmalloc/IsoHeap.h: (bmalloc::api::IsoHeap::IsoHeap): * bmalloc/IsoHeapInlines.h: (bmalloc::api::IsoHeap<Type>::IsoHeap): * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::allocateSlow): (bmalloc::IsoTLS::deallocateSlow): Source/JavaScriptCore: * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * assembler/AssemblerBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * assembler/AssemblerBuffer.h: (JSC::AssemblerData::AssemblerData): (JSC::AssemblerData::operator=): (JSC::AssemblerData::~AssemblerData): (JSC::AssemblerData::grow): * bytecode/AccessCase.cpp: * bytecode/AccessCase.h: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeBasicBlock.h: * bytecode/CodeBlock.cpp: * bytecode/CodeBlock.h: * bytecode/InstructionStream.cpp: * bytecode/InstructionStream.h: * bytecode/PolymorphicAccess.cpp: * bytecode/PolymorphicAccess.h: * bytecode/UnlinkedMetadataTable.cpp: (JSC::UnlinkedMetadataTable::finalize): * bytecode/UnlinkedMetadataTable.h: * bytecode/UnlinkedMetadataTableInlines.h: (JSC::UnlinkedMetadataTable::UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::~UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::link): (JSC::UnlinkedMetadataTable::unlink): * bytecode/ValueProfile.h: (JSC::ValueProfileAndVirtualRegisterBuffer::ValueProfileAndVirtualRegisterBuffer): * bytecode/Watchpoint.cpp: * bytecode/Watchpoint.h: * dfg/DFGBasicBlock.cpp: * dfg/DFGBasicBlock.h: * dfg/DFGNode.cpp: * dfg/DFGNode.h: * dfg/DFGSpeculativeJIT.cpp: * dfg/DFGSpeculativeJIT.h: * heap/BlockDirectory.cpp: * heap/BlockDirectory.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::freeAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GCSegmentedArray.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * heap/GCSegmentedArray.h: * heap/GCSegmentedArrayInlines.h: (JSC::GCArraySegment<T>::create): (JSC::GCArraySegment<T>::destroy): * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/IsoSubspace.cpp: (JSC::IsoSubspace::IsoSubspace): * heap/MarkedBlock.cpp: * heap/MarkedBlock.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::destroy): * heap/WeakBlock.h: * jit/JITCode.cpp: * jit/JITCode.h: * jit/RegisterAtOffsetList.cpp: * jit/RegisterAtOffsetList.h: * parser/Nodes.cpp: * parser/Nodes.h: * parser/ParserArena.cpp: (JSC::ParserArena::deallocateObjects): (JSC::ParserArena::allocateFreeablePool): * parser/ParserArena.h: * parser/SourceProvider.cpp: * parser/SourceProvider.h: * parser/SourceProviderCache.cpp: * parser/SourceProviderCache.h: * parser/SourceProviderCacheItem.h: (JSC::SourceProviderCacheItem::create): * runtime/CachePayload.cpp: (JSC::CachePayload::makeMallocPayload): * runtime/CachePayload.h: * runtime/CachedBytecode.h: (JSC::CachedBytecode::create): * runtime/CachedTypes.cpp: (JSC::Encoder::release): (JSC::Encoder::Page::Page): (JSC::CachedVector::encode): (JSC::CachedVector::decode const): (JSC::CachedInstructionStream::decode const): * runtime/PropertyMapHashTable.h: (JSC::PropertyTable::rehash): * runtime/PropertyTable.cpp: (JSC::PropertyTable::PropertyTable): (JSC::PropertyTable::~PropertyTable): * runtime/SymbolTable.cpp: * runtime/SymbolTable.h: * runtime/VM.cpp: (JSC::VM::~VM): * runtime/VM.h: (JSC::ScratchBuffer::create): (JSC::VM::exceptionFuzzingBuffer): * wasm/WasmInstance.cpp: (JSC::Wasm::Instance::Instance): * wasm/WasmInstance.h: * wasm/WasmTable.cpp: (JSC::Wasm::Table::Table): (JSC::Wasm::FuncRefTable::FuncRefTable): * wasm/WasmTable.h: Source/WebCore: * Sources.txt: * WebCore.xcodeproj/project.pbxproj: * bindings/js/SerializedScriptValue.cpp: * bindings/js/SerializedScriptValue.h: * css/CSSFontFace.cpp: * css/CSSFontFace.h: * css/CSSSelector.cpp: * css/CSSSelector.h: * css/CSSValue.cpp: * css/CSSValue.h: * css/StyleProperties.cpp: (WebCore::ImmutableStyleProperties::create): * css/StyleProperties.h: * css/StyleRule.cpp: * css/StyleRule.h: * dom/ElementData.cpp: (WebCore::ShareableElementData::createWithAttributes): (WebCore::UniqueElementData::makeShareableCopy const): * dom/ElementData.h: * dom/NodeRareData.cpp: * dom/NodeRareData.h: * dom/QualifiedName.cpp: * dom/QualifiedName.h: * html/parser/HTMLDocumentParser.cpp: * html/parser/HTMLDocumentParser.h: * loader/DocumentLoader.cpp: * loader/DocumentLoader.h: * loader/ResourceLoader.cpp: * loader/ResourceLoader.h: * loader/cache/CachedResource.cpp: * loader/cache/CachedResource.h: * page/PerformanceEntry.cpp: * page/PerformanceEntry.h: * platform/graphics/Font.cpp: * platform/graphics/Font.h: * platform/graphics/FontCascadeFonts.cpp: * platform/graphics/FontCascadeFonts.h: * platform/graphics/Region.cpp: * platform/graphics/Region.h: * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::releaseUint8Vector): * platform/graphics/cg/ImageBufferCG.cpp: (WebCore::ImageBuffer::ImageBuffer): * platform/graphics/nicosia/NicosiaBuffer.cpp: (Nicosia::Buffer::Buffer): * platform/network/ResourceHandle.cpp: * platform/network/ResourceHandleInternal.h: * platform/network/cf/FormDataStreamCFNet.cpp: (WebCore::closeCurrentStream): (WebCore::advanceCurrentStream): * rendering/RenderLayer.cpp: * rendering/RenderLayer.h: * rendering/TableLayout.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * rendering/TableLayout.h: * rendering/style/RenderStyle.cpp: * rendering/style/RenderStyle.h: * rendering/style/SVGRenderStyle.cpp: * rendering/style/SVGRenderStyle.h: * rendering/style/SVGRenderStyleDefs.cpp: * rendering/style/SVGRenderStyleDefs.h: * rendering/style/StyleBoxData.cpp: * rendering/style/StyleBoxData.h: * rendering/style/StyleInheritedData.cpp: * rendering/style/StyleInheritedData.h: * rendering/style/StyleRareInheritedData.cpp: * rendering/style/StyleRareInheritedData.h: * rendering/style/StyleRareNonInheritedData.cpp: * rendering/style/StyleRareNonInheritedData.h: * rendering/style/StyleSurroundData.cpp: * rendering/style/StyleSurroundData.h: * rendering/style/StyleTransformData.cpp: * rendering/style/StyleTransformData.h: * style/StyleTreeResolver.cpp: * style/StyleTreeResolver.h: * svg/animation/SMILTimeContainer.cpp: * svg/animation/SMILTimeContainer.h: Source/WebKit: * Shared/ShareableBitmap.cpp: (WebKit::ShareableBitmap::create): (WebKit::ShareableBitmap::~ShareableBitmap): * UIProcess/mac/LegacySessionStateCoding.cpp: (WebKit::HistoryEntryDataEncoder::HistoryEntryDataEncoder): (WebKit::HistoryEntryDataEncoder::finishEncoding): (WebKit::encodeSessionHistoryEntryData): (WebKit::encodeLegacySessionState): Source/WTF: This patch introduces ENABLE(MALLOC_HEAP_BREAKDOWN). If this is enabled, we allocate malloc_zone per malloc kind. This offers the way to investigate the usage of memory per kind by using vmmap, like the following. VIRTUAL RESIDENT DIRTY SWAPPED ALLOCATION BYTES DIRTY+SWAP REGION MALLOC ZONE SIZE SIZE SIZE SIZE COUNT ALLOCATED FRAG SIZE % FRAG COUNT =========== ======= ========= ========= ========= ========= ========= ========= ====== ====== StringImpl_0x116efd000 188.0M 69.3M 30.9M 0K 139456 18.0M 12.9M 42% 34 DefaultMallocZone_0x10f487000 176.0M 53.9M 14.1M 0K 115956 9955K 4497K 32% 22 Vector_0x116eff000 162.0M 56.3M 55.3M 0K 140715 17.3M 37.9M 69% 36 MetadataTable_0x11843b000 152.0M 17.5M 17.5M 0K 14200 2353K 15.2M 87% 26 WebKit Using System Malloc_0x114cbe000 150.0M 31.6M 21.8M 0K 87422 16.7M 5278K 24% 23 InstructionStream_0x118469000 150.0M 5764K 5764K 0K 14470 4688K 1076K 19% 24 AssemblerData_0x117ee6000 150.0M 1928K 1928K 0K 1 16 1928K 100% 24 To achieve this goal without making very large change, we put a template type in various containers. For example, Vector will take Malloc parameter (the default one is FastMalloc allocator). If ENABLE(MALLOC_HEAP_BREAKDOWN) is enabled, we change this to specific VectorMalloc allocator, and vmmap can show memory usage of this allocator. This patch also supports malloc_zone per IsoHeap. So we can see memory allocation per IsoHeap in vmmap. To use this feature, we need to flip two compile time flags, ENABLE(MALLOC_HEAP_BREAKDOWN) in WTF and BENABLE_MALLOC_HEAP_BREAKDOWN in bmalloc. And use `vmmap $PID` to dump malloc zones. To allocate objects of a class with a specific malloc-zone, use WTF_MAKE_FAST_ALLOCATED_WITH_HEAP_IDENTIFIER(HeapIdentifier) for the class, and define allocator by DECLARE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a header and DEFINE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a cpp file. This patch also introduce callstack collector for malloc. Vector, HashMap etc. are used to allocate various things, but the above malloc_zone feature only tells thing like "Vector takes XXX MB memory". But what we want to know in this case is what Vector is consuming memory. We collect StackShot for each malloc call, and combine these information to tell which callsite is consuming much memory, which tell us that what Vector is consuming memory. * WTF.xcodeproj/project.pbxproj: * wtf/Bag.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/Bag.h: (WTF::Private::BagNode::BagNode): Deleted. * wtf/BitVector.cpp: (WTF::BitVector::OutOfLineBits::create): (WTF::BitVector::OutOfLineBits::destroy): * wtf/CMakeLists.txt: * wtf/ConcurrentBuffer.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/ConcurrentBuffer.h: * wtf/DebugHeap.cpp: Copied from Source/JavaScriptCore/runtime/CachePayload.cpp. (WTF::DebugHeap::DebugHeap): (WTF::DebugHeap::malloc): (WTF::DebugHeap::calloc): (WTF::DebugHeap::memalign): (WTF::DebugHeap::realloc): (WTF::DebugHeap::free): * wtf/DebugHeap.h: Added. * wtf/FastBitVector.cpp: (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::~FastBitVectorWordOwner): * wtf/FastMalloc.cpp: (WTF::fastMallocDumpMallocStats): (WTF::AvoidRecordingScope::AvoidRecordingScope): (WTF::AvoidRecordingScope::~AvoidRecordingScope): (WTF::MallocCallTracker::MallocSiteData::MallocSiteData): (WTF::MallocCallTracker::singleton): (WTF::MallocCallTracker::MallocCallTracker): (WTF::MallocCallTracker::recordMalloc): (WTF::MallocCallTracker::recordRealloc): (WTF::MallocCallTracker::recordFree): (WTF::MallocCallTracker::dumpStats): (WTF::fastMalloc): (WTF::fastRealloc): (WTF::fastFree): (WTF::fastAlignedMalloc): (WTF::tryFastAlignedMalloc): (WTF::fastAlignedFree): * wtf/FastMalloc.h: (WTF::FastMalloc::zeroedMalloc): (WTF::FastMalloc::tryZeroedMalloc): * wtf/Forward.h: * wtf/HashTable.cpp: * wtf/HashTable.h: (WTF::KeyTraits>::allocateTable): (WTF::KeyTraits>::deallocateTable): (WTF::KeyTraits>::rehash): * wtf/MallocPtr.h: (WTF::MallocPtr::MallocPtr): (WTF::MallocPtr::malloc): (WTF::MallocPtr::zeroedMalloc): (WTF::MallocPtr::tryMalloc): (WTF::MallocPtr::tryZeroedMalloc): (WTF::adoptMallocPtr): * wtf/MetaAllocator.cpp: (WTF::MetaAllocator::allocFreeSpaceNode): (WTF::MetaAllocator::freeFreeSpaceNode): * wtf/MetaAllocatorHandle.h: * wtf/Platform.h: * wtf/RefCountedArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/RefCountedArray.h: (WTF::RefCountedArray::RefCountedArray): (WTF::RefCountedArray::~RefCountedArray): (WTF::RefCountedArray::assign): * wtf/SegmentedVector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SegmentedVector.h: * wtf/SmallPtrSet.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SmallPtrSet.h: (WTF::SmallPtrSet::~SmallPtrSet): (WTF::SmallPtrSet::grow): * wtf/UniqueArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/UniqueArray.h: (WTF::UniqueArrayFree::operator() const): (WTF::UniqueArrayFree<T::operator() const): * wtf/Vector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::VectorBufferBase::releaseBuffer): (WTF::VectorBuffer::releaseBuffer): (WTF::Vector::swap): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::uncheckedConstructAndAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::Malloc>::isolatedCopy const): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::uncheckedConstructAndAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. (WTF::minCapacity>::isolatedCopy const): Deleted. * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): (WTF::StringBuffer::release): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): * wtf/text/StringImpl.h: (WTF::StringImpl::StringImpl): (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): * wtf/text/cf/StringImplCF.cpp: (WTF::StringWrapperCFAllocator::allocate): (WTF::StringWrapperCFAllocator::reallocate): (WTF::StringWrapperCFAllocator::deallocate): Canonical link: https://commits.webkit.org/218863@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253987 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-01-03 02:36:43 +00:00
FastBitVectorMalloc::zeroedMalloc(other.arrayLength() * sizeof(uint32_t)));
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
memcpy(newArray, other.m_words, other.arrayLength() * sizeof(uint32_t));
if (m_words)
Experiment: create lots of different malloc zones for easier accounting of memory use https://bugs.webkit.org/show_bug.cgi?id=186422 Patch by Yusuke Suzuki <ysuzuki@apple.com> and Simon Fraser <simon.fraser@apple.com> on 2020-01-02 Reviewed by Saam Barati. Source/bmalloc: * bmalloc/BPlatform.h: * bmalloc/Environment.cpp: (bmalloc::Environment::computeIsDebugHeapEnabled): * bmalloc/IsoHeap.h: (bmalloc::api::IsoHeap::IsoHeap): * bmalloc/IsoHeapInlines.h: (bmalloc::api::IsoHeap<Type>::IsoHeap): * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::allocateSlow): (bmalloc::IsoTLS::deallocateSlow): Source/JavaScriptCore: * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * assembler/AssemblerBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * assembler/AssemblerBuffer.h: (JSC::AssemblerData::AssemblerData): (JSC::AssemblerData::operator=): (JSC::AssemblerData::~AssemblerData): (JSC::AssemblerData::grow): * bytecode/AccessCase.cpp: * bytecode/AccessCase.h: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeBasicBlock.h: * bytecode/CodeBlock.cpp: * bytecode/CodeBlock.h: * bytecode/InstructionStream.cpp: * bytecode/InstructionStream.h: * bytecode/PolymorphicAccess.cpp: * bytecode/PolymorphicAccess.h: * bytecode/UnlinkedMetadataTable.cpp: (JSC::UnlinkedMetadataTable::finalize): * bytecode/UnlinkedMetadataTable.h: * bytecode/UnlinkedMetadataTableInlines.h: (JSC::UnlinkedMetadataTable::UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::~UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::link): (JSC::UnlinkedMetadataTable::unlink): * bytecode/ValueProfile.h: (JSC::ValueProfileAndVirtualRegisterBuffer::ValueProfileAndVirtualRegisterBuffer): * bytecode/Watchpoint.cpp: * bytecode/Watchpoint.h: * dfg/DFGBasicBlock.cpp: * dfg/DFGBasicBlock.h: * dfg/DFGNode.cpp: * dfg/DFGNode.h: * dfg/DFGSpeculativeJIT.cpp: * dfg/DFGSpeculativeJIT.h: * heap/BlockDirectory.cpp: * heap/BlockDirectory.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::freeAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GCSegmentedArray.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * heap/GCSegmentedArray.h: * heap/GCSegmentedArrayInlines.h: (JSC::GCArraySegment<T>::create): (JSC::GCArraySegment<T>::destroy): * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/IsoSubspace.cpp: (JSC::IsoSubspace::IsoSubspace): * heap/MarkedBlock.cpp: * heap/MarkedBlock.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::destroy): * heap/WeakBlock.h: * jit/JITCode.cpp: * jit/JITCode.h: * jit/RegisterAtOffsetList.cpp: * jit/RegisterAtOffsetList.h: * parser/Nodes.cpp: * parser/Nodes.h: * parser/ParserArena.cpp: (JSC::ParserArena::deallocateObjects): (JSC::ParserArena::allocateFreeablePool): * parser/ParserArena.h: * parser/SourceProvider.cpp: * parser/SourceProvider.h: * parser/SourceProviderCache.cpp: * parser/SourceProviderCache.h: * parser/SourceProviderCacheItem.h: (JSC::SourceProviderCacheItem::create): * runtime/CachePayload.cpp: (JSC::CachePayload::makeMallocPayload): * runtime/CachePayload.h: * runtime/CachedBytecode.h: (JSC::CachedBytecode::create): * runtime/CachedTypes.cpp: (JSC::Encoder::release): (JSC::Encoder::Page::Page): (JSC::CachedVector::encode): (JSC::CachedVector::decode const): (JSC::CachedInstructionStream::decode const): * runtime/PropertyMapHashTable.h: (JSC::PropertyTable::rehash): * runtime/PropertyTable.cpp: (JSC::PropertyTable::PropertyTable): (JSC::PropertyTable::~PropertyTable): * runtime/SymbolTable.cpp: * runtime/SymbolTable.h: * runtime/VM.cpp: (JSC::VM::~VM): * runtime/VM.h: (JSC::ScratchBuffer::create): (JSC::VM::exceptionFuzzingBuffer): * wasm/WasmInstance.cpp: (JSC::Wasm::Instance::Instance): * wasm/WasmInstance.h: * wasm/WasmTable.cpp: (JSC::Wasm::Table::Table): (JSC::Wasm::FuncRefTable::FuncRefTable): * wasm/WasmTable.h: Source/WebCore: * Sources.txt: * WebCore.xcodeproj/project.pbxproj: * bindings/js/SerializedScriptValue.cpp: * bindings/js/SerializedScriptValue.h: * css/CSSFontFace.cpp: * css/CSSFontFace.h: * css/CSSSelector.cpp: * css/CSSSelector.h: * css/CSSValue.cpp: * css/CSSValue.h: * css/StyleProperties.cpp: (WebCore::ImmutableStyleProperties::create): * css/StyleProperties.h: * css/StyleRule.cpp: * css/StyleRule.h: * dom/ElementData.cpp: (WebCore::ShareableElementData::createWithAttributes): (WebCore::UniqueElementData::makeShareableCopy const): * dom/ElementData.h: * dom/NodeRareData.cpp: * dom/NodeRareData.h: * dom/QualifiedName.cpp: * dom/QualifiedName.h: * html/parser/HTMLDocumentParser.cpp: * html/parser/HTMLDocumentParser.h: * loader/DocumentLoader.cpp: * loader/DocumentLoader.h: * loader/ResourceLoader.cpp: * loader/ResourceLoader.h: * loader/cache/CachedResource.cpp: * loader/cache/CachedResource.h: * page/PerformanceEntry.cpp: * page/PerformanceEntry.h: * platform/graphics/Font.cpp: * platform/graphics/Font.h: * platform/graphics/FontCascadeFonts.cpp: * platform/graphics/FontCascadeFonts.h: * platform/graphics/Region.cpp: * platform/graphics/Region.h: * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::releaseUint8Vector): * platform/graphics/cg/ImageBufferCG.cpp: (WebCore::ImageBuffer::ImageBuffer): * platform/graphics/nicosia/NicosiaBuffer.cpp: (Nicosia::Buffer::Buffer): * platform/network/ResourceHandle.cpp: * platform/network/ResourceHandleInternal.h: * platform/network/cf/FormDataStreamCFNet.cpp: (WebCore::closeCurrentStream): (WebCore::advanceCurrentStream): * rendering/RenderLayer.cpp: * rendering/RenderLayer.h: * rendering/TableLayout.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * rendering/TableLayout.h: * rendering/style/RenderStyle.cpp: * rendering/style/RenderStyle.h: * rendering/style/SVGRenderStyle.cpp: * rendering/style/SVGRenderStyle.h: * rendering/style/SVGRenderStyleDefs.cpp: * rendering/style/SVGRenderStyleDefs.h: * rendering/style/StyleBoxData.cpp: * rendering/style/StyleBoxData.h: * rendering/style/StyleInheritedData.cpp: * rendering/style/StyleInheritedData.h: * rendering/style/StyleRareInheritedData.cpp: * rendering/style/StyleRareInheritedData.h: * rendering/style/StyleRareNonInheritedData.cpp: * rendering/style/StyleRareNonInheritedData.h: * rendering/style/StyleSurroundData.cpp: * rendering/style/StyleSurroundData.h: * rendering/style/StyleTransformData.cpp: * rendering/style/StyleTransformData.h: * style/StyleTreeResolver.cpp: * style/StyleTreeResolver.h: * svg/animation/SMILTimeContainer.cpp: * svg/animation/SMILTimeContainer.h: Source/WebKit: * Shared/ShareableBitmap.cpp: (WebKit::ShareableBitmap::create): (WebKit::ShareableBitmap::~ShareableBitmap): * UIProcess/mac/LegacySessionStateCoding.cpp: (WebKit::HistoryEntryDataEncoder::HistoryEntryDataEncoder): (WebKit::HistoryEntryDataEncoder::finishEncoding): (WebKit::encodeSessionHistoryEntryData): (WebKit::encodeLegacySessionState): Source/WTF: This patch introduces ENABLE(MALLOC_HEAP_BREAKDOWN). If this is enabled, we allocate malloc_zone per malloc kind. This offers the way to investigate the usage of memory per kind by using vmmap, like the following. VIRTUAL RESIDENT DIRTY SWAPPED ALLOCATION BYTES DIRTY+SWAP REGION MALLOC ZONE SIZE SIZE SIZE SIZE COUNT ALLOCATED FRAG SIZE % FRAG COUNT =========== ======= ========= ========= ========= ========= ========= ========= ====== ====== StringImpl_0x116efd000 188.0M 69.3M 30.9M 0K 139456 18.0M 12.9M 42% 34 DefaultMallocZone_0x10f487000 176.0M 53.9M 14.1M 0K 115956 9955K 4497K 32% 22 Vector_0x116eff000 162.0M 56.3M 55.3M 0K 140715 17.3M 37.9M 69% 36 MetadataTable_0x11843b000 152.0M 17.5M 17.5M 0K 14200 2353K 15.2M 87% 26 WebKit Using System Malloc_0x114cbe000 150.0M 31.6M 21.8M 0K 87422 16.7M 5278K 24% 23 InstructionStream_0x118469000 150.0M 5764K 5764K 0K 14470 4688K 1076K 19% 24 AssemblerData_0x117ee6000 150.0M 1928K 1928K 0K 1 16 1928K 100% 24 To achieve this goal without making very large change, we put a template type in various containers. For example, Vector will take Malloc parameter (the default one is FastMalloc allocator). If ENABLE(MALLOC_HEAP_BREAKDOWN) is enabled, we change this to specific VectorMalloc allocator, and vmmap can show memory usage of this allocator. This patch also supports malloc_zone per IsoHeap. So we can see memory allocation per IsoHeap in vmmap. To use this feature, we need to flip two compile time flags, ENABLE(MALLOC_HEAP_BREAKDOWN) in WTF and BENABLE_MALLOC_HEAP_BREAKDOWN in bmalloc. And use `vmmap $PID` to dump malloc zones. To allocate objects of a class with a specific malloc-zone, use WTF_MAKE_FAST_ALLOCATED_WITH_HEAP_IDENTIFIER(HeapIdentifier) for the class, and define allocator by DECLARE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a header and DEFINE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a cpp file. This patch also introduce callstack collector for malloc. Vector, HashMap etc. are used to allocate various things, but the above malloc_zone feature only tells thing like "Vector takes XXX MB memory". But what we want to know in this case is what Vector is consuming memory. We collect StackShot for each malloc call, and combine these information to tell which callsite is consuming much memory, which tell us that what Vector is consuming memory. * WTF.xcodeproj/project.pbxproj: * wtf/Bag.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/Bag.h: (WTF::Private::BagNode::BagNode): Deleted. * wtf/BitVector.cpp: (WTF::BitVector::OutOfLineBits::create): (WTF::BitVector::OutOfLineBits::destroy): * wtf/CMakeLists.txt: * wtf/ConcurrentBuffer.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/ConcurrentBuffer.h: * wtf/DebugHeap.cpp: Copied from Source/JavaScriptCore/runtime/CachePayload.cpp. (WTF::DebugHeap::DebugHeap): (WTF::DebugHeap::malloc): (WTF::DebugHeap::calloc): (WTF::DebugHeap::memalign): (WTF::DebugHeap::realloc): (WTF::DebugHeap::free): * wtf/DebugHeap.h: Added. * wtf/FastBitVector.cpp: (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::~FastBitVectorWordOwner): * wtf/FastMalloc.cpp: (WTF::fastMallocDumpMallocStats): (WTF::AvoidRecordingScope::AvoidRecordingScope): (WTF::AvoidRecordingScope::~AvoidRecordingScope): (WTF::MallocCallTracker::MallocSiteData::MallocSiteData): (WTF::MallocCallTracker::singleton): (WTF::MallocCallTracker::MallocCallTracker): (WTF::MallocCallTracker::recordMalloc): (WTF::MallocCallTracker::recordRealloc): (WTF::MallocCallTracker::recordFree): (WTF::MallocCallTracker::dumpStats): (WTF::fastMalloc): (WTF::fastRealloc): (WTF::fastFree): (WTF::fastAlignedMalloc): (WTF::tryFastAlignedMalloc): (WTF::fastAlignedFree): * wtf/FastMalloc.h: (WTF::FastMalloc::zeroedMalloc): (WTF::FastMalloc::tryZeroedMalloc): * wtf/Forward.h: * wtf/HashTable.cpp: * wtf/HashTable.h: (WTF::KeyTraits>::allocateTable): (WTF::KeyTraits>::deallocateTable): (WTF::KeyTraits>::rehash): * wtf/MallocPtr.h: (WTF::MallocPtr::MallocPtr): (WTF::MallocPtr::malloc): (WTF::MallocPtr::zeroedMalloc): (WTF::MallocPtr::tryMalloc): (WTF::MallocPtr::tryZeroedMalloc): (WTF::adoptMallocPtr): * wtf/MetaAllocator.cpp: (WTF::MetaAllocator::allocFreeSpaceNode): (WTF::MetaAllocator::freeFreeSpaceNode): * wtf/MetaAllocatorHandle.h: * wtf/Platform.h: * wtf/RefCountedArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/RefCountedArray.h: (WTF::RefCountedArray::RefCountedArray): (WTF::RefCountedArray::~RefCountedArray): (WTF::RefCountedArray::assign): * wtf/SegmentedVector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SegmentedVector.h: * wtf/SmallPtrSet.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SmallPtrSet.h: (WTF::SmallPtrSet::~SmallPtrSet): (WTF::SmallPtrSet::grow): * wtf/UniqueArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/UniqueArray.h: (WTF::UniqueArrayFree::operator() const): (WTF::UniqueArrayFree<T::operator() const): * wtf/Vector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::VectorBufferBase::releaseBuffer): (WTF::VectorBuffer::releaseBuffer): (WTF::Vector::swap): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::uncheckedConstructAndAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::Malloc>::isolatedCopy const): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::uncheckedConstructAndAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. (WTF::minCapacity>::isolatedCopy const): Deleted. * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): (WTF::StringBuffer::release): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): * wtf/text/StringImpl.h: (WTF::StringImpl::StringImpl): (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): * wtf/text/cf/StringImplCF.cpp: (WTF::StringWrapperCFAllocator::allocate): (WTF::StringWrapperCFAllocator::reallocate): (WTF::StringWrapperCFAllocator::deallocate): Canonical link: https://commits.webkit.org/218863@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253987 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-01-03 02:36:43 +00:00
FastBitVectorMalloc::free(m_words);
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
m_words = newArray;
m_numBits = other.m_numBits;
}
void FastBitVectorWordOwner::resizeSlow(size_t numBits)
{
size_t newLength = fastBitVectorArrayLength(numBits);
RELEASE_ASSERT(newLength >= arrayLength());
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
// Use fastCalloc instead of fastRealloc because we expect the common
// use case for this method to be initializing the size of the bitvector.
Experiment: create lots of different malloc zones for easier accounting of memory use https://bugs.webkit.org/show_bug.cgi?id=186422 Patch by Yusuke Suzuki <ysuzuki@apple.com> and Simon Fraser <simon.fraser@apple.com> on 2020-01-02 Reviewed by Saam Barati. Source/bmalloc: * bmalloc/BPlatform.h: * bmalloc/Environment.cpp: (bmalloc::Environment::computeIsDebugHeapEnabled): * bmalloc/IsoHeap.h: (bmalloc::api::IsoHeap::IsoHeap): * bmalloc/IsoHeapInlines.h: (bmalloc::api::IsoHeap<Type>::IsoHeap): * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::allocateSlow): (bmalloc::IsoTLS::deallocateSlow): Source/JavaScriptCore: * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * assembler/AssemblerBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * assembler/AssemblerBuffer.h: (JSC::AssemblerData::AssemblerData): (JSC::AssemblerData::operator=): (JSC::AssemblerData::~AssemblerData): (JSC::AssemblerData::grow): * bytecode/AccessCase.cpp: * bytecode/AccessCase.h: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeBasicBlock.h: * bytecode/CodeBlock.cpp: * bytecode/CodeBlock.h: * bytecode/InstructionStream.cpp: * bytecode/InstructionStream.h: * bytecode/PolymorphicAccess.cpp: * bytecode/PolymorphicAccess.h: * bytecode/UnlinkedMetadataTable.cpp: (JSC::UnlinkedMetadataTable::finalize): * bytecode/UnlinkedMetadataTable.h: * bytecode/UnlinkedMetadataTableInlines.h: (JSC::UnlinkedMetadataTable::UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::~UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::link): (JSC::UnlinkedMetadataTable::unlink): * bytecode/ValueProfile.h: (JSC::ValueProfileAndVirtualRegisterBuffer::ValueProfileAndVirtualRegisterBuffer): * bytecode/Watchpoint.cpp: * bytecode/Watchpoint.h: * dfg/DFGBasicBlock.cpp: * dfg/DFGBasicBlock.h: * dfg/DFGNode.cpp: * dfg/DFGNode.h: * dfg/DFGSpeculativeJIT.cpp: * dfg/DFGSpeculativeJIT.h: * heap/BlockDirectory.cpp: * heap/BlockDirectory.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::freeAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GCSegmentedArray.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * heap/GCSegmentedArray.h: * heap/GCSegmentedArrayInlines.h: (JSC::GCArraySegment<T>::create): (JSC::GCArraySegment<T>::destroy): * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/IsoSubspace.cpp: (JSC::IsoSubspace::IsoSubspace): * heap/MarkedBlock.cpp: * heap/MarkedBlock.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::destroy): * heap/WeakBlock.h: * jit/JITCode.cpp: * jit/JITCode.h: * jit/RegisterAtOffsetList.cpp: * jit/RegisterAtOffsetList.h: * parser/Nodes.cpp: * parser/Nodes.h: * parser/ParserArena.cpp: (JSC::ParserArena::deallocateObjects): (JSC::ParserArena::allocateFreeablePool): * parser/ParserArena.h: * parser/SourceProvider.cpp: * parser/SourceProvider.h: * parser/SourceProviderCache.cpp: * parser/SourceProviderCache.h: * parser/SourceProviderCacheItem.h: (JSC::SourceProviderCacheItem::create): * runtime/CachePayload.cpp: (JSC::CachePayload::makeMallocPayload): * runtime/CachePayload.h: * runtime/CachedBytecode.h: (JSC::CachedBytecode::create): * runtime/CachedTypes.cpp: (JSC::Encoder::release): (JSC::Encoder::Page::Page): (JSC::CachedVector::encode): (JSC::CachedVector::decode const): (JSC::CachedInstructionStream::decode const): * runtime/PropertyMapHashTable.h: (JSC::PropertyTable::rehash): * runtime/PropertyTable.cpp: (JSC::PropertyTable::PropertyTable): (JSC::PropertyTable::~PropertyTable): * runtime/SymbolTable.cpp: * runtime/SymbolTable.h: * runtime/VM.cpp: (JSC::VM::~VM): * runtime/VM.h: (JSC::ScratchBuffer::create): (JSC::VM::exceptionFuzzingBuffer): * wasm/WasmInstance.cpp: (JSC::Wasm::Instance::Instance): * wasm/WasmInstance.h: * wasm/WasmTable.cpp: (JSC::Wasm::Table::Table): (JSC::Wasm::FuncRefTable::FuncRefTable): * wasm/WasmTable.h: Source/WebCore: * Sources.txt: * WebCore.xcodeproj/project.pbxproj: * bindings/js/SerializedScriptValue.cpp: * bindings/js/SerializedScriptValue.h: * css/CSSFontFace.cpp: * css/CSSFontFace.h: * css/CSSSelector.cpp: * css/CSSSelector.h: * css/CSSValue.cpp: * css/CSSValue.h: * css/StyleProperties.cpp: (WebCore::ImmutableStyleProperties::create): * css/StyleProperties.h: * css/StyleRule.cpp: * css/StyleRule.h: * dom/ElementData.cpp: (WebCore::ShareableElementData::createWithAttributes): (WebCore::UniqueElementData::makeShareableCopy const): * dom/ElementData.h: * dom/NodeRareData.cpp: * dom/NodeRareData.h: * dom/QualifiedName.cpp: * dom/QualifiedName.h: * html/parser/HTMLDocumentParser.cpp: * html/parser/HTMLDocumentParser.h: * loader/DocumentLoader.cpp: * loader/DocumentLoader.h: * loader/ResourceLoader.cpp: * loader/ResourceLoader.h: * loader/cache/CachedResource.cpp: * loader/cache/CachedResource.h: * page/PerformanceEntry.cpp: * page/PerformanceEntry.h: * platform/graphics/Font.cpp: * platform/graphics/Font.h: * platform/graphics/FontCascadeFonts.cpp: * platform/graphics/FontCascadeFonts.h: * platform/graphics/Region.cpp: * platform/graphics/Region.h: * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::releaseUint8Vector): * platform/graphics/cg/ImageBufferCG.cpp: (WebCore::ImageBuffer::ImageBuffer): * platform/graphics/nicosia/NicosiaBuffer.cpp: (Nicosia::Buffer::Buffer): * platform/network/ResourceHandle.cpp: * platform/network/ResourceHandleInternal.h: * platform/network/cf/FormDataStreamCFNet.cpp: (WebCore::closeCurrentStream): (WebCore::advanceCurrentStream): * rendering/RenderLayer.cpp: * rendering/RenderLayer.h: * rendering/TableLayout.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * rendering/TableLayout.h: * rendering/style/RenderStyle.cpp: * rendering/style/RenderStyle.h: * rendering/style/SVGRenderStyle.cpp: * rendering/style/SVGRenderStyle.h: * rendering/style/SVGRenderStyleDefs.cpp: * rendering/style/SVGRenderStyleDefs.h: * rendering/style/StyleBoxData.cpp: * rendering/style/StyleBoxData.h: * rendering/style/StyleInheritedData.cpp: * rendering/style/StyleInheritedData.h: * rendering/style/StyleRareInheritedData.cpp: * rendering/style/StyleRareInheritedData.h: * rendering/style/StyleRareNonInheritedData.cpp: * rendering/style/StyleRareNonInheritedData.h: * rendering/style/StyleSurroundData.cpp: * rendering/style/StyleSurroundData.h: * rendering/style/StyleTransformData.cpp: * rendering/style/StyleTransformData.h: * style/StyleTreeResolver.cpp: * style/StyleTreeResolver.h: * svg/animation/SMILTimeContainer.cpp: * svg/animation/SMILTimeContainer.h: Source/WebKit: * Shared/ShareableBitmap.cpp: (WebKit::ShareableBitmap::create): (WebKit::ShareableBitmap::~ShareableBitmap): * UIProcess/mac/LegacySessionStateCoding.cpp: (WebKit::HistoryEntryDataEncoder::HistoryEntryDataEncoder): (WebKit::HistoryEntryDataEncoder::finishEncoding): (WebKit::encodeSessionHistoryEntryData): (WebKit::encodeLegacySessionState): Source/WTF: This patch introduces ENABLE(MALLOC_HEAP_BREAKDOWN). If this is enabled, we allocate malloc_zone per malloc kind. This offers the way to investigate the usage of memory per kind by using vmmap, like the following. VIRTUAL RESIDENT DIRTY SWAPPED ALLOCATION BYTES DIRTY+SWAP REGION MALLOC ZONE SIZE SIZE SIZE SIZE COUNT ALLOCATED FRAG SIZE % FRAG COUNT =========== ======= ========= ========= ========= ========= ========= ========= ====== ====== StringImpl_0x116efd000 188.0M 69.3M 30.9M 0K 139456 18.0M 12.9M 42% 34 DefaultMallocZone_0x10f487000 176.0M 53.9M 14.1M 0K 115956 9955K 4497K 32% 22 Vector_0x116eff000 162.0M 56.3M 55.3M 0K 140715 17.3M 37.9M 69% 36 MetadataTable_0x11843b000 152.0M 17.5M 17.5M 0K 14200 2353K 15.2M 87% 26 WebKit Using System Malloc_0x114cbe000 150.0M 31.6M 21.8M 0K 87422 16.7M 5278K 24% 23 InstructionStream_0x118469000 150.0M 5764K 5764K 0K 14470 4688K 1076K 19% 24 AssemblerData_0x117ee6000 150.0M 1928K 1928K 0K 1 16 1928K 100% 24 To achieve this goal without making very large change, we put a template type in various containers. For example, Vector will take Malloc parameter (the default one is FastMalloc allocator). If ENABLE(MALLOC_HEAP_BREAKDOWN) is enabled, we change this to specific VectorMalloc allocator, and vmmap can show memory usage of this allocator. This patch also supports malloc_zone per IsoHeap. So we can see memory allocation per IsoHeap in vmmap. To use this feature, we need to flip two compile time flags, ENABLE(MALLOC_HEAP_BREAKDOWN) in WTF and BENABLE_MALLOC_HEAP_BREAKDOWN in bmalloc. And use `vmmap $PID` to dump malloc zones. To allocate objects of a class with a specific malloc-zone, use WTF_MAKE_FAST_ALLOCATED_WITH_HEAP_IDENTIFIER(HeapIdentifier) for the class, and define allocator by DECLARE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a header and DEFINE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a cpp file. This patch also introduce callstack collector for malloc. Vector, HashMap etc. are used to allocate various things, but the above malloc_zone feature only tells thing like "Vector takes XXX MB memory". But what we want to know in this case is what Vector is consuming memory. We collect StackShot for each malloc call, and combine these information to tell which callsite is consuming much memory, which tell us that what Vector is consuming memory. * WTF.xcodeproj/project.pbxproj: * wtf/Bag.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/Bag.h: (WTF::Private::BagNode::BagNode): Deleted. * wtf/BitVector.cpp: (WTF::BitVector::OutOfLineBits::create): (WTF::BitVector::OutOfLineBits::destroy): * wtf/CMakeLists.txt: * wtf/ConcurrentBuffer.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/ConcurrentBuffer.h: * wtf/DebugHeap.cpp: Copied from Source/JavaScriptCore/runtime/CachePayload.cpp. (WTF::DebugHeap::DebugHeap): (WTF::DebugHeap::malloc): (WTF::DebugHeap::calloc): (WTF::DebugHeap::memalign): (WTF::DebugHeap::realloc): (WTF::DebugHeap::free): * wtf/DebugHeap.h: Added. * wtf/FastBitVector.cpp: (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::~FastBitVectorWordOwner): * wtf/FastMalloc.cpp: (WTF::fastMallocDumpMallocStats): (WTF::AvoidRecordingScope::AvoidRecordingScope): (WTF::AvoidRecordingScope::~AvoidRecordingScope): (WTF::MallocCallTracker::MallocSiteData::MallocSiteData): (WTF::MallocCallTracker::singleton): (WTF::MallocCallTracker::MallocCallTracker): (WTF::MallocCallTracker::recordMalloc): (WTF::MallocCallTracker::recordRealloc): (WTF::MallocCallTracker::recordFree): (WTF::MallocCallTracker::dumpStats): (WTF::fastMalloc): (WTF::fastRealloc): (WTF::fastFree): (WTF::fastAlignedMalloc): (WTF::tryFastAlignedMalloc): (WTF::fastAlignedFree): * wtf/FastMalloc.h: (WTF::FastMalloc::zeroedMalloc): (WTF::FastMalloc::tryZeroedMalloc): * wtf/Forward.h: * wtf/HashTable.cpp: * wtf/HashTable.h: (WTF::KeyTraits>::allocateTable): (WTF::KeyTraits>::deallocateTable): (WTF::KeyTraits>::rehash): * wtf/MallocPtr.h: (WTF::MallocPtr::MallocPtr): (WTF::MallocPtr::malloc): (WTF::MallocPtr::zeroedMalloc): (WTF::MallocPtr::tryMalloc): (WTF::MallocPtr::tryZeroedMalloc): (WTF::adoptMallocPtr): * wtf/MetaAllocator.cpp: (WTF::MetaAllocator::allocFreeSpaceNode): (WTF::MetaAllocator::freeFreeSpaceNode): * wtf/MetaAllocatorHandle.h: * wtf/Platform.h: * wtf/RefCountedArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/RefCountedArray.h: (WTF::RefCountedArray::RefCountedArray): (WTF::RefCountedArray::~RefCountedArray): (WTF::RefCountedArray::assign): * wtf/SegmentedVector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SegmentedVector.h: * wtf/SmallPtrSet.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SmallPtrSet.h: (WTF::SmallPtrSet::~SmallPtrSet): (WTF::SmallPtrSet::grow): * wtf/UniqueArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/UniqueArray.h: (WTF::UniqueArrayFree::operator() const): (WTF::UniqueArrayFree<T::operator() const): * wtf/Vector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::VectorBufferBase::releaseBuffer): (WTF::VectorBuffer::releaseBuffer): (WTF::Vector::swap): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::uncheckedConstructAndAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::Malloc>::isolatedCopy const): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::uncheckedConstructAndAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. (WTF::minCapacity>::isolatedCopy const): Deleted. * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): (WTF::StringBuffer::release): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): * wtf/text/StringImpl.h: (WTF::StringImpl::StringImpl): (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): * wtf/text/cf/StringImplCF.cpp: (WTF::StringWrapperCFAllocator::allocate): (WTF::StringWrapperCFAllocator::reallocate): (WTF::StringWrapperCFAllocator::deallocate): Canonical link: https://commits.webkit.org/218863@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253987 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-01-03 02:36:43 +00:00
uint32_t* newArray = static_cast<uint32_t*>(FastBitVectorMalloc::zeroedMalloc(newLength * sizeof(uint32_t)));
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
memcpy(newArray, m_words, arrayLength() * sizeof(uint32_t));
if (m_words)
Experiment: create lots of different malloc zones for easier accounting of memory use https://bugs.webkit.org/show_bug.cgi?id=186422 Patch by Yusuke Suzuki <ysuzuki@apple.com> and Simon Fraser <simon.fraser@apple.com> on 2020-01-02 Reviewed by Saam Barati. Source/bmalloc: * bmalloc/BPlatform.h: * bmalloc/Environment.cpp: (bmalloc::Environment::computeIsDebugHeapEnabled): * bmalloc/IsoHeap.h: (bmalloc::api::IsoHeap::IsoHeap): * bmalloc/IsoHeapInlines.h: (bmalloc::api::IsoHeap<Type>::IsoHeap): * bmalloc/IsoTLSInlines.h: (bmalloc::IsoTLS::allocateSlow): (bmalloc::IsoTLS::deallocateSlow): Source/JavaScriptCore: * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * assembler/AssemblerBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * assembler/AssemblerBuffer.h: (JSC::AssemblerData::AssemblerData): (JSC::AssemblerData::operator=): (JSC::AssemblerData::~AssemblerData): (JSC::AssemblerData::grow): * bytecode/AccessCase.cpp: * bytecode/AccessCase.h: * bytecode/BytecodeBasicBlock.cpp: * bytecode/BytecodeBasicBlock.h: * bytecode/CodeBlock.cpp: * bytecode/CodeBlock.h: * bytecode/InstructionStream.cpp: * bytecode/InstructionStream.h: * bytecode/PolymorphicAccess.cpp: * bytecode/PolymorphicAccess.h: * bytecode/UnlinkedMetadataTable.cpp: (JSC::UnlinkedMetadataTable::finalize): * bytecode/UnlinkedMetadataTable.h: * bytecode/UnlinkedMetadataTableInlines.h: (JSC::UnlinkedMetadataTable::UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::~UnlinkedMetadataTable): (JSC::UnlinkedMetadataTable::link): (JSC::UnlinkedMetadataTable::unlink): * bytecode/ValueProfile.h: (JSC::ValueProfileAndVirtualRegisterBuffer::ValueProfileAndVirtualRegisterBuffer): * bytecode/Watchpoint.cpp: * bytecode/Watchpoint.h: * dfg/DFGBasicBlock.cpp: * dfg/DFGBasicBlock.h: * dfg/DFGNode.cpp: * dfg/DFGNode.h: * dfg/DFGSpeculativeJIT.cpp: * dfg/DFGSpeculativeJIT.h: * heap/BlockDirectory.cpp: * heap/BlockDirectory.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::FastMallocAlignedMemoryAllocator): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::freeAlignedMemory): (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GCSegmentedArray.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * heap/GCSegmentedArray.h: * heap/GCSegmentedArrayInlines.h: (JSC::GCArraySegment<T>::create): (JSC::GCArraySegment<T>::destroy): * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/IsoSubspace.cpp: (JSC::IsoSubspace::IsoSubspace): * heap/MarkedBlock.cpp: * heap/MarkedBlock.h: * heap/WeakBlock.cpp: (JSC::WeakBlock::create): (JSC::WeakBlock::destroy): * heap/WeakBlock.h: * jit/JITCode.cpp: * jit/JITCode.h: * jit/RegisterAtOffsetList.cpp: * jit/RegisterAtOffsetList.h: * parser/Nodes.cpp: * parser/Nodes.h: * parser/ParserArena.cpp: (JSC::ParserArena::deallocateObjects): (JSC::ParserArena::allocateFreeablePool): * parser/ParserArena.h: * parser/SourceProvider.cpp: * parser/SourceProvider.h: * parser/SourceProviderCache.cpp: * parser/SourceProviderCache.h: * parser/SourceProviderCacheItem.h: (JSC::SourceProviderCacheItem::create): * runtime/CachePayload.cpp: (JSC::CachePayload::makeMallocPayload): * runtime/CachePayload.h: * runtime/CachedBytecode.h: (JSC::CachedBytecode::create): * runtime/CachedTypes.cpp: (JSC::Encoder::release): (JSC::Encoder::Page::Page): (JSC::CachedVector::encode): (JSC::CachedVector::decode const): (JSC::CachedInstructionStream::decode const): * runtime/PropertyMapHashTable.h: (JSC::PropertyTable::rehash): * runtime/PropertyTable.cpp: (JSC::PropertyTable::PropertyTable): (JSC::PropertyTable::~PropertyTable): * runtime/SymbolTable.cpp: * runtime/SymbolTable.h: * runtime/VM.cpp: (JSC::VM::~VM): * runtime/VM.h: (JSC::ScratchBuffer::create): (JSC::VM::exceptionFuzzingBuffer): * wasm/WasmInstance.cpp: (JSC::Wasm::Instance::Instance): * wasm/WasmInstance.h: * wasm/WasmTable.cpp: (JSC::Wasm::Table::Table): (JSC::Wasm::FuncRefTable::FuncRefTable): * wasm/WasmTable.h: Source/WebCore: * Sources.txt: * WebCore.xcodeproj/project.pbxproj: * bindings/js/SerializedScriptValue.cpp: * bindings/js/SerializedScriptValue.h: * css/CSSFontFace.cpp: * css/CSSFontFace.h: * css/CSSSelector.cpp: * css/CSSSelector.h: * css/CSSValue.cpp: * css/CSSValue.h: * css/StyleProperties.cpp: (WebCore::ImmutableStyleProperties::create): * css/StyleProperties.h: * css/StyleRule.cpp: * css/StyleRule.h: * dom/ElementData.cpp: (WebCore::ShareableElementData::createWithAttributes): (WebCore::UniqueElementData::makeShareableCopy const): * dom/ElementData.h: * dom/NodeRareData.cpp: * dom/NodeRareData.h: * dom/QualifiedName.cpp: * dom/QualifiedName.h: * html/parser/HTMLDocumentParser.cpp: * html/parser/HTMLDocumentParser.h: * loader/DocumentLoader.cpp: * loader/DocumentLoader.h: * loader/ResourceLoader.cpp: * loader/ResourceLoader.h: * loader/cache/CachedResource.cpp: * loader/cache/CachedResource.h: * page/PerformanceEntry.cpp: * page/PerformanceEntry.h: * platform/graphics/Font.cpp: * platform/graphics/Font.h: * platform/graphics/FontCascadeFonts.cpp: * platform/graphics/FontCascadeFonts.h: * platform/graphics/Region.cpp: * platform/graphics/Region.h: * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::releaseUint8Vector): * platform/graphics/cg/ImageBufferCG.cpp: (WebCore::ImageBuffer::ImageBuffer): * platform/graphics/nicosia/NicosiaBuffer.cpp: (Nicosia::Buffer::Buffer): * platform/network/ResourceHandle.cpp: * platform/network/ResourceHandleInternal.h: * platform/network/cf/FormDataStreamCFNet.cpp: (WebCore::closeCurrentStream): (WebCore::advanceCurrentStream): * rendering/RenderLayer.cpp: * rendering/RenderLayer.h: * rendering/TableLayout.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * rendering/TableLayout.h: * rendering/style/RenderStyle.cpp: * rendering/style/RenderStyle.h: * rendering/style/SVGRenderStyle.cpp: * rendering/style/SVGRenderStyle.h: * rendering/style/SVGRenderStyleDefs.cpp: * rendering/style/SVGRenderStyleDefs.h: * rendering/style/StyleBoxData.cpp: * rendering/style/StyleBoxData.h: * rendering/style/StyleInheritedData.cpp: * rendering/style/StyleInheritedData.h: * rendering/style/StyleRareInheritedData.cpp: * rendering/style/StyleRareInheritedData.h: * rendering/style/StyleRareNonInheritedData.cpp: * rendering/style/StyleRareNonInheritedData.h: * rendering/style/StyleSurroundData.cpp: * rendering/style/StyleSurroundData.h: * rendering/style/StyleTransformData.cpp: * rendering/style/StyleTransformData.h: * style/StyleTreeResolver.cpp: * style/StyleTreeResolver.h: * svg/animation/SMILTimeContainer.cpp: * svg/animation/SMILTimeContainer.h: Source/WebKit: * Shared/ShareableBitmap.cpp: (WebKit::ShareableBitmap::create): (WebKit::ShareableBitmap::~ShareableBitmap): * UIProcess/mac/LegacySessionStateCoding.cpp: (WebKit::HistoryEntryDataEncoder::HistoryEntryDataEncoder): (WebKit::HistoryEntryDataEncoder::finishEncoding): (WebKit::encodeSessionHistoryEntryData): (WebKit::encodeLegacySessionState): Source/WTF: This patch introduces ENABLE(MALLOC_HEAP_BREAKDOWN). If this is enabled, we allocate malloc_zone per malloc kind. This offers the way to investigate the usage of memory per kind by using vmmap, like the following. VIRTUAL RESIDENT DIRTY SWAPPED ALLOCATION BYTES DIRTY+SWAP REGION MALLOC ZONE SIZE SIZE SIZE SIZE COUNT ALLOCATED FRAG SIZE % FRAG COUNT =========== ======= ========= ========= ========= ========= ========= ========= ====== ====== StringImpl_0x116efd000 188.0M 69.3M 30.9M 0K 139456 18.0M 12.9M 42% 34 DefaultMallocZone_0x10f487000 176.0M 53.9M 14.1M 0K 115956 9955K 4497K 32% 22 Vector_0x116eff000 162.0M 56.3M 55.3M 0K 140715 17.3M 37.9M 69% 36 MetadataTable_0x11843b000 152.0M 17.5M 17.5M 0K 14200 2353K 15.2M 87% 26 WebKit Using System Malloc_0x114cbe000 150.0M 31.6M 21.8M 0K 87422 16.7M 5278K 24% 23 InstructionStream_0x118469000 150.0M 5764K 5764K 0K 14470 4688K 1076K 19% 24 AssemblerData_0x117ee6000 150.0M 1928K 1928K 0K 1 16 1928K 100% 24 To achieve this goal without making very large change, we put a template type in various containers. For example, Vector will take Malloc parameter (the default one is FastMalloc allocator). If ENABLE(MALLOC_HEAP_BREAKDOWN) is enabled, we change this to specific VectorMalloc allocator, and vmmap can show memory usage of this allocator. This patch also supports malloc_zone per IsoHeap. So we can see memory allocation per IsoHeap in vmmap. To use this feature, we need to flip two compile time flags, ENABLE(MALLOC_HEAP_BREAKDOWN) in WTF and BENABLE_MALLOC_HEAP_BREAKDOWN in bmalloc. And use `vmmap $PID` to dump malloc zones. To allocate objects of a class with a specific malloc-zone, use WTF_MAKE_FAST_ALLOCATED_WITH_HEAP_IDENTIFIER(HeapIdentifier) for the class, and define allocator by DECLARE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a header and DEFINE_ALLOCATOR_WITH_HEAP_IDENTIFIER(HeapIdentifier) in a cpp file. This patch also introduce callstack collector for malloc. Vector, HashMap etc. are used to allocate various things, but the above malloc_zone feature only tells thing like "Vector takes XXX MB memory". But what we want to know in this case is what Vector is consuming memory. We collect StackShot for each malloc call, and combine these information to tell which callsite is consuming much memory, which tell us that what Vector is consuming memory. * WTF.xcodeproj/project.pbxproj: * wtf/Bag.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/Bag.h: (WTF::Private::BagNode::BagNode): Deleted. * wtf/BitVector.cpp: (WTF::BitVector::OutOfLineBits::create): (WTF::BitVector::OutOfLineBits::destroy): * wtf/CMakeLists.txt: * wtf/ConcurrentBuffer.cpp: Copied from Source/JavaScriptCore/parser/SourceProviderCache.cpp. * wtf/ConcurrentBuffer.h: * wtf/DebugHeap.cpp: Copied from Source/JavaScriptCore/runtime/CachePayload.cpp. (WTF::DebugHeap::DebugHeap): (WTF::DebugHeap::malloc): (WTF::DebugHeap::calloc): (WTF::DebugHeap::memalign): (WTF::DebugHeap::realloc): (WTF::DebugHeap::free): * wtf/DebugHeap.h: Added. * wtf/FastBitVector.cpp: (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::~FastBitVectorWordOwner): * wtf/FastMalloc.cpp: (WTF::fastMallocDumpMallocStats): (WTF::AvoidRecordingScope::AvoidRecordingScope): (WTF::AvoidRecordingScope::~AvoidRecordingScope): (WTF::MallocCallTracker::MallocSiteData::MallocSiteData): (WTF::MallocCallTracker::singleton): (WTF::MallocCallTracker::MallocCallTracker): (WTF::MallocCallTracker::recordMalloc): (WTF::MallocCallTracker::recordRealloc): (WTF::MallocCallTracker::recordFree): (WTF::MallocCallTracker::dumpStats): (WTF::fastMalloc): (WTF::fastRealloc): (WTF::fastFree): (WTF::fastAlignedMalloc): (WTF::tryFastAlignedMalloc): (WTF::fastAlignedFree): * wtf/FastMalloc.h: (WTF::FastMalloc::zeroedMalloc): (WTF::FastMalloc::tryZeroedMalloc): * wtf/Forward.h: * wtf/HashTable.cpp: * wtf/HashTable.h: (WTF::KeyTraits>::allocateTable): (WTF::KeyTraits>::deallocateTable): (WTF::KeyTraits>::rehash): * wtf/MallocPtr.h: (WTF::MallocPtr::MallocPtr): (WTF::MallocPtr::malloc): (WTF::MallocPtr::zeroedMalloc): (WTF::MallocPtr::tryMalloc): (WTF::MallocPtr::tryZeroedMalloc): (WTF::adoptMallocPtr): * wtf/MetaAllocator.cpp: (WTF::MetaAllocator::allocFreeSpaceNode): (WTF::MetaAllocator::freeFreeSpaceNode): * wtf/MetaAllocatorHandle.h: * wtf/Platform.h: * wtf/RefCountedArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/RefCountedArray.h: (WTF::RefCountedArray::RefCountedArray): (WTF::RefCountedArray::~RefCountedArray): (WTF::RefCountedArray::assign): * wtf/SegmentedVector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SegmentedVector.h: * wtf/SmallPtrSet.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/SmallPtrSet.h: (WTF::SmallPtrSet::~SmallPtrSet): (WTF::SmallPtrSet::grow): * wtf/UniqueArray.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/UniqueArray.h: (WTF::UniqueArrayFree::operator() const): (WTF::UniqueArrayFree<T::operator() const): * wtf/Vector.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::VectorBufferBase::releaseBuffer): (WTF::VectorBuffer::releaseBuffer): (WTF::Vector::swap): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::uncheckedConstructAndAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::Malloc>::isolatedCopy const): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::uncheckedConstructAndAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. (WTF::minCapacity>::isolatedCopy const): Deleted. * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.cpp: Copied from Source/JavaScriptCore/bytecode/InstructionStream.cpp. * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): (WTF::StringBuffer::release): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): * wtf/text/StringImpl.h: (WTF::StringImpl::StringImpl): (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): * wtf/text/cf/StringImplCF.cpp: (WTF::StringWrapperCFAllocator::allocate): (WTF::StringWrapperCFAllocator::reallocate): (WTF::StringWrapperCFAllocator::deallocate): Canonical link: https://commits.webkit.org/218863@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@253987 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-01-03 02:36:43 +00:00
FastBitVectorMalloc::free(m_words);
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
m_words = newArray;
}
The write barrier should be down with TSO https://bugs.webkit.org/show_bug.cgi?id=162316 Reviewed by Geoffrey Garen. Source/JavaScriptCore: This makes our write barrier behave correctly when it races with the collector. The collector wants to do this when visiting: object->cellState = black visit(object) The mutator wants to do this when storing: object->property = newValue if (object->cellState == black) remember(object) Prior to this change, this didn't work right because the compiler would sometimes place barriers before the store to the property and because the mutator did not have adequate fences. Prior to this change, the DFG and FTL would emit this: if (object->cellState == black) remember(object) object->property = newValue Which is wrong, because the object could start being scanned just after the cellState check, at which point the store would be lost. We need to confirm that the state was not black *after* the store! This change was harder than you'd expect: placing the barrier after the store broke B3's ability to do its super crazy ninja CSE on some store-load redundancies. Because the B3 CSE has some moves that the DFG CSE lacks, the DFG CSE's ability to ignore barriers didn't help. I fixed this by having the FTL convey precise heap ranges for the patchpoint corresponding to the barrier slow path. It reads the world (because of the store-load fence) and it writes only cellState (because the B3 heap ranges don't have any way to represent any of the GC's other state, which means that B3 does not have to worry about aliasing with any of that). The collector already uses a store-load fence on x86 just after setting the cellState and before visiting the object. The mutator needs to do the same. But we cannot put a store-load fence of any kind before store barriers, because that causes enormous slow downs. In the worst case, Octane/richards slowed down by 90%! That's crazy! However, the overall slow downs were small enough (0-15% on benchmark suite aggregates) that it would be reasonable if the slow down only happened while the GC was running. Then, the concurrent GC would lift throughput-while-collecting from 0% of peak to 85% of peak. This changes the barrier so that it looks like this: if (object->cellState <= heap.sneakyBlackThreshold) slowPath(object) Where sneakyBlackThreshold is the normal blackThreshold when we're not collecting, or a tautoligical threshold (that makes everything look black) when we are collecting. This turns out to not be any more expensive than the barrier in tip of tree when the GC is not running, or a 0-15% slow-down when it is "running". (Of course we don't run the GC concurrently yet. I still have more work to do.) The slowPath() does some extra work to check if we are concurrently collecting; if so, it does a fence and rechecks if the object really did need that barrier. This also reintroduces elimination of redundant store barriers, which was lost in the last store barrier change. We can only do it when there is no possibility of GC, exit, or exceptions between the two store barriers. We could remove the exit/exception limitation if we taught OSR exit how to buffer store barriers, which is an insane thing to do considering that I've never been able to detect a win from redundant store barrier elimination. I just want us to have it for stupidly obvious situations, like a tight sequence of stores to the same object. This same optimization also sometimes strength-reduces the store barrier so that it uses a constant black threshold rather than the sneaky one, thereby saving one load. Even with all of those optimizations, I still had problems with barrier cost. I found that one of the benchmarks that was being hit particularly hard was JetStream/regexp-2010. Fortunately that benchmark does most of its barriers in a tight C++ loop in RegExpMatchesArray.h. When we know what we're doing, we can defer GC around a bunch of object initializations and then remove all of the barriers between any of the objects allocated within the deferral. Unfortunately, our GC deferral mechanism isn't really performant enough to make this be a worthwhile optimization. The most efficient version of such an optimization that I could come up with was to have a DeferralContext object that houses a boolean that is false by default, but the GC writes true into it if it would have wanted to GC. You thread a pointer to the deferralContext through all of your allocations. This kind of mechanism has the overhead of a zero initialization on the stack on entry and a zero check on exit. This is probably even efficient enough that we could start thinking about having the DFG use it, for example if we found a bounded-time section of code with a lot of barriers and entry/exit sites that aren't totally wacky. This optimization took this patch from 0.68% JetStream regressed to neutral, according to my latest data. Finally, an earlier version of this change put the store-load fence in B3 IR, so I ended up adding FTLOutput support for it and AbstractHeapRepository magic for decorating the heaps. I think we might as well keep that, it'll be useful. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * assembler/MacroAssembler.h: (JSC::MacroAssembler::branch32): * assembler/MacroAssemblerX86_64.h: (JSC::MacroAssemblerX86_64::branch32): (JSC::MacroAssemblerX86_64::branch64): Deleted. * bytecode/PolymorphicAccess.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGAbstractHeap.h: * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGClobbersExitState.cpp: (JSC::DFG::clobbersExitState): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGMayExit.cpp: * dfg/DFGNode.h: (JSC::DFG::Node::isStoreBarrier): * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileStoreBarrier): (JSC::DFG::SpeculativeJIT::storeToWriteBarrierBuffer): Deleted. (JSC::DFG::SpeculativeJIT::writeBarrier): Deleted. * dfg/DFGSpeculativeJIT.h: * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileBaseValueStoreBarrier): Deleted. (JSC::DFG::SpeculativeJIT::writeBarrier): Deleted. * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): (JSC::DFG::SpeculativeJIT::compileBaseValueStoreBarrier): Deleted. (JSC::DFG::SpeculativeJIT::writeBarrier): Deleted. * dfg/DFGStoreBarrierClusteringPhase.cpp: Added. (JSC::DFG::performStoreBarrierClustering): * dfg/DFGStoreBarrierClusteringPhase.h: Added. * dfg/DFGStoreBarrierInsertionPhase.cpp: * dfg/DFGStoreBarrierInsertionPhase.h: * ftl/FTLAbstractHeap.h: (JSC::FTL::AbsoluteAbstractHeap::at): (JSC::FTL::AbsoluteAbstractHeap::operator[]): * ftl/FTLAbstractHeapRepository.cpp: (JSC::FTL::AbstractHeapRepository::decorateFenceRead): (JSC::FTL::AbstractHeapRepository::decorateFenceWrite): (JSC::FTL::AbstractHeapRepository::computeRangesAndDecorateInstructions): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileStoreBarrier): (JSC::FTL::DFG::LowerDFGToB3::storageForTransition): (JSC::FTL::DFG::LowerDFGToB3::lazySlowPath): (JSC::FTL::DFG::LowerDFGToB3::emitStoreBarrier): * ftl/FTLOutput.cpp: (JSC::FTL::Output::fence): (JSC::FTL::Output::absolute): * ftl/FTLOutput.h: * heap/CellState.h: (JSC::isWithinThreshold): (JSC::isBlack): * heap/Heap.cpp: (JSC::Heap::writeBarrierSlowPath): * heap/Heap.h: (JSC::Heap::barrierShouldBeFenced): (JSC::Heap::addressOfBarrierShouldBeFenced): (JSC::Heap::sneakyBlackThreshold): (JSC::Heap::addressOfSneakyBlackThreshold): * heap/HeapInlines.h: (JSC::Heap::writeBarrier): (JSC::Heap::writeBarrierWithoutFence): * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::jumpIfIsRememberedOrInEdenWithoutFence): (JSC::AssemblyHelpers::sneakyJumpIfIsRememberedOrInEden): (JSC::AssemblyHelpers::jumpIfIsRememberedOrInEden): (JSC::AssemblyHelpers::storeBarrierStoreLoadFence): (JSC::AssemblyHelpers::jumpIfStoreBarrierStoreLoadFenceNotNeeded): * jit/JITOperations.cpp: * jit/JITOperations.h: * jit/JITPropertyAccess.cpp: (JSC::JIT::emit_op_put_by_id): (JSC::JIT::emitWriteBarrier): (JSC::JIT::privateCompilePutByVal): * jit/JITPropertyAccess32_64.cpp: (JSC::JIT::emit_op_put_by_id): * llint/LowLevelInterpreter.asm: * offlineasm/x86.rb: * runtime/Options.h: Source/WTF: Added clearRange(), which quickly clears a range of bits. This turned out to be useful for a DFG optimization pass. * wtf/FastBitVector.cpp: (WTF::FastBitVector::clearRange): * wtf/FastBitVector.h: Canonical link: https://commits.webkit.org/180647@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206555 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-28 21:55:53 +00:00
void FastBitVector::clearRange(size_t begin, size_t end)
{
if (end - begin < 32) {
for (size_t i = begin; i < end; ++i)
at(i) = false;
return;
}
size_t endBeginSlop = (begin + 31) & ~31;
size_t beginEndSlop = end & ~31;
for (size_t i = begin; i < endBeginSlop; ++i)
at(i) = false;
for (size_t i = beginEndSlop; i < end; ++i)
at(i) = false;
for (size_t i = endBeginSlop / 32; i < beginEndSlop / 32; ++i)
m_words.word(i) = 0;
}
Make MarkedBlock state tracking support overlapped allocation and marking state https://bugs.webkit.org/show_bug.cgi?id=161581 Reviewed by Geoffrey Garen. JSTests: Add a microbenchmark for why we want to reclaim empty blocks from other allocators. * microbenchmarks/switching-size-classes.js: Added. Source/JavaScriptCore: Concurrent GCs must allow for mutation and allocation during collection. We already know how to mutate during collection. We have a write barrier for that. Allocation during collection is more involved: the collector modifies the the mark bits, as well as other kinds of MarkedBlock state, in-place during a collection. The allocator uses that same MarkedBlock state to decide which regions of memory are free. This works if the allocator never runs while the collector is running, but if we want to allow them to run at the same time, then we need to have two versions of the state: one version built up by the collector and another consumed by the allocator. We clear the collector state at the beginning of collection, and splat the collector state onto the allocator state after collection. This could be super expensive, but we can make it cheap with some cleverness. The biggest observation is just that most of the state is a handful of bits per block: is the block free-listed? is it completely full? completely empty? in the incremental sweeper's snapshot? is it retired? is it in eden? There is also state inside blocks, like the mark bits, but I have a solid plan there and I'll save it for another patch. Once we view the state of blocks as bits, we can put that state into bitvectors, so that if the collector needs to transform the state of some blocks, it can do it with a single operation over bitvectors. I like to think of this as 32-way parallelizing block operations, since doing one operation on a 32-bit word in one of those bitvectors instantly affects 32 blocks. This change converts all previous collections of MarkedBlocks, along with the MarkedBlock state, into 8 bitvectors (live, empty, allocated, canAllocateButNotEmpty, eden, unswept, markingNotEmpty, and markingRetired). The bitvectors separate allocator state (empty, allocated, canAllocateButNotEmpty) from marking state (markingNotEmpty, markingRetired). As a nice side-effect of switching to bitvectors, we get size class rebalancing for free. It used to be that if a MarkedAllocator had an empty block, we would only allow that memory to be reused by a different MarkedAllocator if we did an incremental sweep or a full eager sweep. Now we hunt down all destructorless empty blocks before allocating new MarkedBlocks. It would be relatively easy to also hunt down destructor empty blocks, but the theory is that those might be expensive to sweep, so it might still be better to leave those to the incremental sweeper. This change is perf-neutral all around. I did some tests with two different kinds of allocation strategies - something that is somewhat easier to do now that you can look for blocks that are candidates for allocation by just scanning some bitvectors. I tried two variants: - Allocate out of non-empty blocks first, leaving empty blocks for last in case a different allocator needed them. This is sort of a best-fit strategy. I tried this first, and it can be expressed as: m_allocationCursor = m_canAllocateButNotEmpty.findBit(m_allocationCursor, true) - Allocate out of lower-indexed blocks first, treating empty and canAllocateButNotEmpty blocks equally. This is sort of a first-fit strategy. This is what I ended up settling on, and it can be expressed as: m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true) The best-fit strategy meant 1% regressions in LongSpider and Octane overall, and a 11% regression on Octane/earley. First-fit means perf-neutrality. Most great allocators skew towards first-fit because it's empirically better, so this result is not surprising. Overall, the performance of this patch on my machine is as follows, where "neutral" means less than 1% and not statistically significant. run-jsc-benchmarks: SunSpider: neutral LongSpider: 0.6% slower V8Spider: neutral Octane: neutral Kraken: neutral Microbenchmarks: 0.37% slower AsmBench: neutral CompressionBench: maybe 1% faster For browser benchmarks, I report the ratio of means (bigger / smaller) along with a T-test from Mathematica reported as % chance of not [sic] the null hypothesis. Note that we normally consider anything less than 95% confidence to be inconclusive. Browser benchmarks: PLT3: 0.3% faster with 67% confidence membuster: Snap2FinishedLoadingPost: 0.68% more memory with 50% confidence Snap3EndPost: 2.4% more memory with 61% confidence JetStream: 0.2% slower with 32% confidence Speedometer: 0.7% faster with 82% confidence Additionally, Octane/splay's heap capacity goes down to ~180KB from ~200KB, so about a 10% progression. This is due to the allocator rebalancing feature. Finally, this breaks --useImmortalObjects. It was already broken as far as I can tell. I filed a bug to reimplement it (bug 162296). Unless someone urgently needs this internal tool, it's probably best to reimplement it after I'm done refactoring MarkedSpace. * JavaScriptCore.xcodeproj/project.pbxproj: * debugger/Debugger.cpp: * heap/CellContainer.h: * heap/CellContainerInlines.h: (JSC::CellContainer::vm): (JSC::CellContainer::heap): (JSC::CellContainer::isMarkedOrNewlyAllocated): (JSC::CellContainer::aboutToMark): (JSC::CellContainer::isMarked): Deleted. (JSC::CellContainer::flipIfNecessary): Deleted. * heap/ConservativeRoots.cpp: * heap/Heap.cpp: (JSC::Heap::beginMarking): (JSC::Heap::endMarking): (JSC::Heap::collectAllGarbage): (JSC::Heap::collectImpl): (JSC::Heap::snapshotMarkedSpace): (JSC::Heap::prepareForAllocation): (JSC::Heap::zombifyDeadObjects): (JSC::MarkedBlockSnapshotFunctor::MarkedBlockSnapshotFunctor): Deleted. (JSC::MarkedBlockSnapshotFunctor::operator()): Deleted. (JSC::Heap::resetAllocators): Deleted. * heap/Heap.h: * heap/HeapInlines.h: (JSC::Heap::isMarked): (JSC::Heap::isMarkedConcurrently): (JSC::Heap::testAndSetMarked): * heap/HeapStatistics.cpp: * heap/HeapUtil.h: (JSC::HeapUtil::findGCObjectPointersForMarking): (JSC::HeapUtil::isPointerGCObjectJSCell): * heap/HeapVerifier.cpp: * heap/IncrementalSweeper.cpp: (JSC::IncrementalSweeper::IncrementalSweeper): (JSC::IncrementalSweeper::doSweep): (JSC::IncrementalSweeper::sweepNextBlock): (JSC::IncrementalSweeper::startSweeping): (JSC::IncrementalSweeper::willFinishSweeping): * heap/IncrementalSweeper.h: * heap/LargeAllocation.h: (JSC::LargeAllocation::isMarked): (JSC::LargeAllocation::isMarkedConcurrently): (JSC::LargeAllocation::isMarkedOrNewlyAllocated): (JSC::LargeAllocation::aboutToMark): (JSC::LargeAllocation::isMarkedDuringWeakVisiting): Deleted. (JSC::LargeAllocation::flipIfNecessary): Deleted. (JSC::LargeAllocation::flipIfNecessaryDuringMarking): Deleted. * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::isPagedOut): (JSC::MarkedAllocator::findEmptyBlock): (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl): (JSC::MarkedAllocator::allocateIn): (JSC::MarkedAllocator::tryAllocateIn): (JSC::MarkedAllocator::allocateSlowCaseImpl): (JSC::MarkedAllocator::tryAllocateBlock): (JSC::MarkedAllocator::addBlock): (JSC::MarkedAllocator::removeBlock): (JSC::MarkedAllocator::stopAllocating): (JSC::MarkedAllocator::prepareForAllocation): (JSC::MarkedAllocator::lastChanceToFinalize): (JSC::MarkedAllocator::resumeAllocating): (JSC::MarkedAllocator::beginMarkingForFullCollection): (JSC::MarkedAllocator::endMarking): (JSC::MarkedAllocator::snapshotForEdenCollection): (JSC::MarkedAllocator::snapshotForFullCollection): (JSC::MarkedAllocator::findBlockToSweep): (JSC::MarkedAllocator::sweep): (JSC::MarkedAllocator::shrink): (JSC::MarkedAllocator::assertSnapshotEmpty): (JSC::MarkedAllocator::dump): (JSC::MarkedAllocator::dumpBits): (JSC::MarkedAllocator::retire): Deleted. (JSC::MarkedAllocator::filterNextBlock): Deleted. (JSC::MarkedAllocator::setNextBlockToSweep): Deleted. (JSC::MarkedAllocator::reset): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::forEachBitVector): (JSC::MarkedAllocator::forEachBitVectorWithName): (JSC::MarkedAllocator::nextAllocator): (JSC::MarkedAllocator::setNextAllocator): (JSC::MarkedAllocator::forEachBlock): (JSC::MarkedAllocator::resumeAllocating): Deleted. * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::MarkedBlock): (JSC::MarkedBlock::Handle::specializedSweep): (JSC::MarkedBlock::Handle::sweep): (JSC::MarkedBlock::Handle::sweepHelperSelectScribbleMode): (JSC::MarkedBlock::Handle::sweepHelperSelectEmptyMode): (JSC::MarkedBlock::Handle::sweepHelperSelectHasNewlyAllocated): (JSC::MarkedBlock::Handle::sweepHelperSelectSweepMode): (JSC::MarkedBlock::Handle::sweepHelperSelectFlipMode): (JSC::MarkedBlock::Handle::unsweepWithNoNewlyAllocated): (JSC::MarkedBlock::Handle::setIsFreeListed): (JSC::MarkedBlock::Handle::stopAllocating): (JSC::MarkedBlock::Handle::lastChanceToFinalize): (JSC::MarkedBlock::Handle::resumeAllocating): (JSC::MarkedBlock::aboutToMarkSlow): (JSC::MarkedBlock::clearMarks): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::didConsumeFreeList): (JSC::MarkedBlock::markCount): (JSC::MarkedBlock::Handle::isEmpty): (JSC::MarkedBlock::noteMarkedSlow): (JSC::MarkedBlock::Handle::removeFromAllocator): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::didRemoveFromAllocator): (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::sweepHelperSelectStateAndSweepMode): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessarySlow): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarkingSlow): Deleted. (JSC::MarkedBlock::Handle::willRemoveBlock): Deleted. (WTF::printInternal): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::isFreeListed): (JSC::MarkedBlock::Handle::index): (JSC::MarkedBlock::aboutToMark): (JSC::MarkedBlock::isMarked): (JSC::MarkedBlock::isMarkedConcurrently): (JSC::MarkedBlock::Handle::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::isMarkedOrNewlyAllocated): (JSC::MarkedBlock::Handle::isOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::setIsOnBlocksToSweep): Deleted. (JSC::MarkedBlock::Handle::state): Deleted. (JSC::MarkedBlock::flipIfNecessary): Deleted. (JSC::MarkedBlock::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessary): Deleted. (JSC::MarkedBlock::Handle::flipIfNecessaryDuringMarking): Deleted. (JSC::MarkedBlock::Handle::flipForEdenCollection): Deleted. (JSC::MarkedBlock::isMarkedDuringWeakVisiting): Deleted. (JSC::MarkedBlock::Handle::isLive): Deleted. (JSC::MarkedBlock::Handle::isLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachLiveCell): Deleted. (JSC::MarkedBlock::Handle::forEachDeadCell): Deleted. (JSC::MarkedBlock::Handle::needsSweeping): Deleted. (JSC::MarkedBlock::Handle::isAllocated): Deleted. (JSC::MarkedBlock::Handle::isMarked): Deleted. * heap/MarkedBlockInlines.h: Added. (JSC::MarkedBlock::Handle::isLive): (JSC::MarkedBlock::Handle::isLiveCell): (JSC::MarkedBlock::Handle::forEachLiveCell): (JSC::MarkedBlock::Handle::forEachDeadCell): (JSC::MarkedBlock::resetVersion): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::MarkedSpace): (JSC::MarkedSpace::allocate): (JSC::MarkedSpace::tryAllocate): (JSC::MarkedSpace::sweep): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::shrink): (JSC::MarkedSpace::clearNewlyAllocated): (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::didAllocateInBlock): (JSC::MarkedSpace::findEmptyBlock): (JSC::MarkedSpace::snapshot): (JSC::MarkedSpace::assertSnapshotEmpty): (JSC::MarkedSpace::dumpBits): (JSC::MarkedSpace::zombifySweep): Deleted. (JSC::MarkedSpace::resetAllocators): Deleted. (JSC::VerifyMarked::operator()): Deleted. (JSC::MarkedSpace::flip): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::nextVersion): (JSC::MarkedSpace::firstAllocator): (JSC::MarkedSpace::allocatorForEmptyAllocation): (JSC::MarkedSpace::forEachAllocator): (JSC::MarkedSpace::blocksWithNewObjects): Deleted. (JSC::MarkedSpace::setIsMarking): Deleted. (JSC::MarkedSpace::forEachLiveCell): Deleted. (JSC::MarkedSpace::forEachDeadCell): Deleted. * heap/MarkedSpaceInlines.h: Added. (JSC::MarkedSpace::forEachLiveCell): (JSC::MarkedSpace::forEachDeadCell): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::setMarkedAndAppendToMarkStack): (JSC::SlotVisitor::markAuxiliary): (JSC::SlotVisitor::visitChildren): * heap/Weak.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): (WTF::HashTraits<JSC::Weak<T>>::peek): * heap/WeakBlock.cpp: (JSC::WeakBlock::specializedVisit): (JSC::WeakBlock::reap): * heap/WeakInlines.h: (WTF::HashTraits<JSC::Weak<T>>::emptyValue): Deleted. (WTF::HashTraits<JSC::Weak<T>>::peek): Deleted. * jit/JITThunks.h: * runtime/JSGlobalObject.cpp: * runtime/PrototypeMap.h: * runtime/SamplingProfiler.cpp: * runtime/WeakGCMap.h: * tools/JSDollarVMPrototype.cpp: Source/WTF: The main change here is to bring back FastBitVector.cpp, so that I could outline some large slow path functions. This also adds some utilities, like atomicSetAndCheck() and isEmpty(). The GC uses these. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastBitVector.cpp: Added. (WTF::FastBitVectorWordOwner::setEqualsSlow): (WTF::FastBitVectorWordOwner::resizeSlow): * wtf/FastBitVector.h: (WTF::FastBitVectorWordOwner::operator=): (WTF::FastBitVectorWordOwner::resize): (WTF::FastBitVectorImpl::isEmpty): (WTF::FastBitVector::atomicSetAndCheck): (WTF::FastBitVector::operator[]): Deleted. Tools: Remove the always-trigger-copy-phase configuration. * Scripts/run-jsc-stress-tests: Canonical link: https://commits.webkit.org/180306@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@206154 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2016-09-20 18:12:18 +00:00
} // namespace WTF