Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
/*
|
2018-04-10 23:34:42 +00:00
|
|
|
* Copyright (C) 2017-2018 Apple Inc. All rights reserved.
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
|
|
|
|
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
|
|
|
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
|
|
|
|
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
|
|
|
|
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
|
|
|
|
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
|
|
|
|
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
|
|
|
|
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
|
|
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
|
|
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "Scavenger.h"
|
|
|
|
|
bmalloc should support strictly type-segregated isolated heaps
https://bugs.webkit.org/show_bug.cgi?id=178108
Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa.
Source/bmalloc:
This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by
type and created in static storage. When unused, it takes only a few words. When you do use
it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs
in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will
point to the same object even though they should not have.
IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide
(the JSC GC):
Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls
pages. Pages are collected into directories. Directories track pages using bitvectors, so
that it's easy to quickly find a completely free page or one that has at least one free
object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and
line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit
discipline. However, the real reason why I wrote it that was is that this is what I'm more
familiar with. This is a part of the design I want to revisit (bug 179278).
Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data
structures can be locked with a coarse-grained lock, since the deallocator only grabs it when
flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop
FreeList.
This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's
RenderObject.
Note that despite the use of GC concepts, it's not a goal to make this code directly sharable
with GC. The GC will probably have to do isolated heaps its own way (likely a special
Subspace or something like that).
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Algorithm.h:
(bmalloc::findBitInWord):
* bmalloc/AllIsoHeaps.cpp: Added.
(bmalloc::AllIsoHeaps::AllIsoHeaps):
(bmalloc::AllIsoHeaps::add):
(bmalloc::AllIsoHeaps::head):
* bmalloc/AllIsoHeaps.h: Added.
* bmalloc/AllIsoHeapsInlines.h: Added.
(bmalloc::AllIsoHeaps::forEach):
* bmalloc/BMalloced.h: Added.
* bmalloc/Bits.h: Added.
(bmalloc::bitsArrayLength):
(bmalloc::BitsWordView::BitsWordView):
(bmalloc::BitsWordView::numBits const):
(bmalloc::BitsWordView::word const):
(bmalloc::BitsWordOwner::BitsWordOwner):
(bmalloc::BitsWordOwner::view const):
(bmalloc::BitsWordOwner::operator=):
(bmalloc::BitsWordOwner::setAll):
(bmalloc::BitsWordOwner::clearAll):
(bmalloc::BitsWordOwner::set):
(bmalloc::BitsWordOwner::numBits const):
(bmalloc::BitsWordOwner::arrayLength const):
(bmalloc::BitsWordOwner::word const):
(bmalloc::BitsWordOwner::word):
(bmalloc::BitsWordOwner::words const):
(bmalloc::BitsWordOwner::words):
(bmalloc::BitsAndWords::BitsAndWords):
(bmalloc::BitsAndWords::view const):
(bmalloc::BitsAndWords::numBits const):
(bmalloc::BitsAndWords::word const):
(bmalloc::BitsOrWords::BitsOrWords):
(bmalloc::BitsOrWords::view const):
(bmalloc::BitsOrWords::numBits const):
(bmalloc::BitsOrWords::word const):
(bmalloc::BitsNotWords::BitsNotWords):
(bmalloc::BitsNotWords::view const):
(bmalloc::BitsNotWords::numBits const):
(bmalloc::BitsNotWords::word const):
(bmalloc::BitsImpl::BitsImpl):
(bmalloc::BitsImpl::numBits const):
(bmalloc::BitsImpl::size const):
(bmalloc::BitsImpl::arrayLength const):
(bmalloc::BitsImpl::operator== const):
(bmalloc::BitsImpl::operator!= const):
(bmalloc::BitsImpl::at const):
(bmalloc::BitsImpl::operator[] const):
(bmalloc::BitsImpl::isEmpty const):
(bmalloc::BitsImpl::operator& const):
(bmalloc::BitsImpl::operator| const):
(bmalloc::BitsImpl::operator~ const):
(bmalloc::BitsImpl::forEachSetBit const):
(bmalloc::BitsImpl::forEachClearBit const):
(bmalloc::BitsImpl::forEachBit const):
(bmalloc::BitsImpl::findBit const):
(bmalloc::BitsImpl::findSetBit const):
(bmalloc::BitsImpl::findClearBit const):
(bmalloc::BitsImpl::wordView const):
(bmalloc::BitsImpl::atImpl const):
(bmalloc::Bits::Bits):
(bmalloc::Bits::operator=):
(bmalloc::Bits::resize):
(bmalloc::Bits::setAll):
(bmalloc::Bits::clearAll):
(bmalloc::Bits::setAndCheck):
(bmalloc::Bits::operator|=):
(bmalloc::Bits::operator&=):
(bmalloc::Bits::at const):
(bmalloc::Bits::operator[] const):
(bmalloc::Bits::BitReference::BitReference):
(bmalloc::Bits::BitReference::operator bool const):
(bmalloc::Bits::BitReference::operator=):
(bmalloc::Bits::at):
(bmalloc::Bits::operator[]):
* bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp.
(bmalloc::cryptoRandom):
* bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h.
* bmalloc/DeferredDecommit.h: Added.
* bmalloc/DeferredDecommitInlines.h: Added.
(bmalloc::DeferredDecommit::DeferredDecommit):
* bmalloc/DeferredTrigger.h: Added.
(bmalloc::DeferredTrigger::DeferredTrigger):
* bmalloc/DeferredTriggerInlines.h: Added.
(bmalloc::DeferredTrigger<trigger>::didBecome):
(bmalloc::DeferredTrigger<trigger>::handleDeferral):
* bmalloc/EligibilityResult.h: Added.
(bmalloc::EligibilityResult::EligibilityResult):
* bmalloc/EligibilityResultInlines.h: Added.
(bmalloc::EligibilityResult<Config>::EligibilityResult):
* bmalloc/FixedVector.h:
* bmalloc/FreeList.cpp: Added.
(bmalloc::FreeList::FreeList):
(bmalloc::FreeList::~FreeList):
(bmalloc::FreeList::clear):
(bmalloc::FreeList::initializeList):
(bmalloc::FreeList::initializeBump):
(bmalloc::FreeList::contains const):
* bmalloc/FreeList.h: Added.
(bmalloc::FreeCell::scramble):
(bmalloc::FreeCell::descramble):
(bmalloc::FreeCell::setNext):
(bmalloc::FreeCell::next const):
(bmalloc::FreeList::allocationWillFail const):
(bmalloc::FreeList::allocationWillSucceed const):
(bmalloc::FreeList::originalSize const):
(bmalloc::FreeList::head const):
* bmalloc/FreeListInlines.h: Added.
(bmalloc::FreeList::allocate):
(bmalloc::FreeList::forEach const):
* bmalloc/IsoAllocator.h: Added.
* bmalloc/IsoAllocatorInlines.h: Added.
(bmalloc::IsoAllocator<Config>::IsoAllocator):
(bmalloc::IsoAllocator<Config>::~IsoAllocator):
(bmalloc::IsoAllocator<Config>::allocate):
(bmalloc::IsoAllocator<Config>::allocateSlow):
(bmalloc::IsoAllocator<Config>::scavenge):
* bmalloc/IsoConfig.h: Added.
* bmalloc/IsoDeallocator.h: Added.
* bmalloc/IsoDeallocatorInlines.h: Added.
(bmalloc::IsoDeallocator<Config>::IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::~IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::deallocate):
(bmalloc::IsoDeallocator<Config>::scavenge):
* bmalloc/IsoDirectory.h: Added.
(bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBase::heap):
* bmalloc/IsoDirectoryInlines.h: Added.
(bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase):
(bmalloc::passedNumPages>::IsoDirectory):
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
(bmalloc::passedNumPages>::didDecommit):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::forEachCommittedPage):
* bmalloc/IsoDirectoryPage.h: Added.
(bmalloc::IsoDirectoryPage::index const):
* bmalloc/IsoDirectoryPageInlines.h: Added.
(bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage):
(bmalloc::IsoDirectoryPage<Config>::pageFor):
* bmalloc/IsoHeap.h: Added.
(bmalloc::api::IsoHeap::allocatorOffset):
(bmalloc::api::IsoHeap::setAllocatorOffset):
(bmalloc::api::IsoHeap::deallocatorOffset):
(bmalloc::api::IsoHeap::setDeallocatorOffset):
* bmalloc/IsoHeapImpl.cpp: Added.
(bmalloc::IsoHeapImplBase::IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::~IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::scavengeNow):
(bmalloc::IsoHeapImplBase::finishScavenging):
* bmalloc/IsoHeapImpl.h: Added.
* bmalloc/IsoHeapImplInlines.h: Added.
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
(bmalloc::IsoHeapImpl<Config>::takeFirstEligible):
(bmalloc::IsoHeapImpl<Config>::didBecomeEligible):
(bmalloc::IsoHeapImpl<Config>::scavenge):
(bmalloc::IsoHeapImpl<Config>::allocatorOffset):
(bmalloc::IsoHeapImpl<Config>::deallocatorOffset):
(bmalloc::IsoHeapImpl<Config>::numLiveObjects):
(bmalloc::IsoHeapImpl<Config>::numCommittedPages):
(bmalloc::IsoHeapImpl<Config>::forEachDirectory):
(bmalloc::IsoHeapImpl<Config>::forEachCommittedPage):
(bmalloc::IsoHeapImpl<Config>::forEachLiveObject):
* bmalloc/IsoHeapInlines.h: Added.
(bmalloc::api::IsoHeap<Type>::allocate):
(bmalloc::api::IsoHeap<Type>::tryAllocate):
(bmalloc::api::IsoHeap<Type>::deallocate):
(bmalloc::api::IsoHeap<Type>::scavenge):
(bmalloc::api::IsoHeap<Type>::isInitialized):
(bmalloc::api::IsoHeap<Type>::impl):
* bmalloc/IsoPage.h: Added.
(bmalloc::IsoPage::index const):
(bmalloc::IsoPage::directory):
(bmalloc::IsoPage::isInUseForAllocation const):
(bmalloc::IsoPage::indexOfFirstObject):
* bmalloc/IsoPageInlines.h: Added.
(bmalloc::IsoPage<Config>::tryCreate):
(bmalloc::IsoPage<Config>::IsoPage):
(bmalloc::IsoPage<Config>::free):
(bmalloc::IsoPage<Config>::startAllocating):
(bmalloc::IsoPage<Config>::stopAllocating):
(bmalloc::IsoPage<Config>::forEachLiveObject):
* bmalloc/IsoPageTrigger.h: Added.
* bmalloc/IsoTLS.cpp: Added.
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::destructor):
(bmalloc::IsoTLS::sizeForCapacity):
(bmalloc::IsoTLS::capacityForSize):
(bmalloc::IsoTLS::size):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLS.h: Added.
* bmalloc/IsoTLSAllocatorEntry.h: Added.
* bmalloc/IsoTLSAllocatorEntryInlines.h: Added.
(bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::construct):
* bmalloc/IsoTLSDeallocatorEntry.h: Added.
* bmalloc/IsoTLSDeallocatorEntryInlines.h: Added.
(bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::construct):
* bmalloc/IsoTLSEntry.cpp: Added.
(bmalloc::IsoTLSEntry::IsoTLSEntry):
(bmalloc::IsoTLSEntry::~IsoTLSEntry):
* bmalloc/IsoTLSEntry.h: Added.
(bmalloc::IsoTLSEntry::offset const):
(bmalloc::IsoTLSEntry::alignment const):
(bmalloc::IsoTLSEntry::size const):
(bmalloc::IsoTLSEntry::extent const):
* bmalloc/IsoTLSEntryInlines.h: Added.
(bmalloc::IsoTLSEntry::walkUpToInclusive):
(bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::move):
(bmalloc::DefaultIsoTLSEntry<EntryType>::destruct):
(bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge):
* bmalloc/IsoTLSInlines.h: Added.
(bmalloc::IsoTLS::allocate):
(bmalloc::IsoTLS::deallocate):
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::allocator):
(bmalloc::IsoTLS::deallocator):
(bmalloc::IsoTLS::get):
(bmalloc::IsoTLS::set):
(bmalloc::IsoTLS::ensureHeap):
(bmalloc::IsoTLS::ensureHeapAndEntries):
* bmalloc/IsoTLSLayout.cpp: Added.
(bmalloc::IsoTLSLayout::IsoTLSLayout):
(bmalloc::IsoTLSLayout::add):
* bmalloc/IsoTLSLayout.h: Added.
(bmalloc::IsoTLSLayout::head const):
* bmalloc/PerHeapKind.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h:
* bmalloc/bmalloc.h:
(bmalloc::api::scavengeThisThread):
* test: Added.
* test/testbmalloc.cpp: Added.
(hiddenTruthBecauseNoReturnIsStupid):
(usage):
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
(testIsoSimple):
(testIsoSimpleScavengeBeforeDealloc):
(testIsoFlipFlopFragmentedPages):
(testIsoFlipFlopFragmentedPagesScavengeInMiddle):
(BisoMalloced::BisoMalloced):
(testBisoMalloced):
(BisoMallocedInline::BisoMallocedInline):
(testBisoMallocedInline):
(run):
(main):
Source/WebCore:
No new tests because no new change in behavior. Though, the bmalloc change has a unit test.
Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we
opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never
have to do any kind of switch or dynamic lookup to find the right allocator for a type.
This change is perf-neutral on MotionMark, PLT3, and membuster.
* Sources.txt:
* html/shadow/SliderThumbElement.cpp:
* html/shadow/SliderThumbElement.h:
* html/shadow/mac/ImageControlsButtonElementMac.cpp:
* html/shadow/mac/ImageControlsRootElementMac.cpp:
* rendering/RenderAttachment.cpp:
* rendering/RenderAttachment.h:
* rendering/RenderBlock.cpp:
* rendering/RenderBlock.h:
* rendering/RenderBlockFlow.cpp:
* rendering/RenderBlockFlow.h:
* rendering/RenderBox.cpp:
* rendering/RenderBox.h:
* rendering/RenderBoxModelObject.cpp:
* rendering/RenderBoxModelObject.h:
* rendering/RenderButton.cpp:
* rendering/RenderButton.h:
* rendering/RenderCombineText.cpp:
* rendering/RenderCombineText.h:
* rendering/RenderCounter.cpp:
* rendering/RenderCounter.h:
* rendering/RenderDeprecatedFlexibleBox.cpp:
* rendering/RenderDeprecatedFlexibleBox.h:
* rendering/RenderDetailsMarker.cpp:
* rendering/RenderDetailsMarker.h:
* rendering/RenderElement.cpp:
* rendering/RenderElement.h:
* rendering/RenderEmbeddedObject.cpp:
* rendering/RenderEmbeddedObject.h:
* rendering/RenderFileUploadControl.cpp:
* rendering/RenderFileUploadControl.h:
* rendering/RenderFlexibleBox.cpp:
* rendering/RenderFlexibleBox.h:
* rendering/RenderFragmentContainer.cpp:
* rendering/RenderFragmentContainer.h:
* rendering/RenderFragmentContainerSet.cpp:
* rendering/RenderFragmentContainerSet.h:
* rendering/RenderFragmentedFlow.cpp:
* rendering/RenderFragmentedFlow.h:
* rendering/RenderFrameBase.cpp:
* rendering/RenderFrameBase.h:
* rendering/RenderFrameSet.cpp:
* rendering/RenderFrameSet.h:
* rendering/RenderFullScreen.cpp:
* rendering/RenderFullScreen.h:
* rendering/RenderGrid.cpp:
* rendering/RenderGrid.h:
* rendering/RenderHTMLCanvas.cpp:
* rendering/RenderHTMLCanvas.h:
* rendering/RenderImage.cpp:
* rendering/RenderImage.h:
* rendering/RenderImageResourceStyleImage.cpp:
* rendering/RenderImageResourceStyleImage.h:
* rendering/RenderInline.cpp:
* rendering/RenderInline.h:
* rendering/RenderLayerModelObject.cpp:
* rendering/RenderLayerModelObject.h:
* rendering/RenderLineBreak.cpp:
* rendering/RenderLineBreak.h:
* rendering/RenderListBox.cpp:
* rendering/RenderListBox.h:
* rendering/RenderListItem.cpp:
* rendering/RenderListItem.h:
* rendering/RenderListMarker.cpp:
* rendering/RenderListMarker.h:
* rendering/RenderMedia.cpp:
* rendering/RenderMedia.h:
* rendering/RenderMediaControlElements.cpp:
* rendering/RenderMediaControlElements.h:
* rendering/RenderMenuList.cpp:
* rendering/RenderMenuList.h:
* rendering/RenderMeter.cpp:
* rendering/RenderMeter.h:
* rendering/RenderMultiColumnFlow.cpp:
* rendering/RenderMultiColumnFlow.h:
* rendering/RenderMultiColumnSet.cpp:
* rendering/RenderMultiColumnSet.h:
* rendering/RenderMultiColumnSpannerPlaceholder.cpp:
* rendering/RenderMultiColumnSpannerPlaceholder.h:
* rendering/RenderObject.cpp:
* rendering/RenderObject.h:
* rendering/RenderProgress.cpp:
* rendering/RenderProgress.h:
* rendering/RenderQuote.cpp:
* rendering/RenderQuote.h:
* rendering/RenderReplaced.cpp:
* rendering/RenderReplaced.h:
* rendering/RenderReplica.cpp:
* rendering/RenderReplica.h:
* rendering/RenderRuby.cpp:
* rendering/RenderRuby.h:
* rendering/RenderRubyBase.cpp:
* rendering/RenderRubyBase.h:
* rendering/RenderRubyRun.cpp:
* rendering/RenderRubyRun.h:
* rendering/RenderRubyText.cpp:
* rendering/RenderRubyText.h:
* rendering/RenderScrollbarPart.cpp:
* rendering/RenderScrollbarPart.h:
* rendering/RenderSearchField.cpp:
* rendering/RenderSearchField.h:
* rendering/RenderSlider.cpp:
* rendering/RenderSlider.h:
* rendering/RenderTable.cpp:
* rendering/RenderTable.h:
* rendering/RenderTableCaption.cpp:
* rendering/RenderTableCaption.h:
* rendering/RenderTableCell.cpp:
* rendering/RenderTableCell.h:
* rendering/RenderTableCol.cpp:
* rendering/RenderTableCol.h:
* rendering/RenderTableRow.cpp:
* rendering/RenderTableRow.h:
* rendering/RenderTableSection.cpp:
* rendering/RenderTableSection.h:
* rendering/RenderText.cpp:
* rendering/RenderText.h:
* rendering/RenderTextControl.cpp:
* rendering/RenderTextControl.h:
* rendering/RenderTextControlMultiLine.cpp:
* rendering/RenderTextControlMultiLine.h:
* rendering/RenderTextControlSingleLine.cpp:
* rendering/RenderTextControlSingleLine.h:
* rendering/RenderTextFragment.cpp:
* rendering/RenderTextFragment.h:
* rendering/RenderVTTCue.cpp:
* rendering/RenderVTTCue.h:
* rendering/RenderVideo.cpp:
* rendering/RenderVideo.h:
* rendering/RenderView.cpp:
* rendering/RenderView.h:
* rendering/RenderWidget.cpp:
* rendering/RenderWidget.h:
* rendering/mathml/RenderMathMLBlock.cpp:
* rendering/mathml/RenderMathMLBlock.h:
* rendering/mathml/RenderMathMLFenced.cpp:
* rendering/mathml/RenderMathMLFenced.h:
* rendering/mathml/RenderMathMLFencedOperator.cpp:
* rendering/mathml/RenderMathMLFencedOperator.h:
* rendering/mathml/RenderMathMLFraction.cpp:
* rendering/mathml/RenderMathMLFraction.h:
* rendering/mathml/RenderMathMLMath.cpp:
* rendering/mathml/RenderMathMLMath.h:
* rendering/mathml/RenderMathMLMenclose.cpp:
* rendering/mathml/RenderMathMLMenclose.h:
* rendering/mathml/RenderMathMLOperator.cpp:
* rendering/mathml/RenderMathMLOperator.h:
* rendering/mathml/RenderMathMLPadded.cpp:
* rendering/mathml/RenderMathMLPadded.h:
* rendering/mathml/RenderMathMLRoot.cpp:
* rendering/mathml/RenderMathMLRoot.h:
* rendering/mathml/RenderMathMLRow.cpp:
* rendering/mathml/RenderMathMLRow.h:
* rendering/mathml/RenderMathMLScripts.cpp:
* rendering/mathml/RenderMathMLScripts.h:
* rendering/mathml/RenderMathMLSpace.cpp:
* rendering/mathml/RenderMathMLSpace.h:
* rendering/mathml/RenderMathMLToken.cpp:
* rendering/mathml/RenderMathMLToken.h:
* rendering/mathml/RenderMathMLUnderOver.cpp:
* rendering/mathml/RenderMathMLUnderOver.h:
* rendering/svg/RenderSVGBlock.cpp:
* rendering/svg/RenderSVGBlock.h:
* rendering/svg/RenderSVGContainer.cpp:
* rendering/svg/RenderSVGContainer.h:
* rendering/svg/RenderSVGEllipse.cpp:
* rendering/svg/RenderSVGEllipse.h:
* rendering/svg/RenderSVGForeignObject.cpp:
* rendering/svg/RenderSVGForeignObject.h:
* rendering/svg/RenderSVGGradientStop.cpp:
* rendering/svg/RenderSVGGradientStop.h:
* rendering/svg/RenderSVGHiddenContainer.cpp:
* rendering/svg/RenderSVGHiddenContainer.h:
* rendering/svg/RenderSVGImage.cpp:
* rendering/svg/RenderSVGImage.h:
* rendering/svg/RenderSVGInline.cpp:
* rendering/svg/RenderSVGInline.h:
* rendering/svg/RenderSVGInlineText.cpp:
* rendering/svg/RenderSVGInlineText.h:
* rendering/svg/RenderSVGModelObject.cpp:
* rendering/svg/RenderSVGModelObject.h:
* rendering/svg/RenderSVGPath.cpp:
* rendering/svg/RenderSVGPath.h:
* rendering/svg/RenderSVGRect.cpp:
* rendering/svg/RenderSVGRect.h:
* rendering/svg/RenderSVGResourceClipper.cpp:
* rendering/svg/RenderSVGResourceClipper.h:
* rendering/svg/RenderSVGResourceContainer.cpp:
* rendering/svg/RenderSVGResourceContainer.h:
* rendering/svg/RenderSVGResourceFilter.cpp:
* rendering/svg/RenderSVGResourceFilter.h:
* rendering/svg/RenderSVGResourceFilterPrimitive.cpp:
* rendering/svg/RenderSVGResourceFilterPrimitive.h:
* rendering/svg/RenderSVGResourceGradient.cpp:
* rendering/svg/RenderSVGResourceGradient.h:
* rendering/svg/RenderSVGResourceLinearGradient.cpp:
* rendering/svg/RenderSVGResourceLinearGradient.h:
* rendering/svg/RenderSVGResourceMarker.cpp:
* rendering/svg/RenderSVGResourceMarker.h:
* rendering/svg/RenderSVGResourceMasker.cpp:
* rendering/svg/RenderSVGResourceMasker.h:
* rendering/svg/RenderSVGResourcePattern.cpp:
* rendering/svg/RenderSVGResourcePattern.h:
* rendering/svg/RenderSVGResourceRadialGradient.cpp:
* rendering/svg/RenderSVGResourceRadialGradient.h:
* rendering/svg/RenderSVGRoot.cpp:
* rendering/svg/RenderSVGRoot.h:
* rendering/svg/RenderSVGShape.cpp:
* rendering/svg/RenderSVGShape.h:
* rendering/svg/RenderSVGTSpan.cpp: Added.
* rendering/svg/RenderSVGTSpan.h:
* rendering/svg/RenderSVGText.cpp:
* rendering/svg/RenderSVGText.h:
* rendering/svg/RenderSVGTextPath.cpp:
* rendering/svg/RenderSVGTextPath.h:
* rendering/svg/RenderSVGTransformableContainer.cpp:
* rendering/svg/RenderSVGTransformableContainer.h:
* rendering/svg/RenderSVGViewportContainer.cpp:
* rendering/svg/RenderSVGViewportContainer.h:
Source/WTF:
This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not
enabled, these macros will just make the object FastMalloced.
* WTF.xcodeproj/project.pbxproj:
* wtf/FastTLS.h:
* wtf/IsoMalloc.h: Added.
* wtf/IsoMallocInlines.h: Added.
Canonical link: https://commits.webkit.org/195445@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
|
|
|
#include "AllIsoHeapsInlines.h"
|
2017-10-06 16:34:41 +00:00
|
|
|
#include "AvailableMemory.h"
|
2018-04-10 23:34:42 +00:00
|
|
|
#include "BulkDecommit.h"
|
2018-04-02 21:09:45 +00:00
|
|
|
#include "Environment.h"
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
#include "Heap.h"
|
2020-01-16 22:05:00 +00:00
|
|
|
#include "IsoHeapImplInlines.h"
|
2018-04-02 21:09:45 +00:00
|
|
|
#if BOS(DARWIN)
|
|
|
|
#import <dispatch/dispatch.h>
|
|
|
|
#import <mach/host_info.h>
|
|
|
|
#import <mach/mach.h>
|
|
|
|
#import <mach/mach_error.h>
|
|
|
|
#endif
|
2018-04-23 15:23:35 +00:00
|
|
|
#include <stdio.h>
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
#include <thread>
|
|
|
|
|
2020-02-27 04:56:58 +00:00
|
|
|
#if BPLATFORM(PLAYSTATION)
|
|
|
|
#include <pthread_np.h>
|
|
|
|
#endif
|
|
|
|
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
namespace bmalloc {
|
|
|
|
|
2018-04-02 21:09:45 +00:00
|
|
|
static constexpr bool verbose = false;
|
|
|
|
|
2018-04-10 23:34:42 +00:00
|
|
|
struct PrintTime {
|
|
|
|
PrintTime(const char* str)
|
|
|
|
: string(str)
|
|
|
|
{ }
|
|
|
|
|
|
|
|
~PrintTime()
|
|
|
|
{
|
|
|
|
if (!printed)
|
|
|
|
print();
|
|
|
|
}
|
|
|
|
void print()
|
|
|
|
{
|
|
|
|
if (verbose) {
|
|
|
|
fprintf(stderr, "%s %lfms\n", string, static_cast<double>(std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now() - start).count()) / 1000);
|
|
|
|
printed = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
const char* string;
|
|
|
|
std::chrono::steady_clock::time_point start { std::chrono::steady_clock::now() };
|
|
|
|
bool printed { false };
|
|
|
|
};
|
|
|
|
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
DEFINE_STATIC_PER_PROCESS_STORAGE(Scavenger);
|
|
|
|
|
2020-01-18 00:43:00 +00:00
|
|
|
Scavenger::Scavenger(const LockHolder&)
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
{
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
BASSERT(!Environment::get()->isDebugHeapEnabled());
|
2019-02-15 08:43:07 +00:00
|
|
|
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
#if BOS(DARWIN)
|
|
|
|
auto queue = dispatch_queue_create("WebKit Malloc Memory Pressure Handler", DISPATCH_QUEUE_SERIAL);
|
|
|
|
m_pressureHandlerDispatchSource = dispatch_source_create(DISPATCH_SOURCE_TYPE_MEMORYPRESSURE, 0, DISPATCH_MEMORYPRESSURE_CRITICAL, queue);
|
|
|
|
dispatch_source_set_event_handler(m_pressureHandlerDispatchSource, ^{
|
|
|
|
scavenge();
|
|
|
|
});
|
|
|
|
dispatch_resume(m_pressureHandlerDispatchSource);
|
|
|
|
dispatch_release(queue);
|
|
|
|
#endif
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
m_waitTime = std::chrono::milliseconds(10);
|
|
|
|
|
2017-10-06 16:34:41 +00:00
|
|
|
m_thread = std::thread(&threadEntryPoint, this);
|
|
|
|
}
|
|
|
|
|
|
|
|
void Scavenger::run()
|
|
|
|
{
|
2020-01-18 00:43:00 +00:00
|
|
|
LockHolder lock(mutex());
|
2020-01-21 20:33:40 +00:00
|
|
|
run(lock);
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
|
|
|
|
2020-01-21 20:33:40 +00:00
|
|
|
void Scavenger::run(const LockHolder&)
|
2017-10-06 16:34:41 +00:00
|
|
|
{
|
|
|
|
m_state = State::Run;
|
|
|
|
m_condition.notify_all();
|
|
|
|
}
|
|
|
|
|
|
|
|
void Scavenger::runSoon()
|
|
|
|
{
|
2020-01-18 00:43:00 +00:00
|
|
|
LockHolder lock(mutex());
|
2020-01-21 20:33:40 +00:00
|
|
|
runSoon(lock);
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
|
|
|
|
2020-01-21 20:33:40 +00:00
|
|
|
void Scavenger::runSoon(const LockHolder&)
|
2017-10-06 16:34:41 +00:00
|
|
|
{
|
|
|
|
if (willRunSoon())
|
|
|
|
return;
|
|
|
|
m_state = State::RunSoon;
|
|
|
|
m_condition.notify_all();
|
|
|
|
}
|
|
|
|
|
|
|
|
void Scavenger::scheduleIfUnderMemoryPressure(size_t bytes)
|
|
|
|
{
|
2020-01-18 00:43:00 +00:00
|
|
|
LockHolder lock(mutex());
|
2020-01-21 20:33:40 +00:00
|
|
|
scheduleIfUnderMemoryPressure(lock, bytes);
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
|
|
|
|
2020-01-21 20:33:40 +00:00
|
|
|
void Scavenger::scheduleIfUnderMemoryPressure(const LockHolder& lock, size_t bytes)
|
2017-10-06 16:34:41 +00:00
|
|
|
{
|
|
|
|
m_scavengerBytes += bytes;
|
|
|
|
if (m_scavengerBytes < scavengerBytesPerMemoryPressureCheck)
|
|
|
|
return;
|
|
|
|
|
|
|
|
m_scavengerBytes = 0;
|
|
|
|
|
|
|
|
if (willRun())
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (!isUnderMemoryPressure())
|
|
|
|
return;
|
|
|
|
|
2020-01-21 20:33:40 +00:00
|
|
|
run(lock);
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void Scavenger::schedule(size_t bytes)
|
|
|
|
{
|
2020-01-18 00:43:00 +00:00
|
|
|
LockHolder lock(mutex());
|
2020-01-21 20:33:40 +00:00
|
|
|
scheduleIfUnderMemoryPressure(lock, bytes);
|
2017-10-06 16:34:41 +00:00
|
|
|
|
|
|
|
if (willRunSoon())
|
|
|
|
return;
|
|
|
|
|
2020-01-21 20:33:40 +00:00
|
|
|
runSoon(lock);
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
}
|
|
|
|
|
2018-04-02 21:09:45 +00:00
|
|
|
inline void dumpStats()
|
|
|
|
{
|
|
|
|
auto dump = [] (auto* string, auto size) {
|
|
|
|
fprintf(stderr, "%s %zuMB\n", string, static_cast<size_t>(size) / 1024 / 1024);
|
|
|
|
};
|
|
|
|
|
|
|
|
#if BOS(DARWIN)
|
|
|
|
task_vm_info_data_t vmInfo;
|
|
|
|
mach_msg_type_number_t vmSize = TASK_VM_INFO_COUNT;
|
|
|
|
if (KERN_SUCCESS == task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)(&vmInfo), &vmSize)) {
|
2018-06-27 18:22:06 +00:00
|
|
|
dump("phys_footprint", vmInfo.phys_footprint);
|
2018-04-02 21:09:45 +00:00
|
|
|
dump("internal+compressed", vmInfo.internal + vmInfo.compressed);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
dump("bmalloc-freeable", Scavenger::get()->freeableMemory());
|
|
|
|
dump("bmalloc-footprint", Scavenger::get()->footprint());
|
2018-04-02 21:09:45 +00:00
|
|
|
}
|
|
|
|
|
2018-04-10 23:34:42 +00:00
|
|
|
std::chrono::milliseconds Scavenger::timeSinceLastFullScavenge()
|
|
|
|
{
|
2020-01-18 00:43:00 +00:00
|
|
|
UniqueLockHolder lock(mutex());
|
2018-04-10 23:34:42 +00:00
|
|
|
return std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - m_lastFullScavengeTime);
|
|
|
|
}
|
|
|
|
|
2018-05-29 18:56:05 +00:00
|
|
|
void Scavenger::enableMiniMode()
|
|
|
|
{
|
|
|
|
m_isInMiniMode = true; // We just store to this racily. The scavenger thread will eventually pick up the right value.
|
|
|
|
if (m_state == State::RunSoon)
|
|
|
|
run();
|
|
|
|
}
|
|
|
|
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
void Scavenger::scavenge()
|
|
|
|
{
|
2020-04-07 18:41:13 +00:00
|
|
|
if (!m_isEnabled)
|
|
|
|
return;
|
|
|
|
|
2020-01-18 00:43:00 +00:00
|
|
|
UniqueLockHolder lock(m_scavengingMutex);
|
2018-04-10 23:34:42 +00:00
|
|
|
|
2018-04-02 21:09:45 +00:00
|
|
|
if (verbose) {
|
|
|
|
fprintf(stderr, "--------------------------------\n");
|
|
|
|
fprintf(stderr, "--before scavenging--\n");
|
|
|
|
dumpStats();
|
|
|
|
}
|
|
|
|
|
bmalloc should support strictly type-segregated isolated heaps
https://bugs.webkit.org/show_bug.cgi?id=178108
Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa.
Source/bmalloc:
This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by
type and created in static storage. When unused, it takes only a few words. When you do use
it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs
in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will
point to the same object even though they should not have.
IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide
(the JSC GC):
Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls
pages. Pages are collected into directories. Directories track pages using bitvectors, so
that it's easy to quickly find a completely free page or one that has at least one free
object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and
line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit
discipline. However, the real reason why I wrote it that was is that this is what I'm more
familiar with. This is a part of the design I want to revisit (bug 179278).
Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data
structures can be locked with a coarse-grained lock, since the deallocator only grabs it when
flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop
FreeList.
This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's
RenderObject.
Note that despite the use of GC concepts, it's not a goal to make this code directly sharable
with GC. The GC will probably have to do isolated heaps its own way (likely a special
Subspace or something like that).
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Algorithm.h:
(bmalloc::findBitInWord):
* bmalloc/AllIsoHeaps.cpp: Added.
(bmalloc::AllIsoHeaps::AllIsoHeaps):
(bmalloc::AllIsoHeaps::add):
(bmalloc::AllIsoHeaps::head):
* bmalloc/AllIsoHeaps.h: Added.
* bmalloc/AllIsoHeapsInlines.h: Added.
(bmalloc::AllIsoHeaps::forEach):
* bmalloc/BMalloced.h: Added.
* bmalloc/Bits.h: Added.
(bmalloc::bitsArrayLength):
(bmalloc::BitsWordView::BitsWordView):
(bmalloc::BitsWordView::numBits const):
(bmalloc::BitsWordView::word const):
(bmalloc::BitsWordOwner::BitsWordOwner):
(bmalloc::BitsWordOwner::view const):
(bmalloc::BitsWordOwner::operator=):
(bmalloc::BitsWordOwner::setAll):
(bmalloc::BitsWordOwner::clearAll):
(bmalloc::BitsWordOwner::set):
(bmalloc::BitsWordOwner::numBits const):
(bmalloc::BitsWordOwner::arrayLength const):
(bmalloc::BitsWordOwner::word const):
(bmalloc::BitsWordOwner::word):
(bmalloc::BitsWordOwner::words const):
(bmalloc::BitsWordOwner::words):
(bmalloc::BitsAndWords::BitsAndWords):
(bmalloc::BitsAndWords::view const):
(bmalloc::BitsAndWords::numBits const):
(bmalloc::BitsAndWords::word const):
(bmalloc::BitsOrWords::BitsOrWords):
(bmalloc::BitsOrWords::view const):
(bmalloc::BitsOrWords::numBits const):
(bmalloc::BitsOrWords::word const):
(bmalloc::BitsNotWords::BitsNotWords):
(bmalloc::BitsNotWords::view const):
(bmalloc::BitsNotWords::numBits const):
(bmalloc::BitsNotWords::word const):
(bmalloc::BitsImpl::BitsImpl):
(bmalloc::BitsImpl::numBits const):
(bmalloc::BitsImpl::size const):
(bmalloc::BitsImpl::arrayLength const):
(bmalloc::BitsImpl::operator== const):
(bmalloc::BitsImpl::operator!= const):
(bmalloc::BitsImpl::at const):
(bmalloc::BitsImpl::operator[] const):
(bmalloc::BitsImpl::isEmpty const):
(bmalloc::BitsImpl::operator& const):
(bmalloc::BitsImpl::operator| const):
(bmalloc::BitsImpl::operator~ const):
(bmalloc::BitsImpl::forEachSetBit const):
(bmalloc::BitsImpl::forEachClearBit const):
(bmalloc::BitsImpl::forEachBit const):
(bmalloc::BitsImpl::findBit const):
(bmalloc::BitsImpl::findSetBit const):
(bmalloc::BitsImpl::findClearBit const):
(bmalloc::BitsImpl::wordView const):
(bmalloc::BitsImpl::atImpl const):
(bmalloc::Bits::Bits):
(bmalloc::Bits::operator=):
(bmalloc::Bits::resize):
(bmalloc::Bits::setAll):
(bmalloc::Bits::clearAll):
(bmalloc::Bits::setAndCheck):
(bmalloc::Bits::operator|=):
(bmalloc::Bits::operator&=):
(bmalloc::Bits::at const):
(bmalloc::Bits::operator[] const):
(bmalloc::Bits::BitReference::BitReference):
(bmalloc::Bits::BitReference::operator bool const):
(bmalloc::Bits::BitReference::operator=):
(bmalloc::Bits::at):
(bmalloc::Bits::operator[]):
* bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp.
(bmalloc::cryptoRandom):
* bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h.
* bmalloc/DeferredDecommit.h: Added.
* bmalloc/DeferredDecommitInlines.h: Added.
(bmalloc::DeferredDecommit::DeferredDecommit):
* bmalloc/DeferredTrigger.h: Added.
(bmalloc::DeferredTrigger::DeferredTrigger):
* bmalloc/DeferredTriggerInlines.h: Added.
(bmalloc::DeferredTrigger<trigger>::didBecome):
(bmalloc::DeferredTrigger<trigger>::handleDeferral):
* bmalloc/EligibilityResult.h: Added.
(bmalloc::EligibilityResult::EligibilityResult):
* bmalloc/EligibilityResultInlines.h: Added.
(bmalloc::EligibilityResult<Config>::EligibilityResult):
* bmalloc/FixedVector.h:
* bmalloc/FreeList.cpp: Added.
(bmalloc::FreeList::FreeList):
(bmalloc::FreeList::~FreeList):
(bmalloc::FreeList::clear):
(bmalloc::FreeList::initializeList):
(bmalloc::FreeList::initializeBump):
(bmalloc::FreeList::contains const):
* bmalloc/FreeList.h: Added.
(bmalloc::FreeCell::scramble):
(bmalloc::FreeCell::descramble):
(bmalloc::FreeCell::setNext):
(bmalloc::FreeCell::next const):
(bmalloc::FreeList::allocationWillFail const):
(bmalloc::FreeList::allocationWillSucceed const):
(bmalloc::FreeList::originalSize const):
(bmalloc::FreeList::head const):
* bmalloc/FreeListInlines.h: Added.
(bmalloc::FreeList::allocate):
(bmalloc::FreeList::forEach const):
* bmalloc/IsoAllocator.h: Added.
* bmalloc/IsoAllocatorInlines.h: Added.
(bmalloc::IsoAllocator<Config>::IsoAllocator):
(bmalloc::IsoAllocator<Config>::~IsoAllocator):
(bmalloc::IsoAllocator<Config>::allocate):
(bmalloc::IsoAllocator<Config>::allocateSlow):
(bmalloc::IsoAllocator<Config>::scavenge):
* bmalloc/IsoConfig.h: Added.
* bmalloc/IsoDeallocator.h: Added.
* bmalloc/IsoDeallocatorInlines.h: Added.
(bmalloc::IsoDeallocator<Config>::IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::~IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::deallocate):
(bmalloc::IsoDeallocator<Config>::scavenge):
* bmalloc/IsoDirectory.h: Added.
(bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBase::heap):
* bmalloc/IsoDirectoryInlines.h: Added.
(bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase):
(bmalloc::passedNumPages>::IsoDirectory):
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
(bmalloc::passedNumPages>::didDecommit):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::forEachCommittedPage):
* bmalloc/IsoDirectoryPage.h: Added.
(bmalloc::IsoDirectoryPage::index const):
* bmalloc/IsoDirectoryPageInlines.h: Added.
(bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage):
(bmalloc::IsoDirectoryPage<Config>::pageFor):
* bmalloc/IsoHeap.h: Added.
(bmalloc::api::IsoHeap::allocatorOffset):
(bmalloc::api::IsoHeap::setAllocatorOffset):
(bmalloc::api::IsoHeap::deallocatorOffset):
(bmalloc::api::IsoHeap::setDeallocatorOffset):
* bmalloc/IsoHeapImpl.cpp: Added.
(bmalloc::IsoHeapImplBase::IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::~IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::scavengeNow):
(bmalloc::IsoHeapImplBase::finishScavenging):
* bmalloc/IsoHeapImpl.h: Added.
* bmalloc/IsoHeapImplInlines.h: Added.
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
(bmalloc::IsoHeapImpl<Config>::takeFirstEligible):
(bmalloc::IsoHeapImpl<Config>::didBecomeEligible):
(bmalloc::IsoHeapImpl<Config>::scavenge):
(bmalloc::IsoHeapImpl<Config>::allocatorOffset):
(bmalloc::IsoHeapImpl<Config>::deallocatorOffset):
(bmalloc::IsoHeapImpl<Config>::numLiveObjects):
(bmalloc::IsoHeapImpl<Config>::numCommittedPages):
(bmalloc::IsoHeapImpl<Config>::forEachDirectory):
(bmalloc::IsoHeapImpl<Config>::forEachCommittedPage):
(bmalloc::IsoHeapImpl<Config>::forEachLiveObject):
* bmalloc/IsoHeapInlines.h: Added.
(bmalloc::api::IsoHeap<Type>::allocate):
(bmalloc::api::IsoHeap<Type>::tryAllocate):
(bmalloc::api::IsoHeap<Type>::deallocate):
(bmalloc::api::IsoHeap<Type>::scavenge):
(bmalloc::api::IsoHeap<Type>::isInitialized):
(bmalloc::api::IsoHeap<Type>::impl):
* bmalloc/IsoPage.h: Added.
(bmalloc::IsoPage::index const):
(bmalloc::IsoPage::directory):
(bmalloc::IsoPage::isInUseForAllocation const):
(bmalloc::IsoPage::indexOfFirstObject):
* bmalloc/IsoPageInlines.h: Added.
(bmalloc::IsoPage<Config>::tryCreate):
(bmalloc::IsoPage<Config>::IsoPage):
(bmalloc::IsoPage<Config>::free):
(bmalloc::IsoPage<Config>::startAllocating):
(bmalloc::IsoPage<Config>::stopAllocating):
(bmalloc::IsoPage<Config>::forEachLiveObject):
* bmalloc/IsoPageTrigger.h: Added.
* bmalloc/IsoTLS.cpp: Added.
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::destructor):
(bmalloc::IsoTLS::sizeForCapacity):
(bmalloc::IsoTLS::capacityForSize):
(bmalloc::IsoTLS::size):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLS.h: Added.
* bmalloc/IsoTLSAllocatorEntry.h: Added.
* bmalloc/IsoTLSAllocatorEntryInlines.h: Added.
(bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::construct):
* bmalloc/IsoTLSDeallocatorEntry.h: Added.
* bmalloc/IsoTLSDeallocatorEntryInlines.h: Added.
(bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::construct):
* bmalloc/IsoTLSEntry.cpp: Added.
(bmalloc::IsoTLSEntry::IsoTLSEntry):
(bmalloc::IsoTLSEntry::~IsoTLSEntry):
* bmalloc/IsoTLSEntry.h: Added.
(bmalloc::IsoTLSEntry::offset const):
(bmalloc::IsoTLSEntry::alignment const):
(bmalloc::IsoTLSEntry::size const):
(bmalloc::IsoTLSEntry::extent const):
* bmalloc/IsoTLSEntryInlines.h: Added.
(bmalloc::IsoTLSEntry::walkUpToInclusive):
(bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::move):
(bmalloc::DefaultIsoTLSEntry<EntryType>::destruct):
(bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge):
* bmalloc/IsoTLSInlines.h: Added.
(bmalloc::IsoTLS::allocate):
(bmalloc::IsoTLS::deallocate):
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::allocator):
(bmalloc::IsoTLS::deallocator):
(bmalloc::IsoTLS::get):
(bmalloc::IsoTLS::set):
(bmalloc::IsoTLS::ensureHeap):
(bmalloc::IsoTLS::ensureHeapAndEntries):
* bmalloc/IsoTLSLayout.cpp: Added.
(bmalloc::IsoTLSLayout::IsoTLSLayout):
(bmalloc::IsoTLSLayout::add):
* bmalloc/IsoTLSLayout.h: Added.
(bmalloc::IsoTLSLayout::head const):
* bmalloc/PerHeapKind.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h:
* bmalloc/bmalloc.h:
(bmalloc::api::scavengeThisThread):
* test: Added.
* test/testbmalloc.cpp: Added.
(hiddenTruthBecauseNoReturnIsStupid):
(usage):
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
(testIsoSimple):
(testIsoSimpleScavengeBeforeDealloc):
(testIsoFlipFlopFragmentedPages):
(testIsoFlipFlopFragmentedPagesScavengeInMiddle):
(BisoMalloced::BisoMalloced):
(testBisoMalloced):
(BisoMallocedInline::BisoMallocedInline):
(testBisoMallocedInline):
(run):
(main):
Source/WebCore:
No new tests because no new change in behavior. Though, the bmalloc change has a unit test.
Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we
opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never
have to do any kind of switch or dynamic lookup to find the right allocator for a type.
This change is perf-neutral on MotionMark, PLT3, and membuster.
* Sources.txt:
* html/shadow/SliderThumbElement.cpp:
* html/shadow/SliderThumbElement.h:
* html/shadow/mac/ImageControlsButtonElementMac.cpp:
* html/shadow/mac/ImageControlsRootElementMac.cpp:
* rendering/RenderAttachment.cpp:
* rendering/RenderAttachment.h:
* rendering/RenderBlock.cpp:
* rendering/RenderBlock.h:
* rendering/RenderBlockFlow.cpp:
* rendering/RenderBlockFlow.h:
* rendering/RenderBox.cpp:
* rendering/RenderBox.h:
* rendering/RenderBoxModelObject.cpp:
* rendering/RenderBoxModelObject.h:
* rendering/RenderButton.cpp:
* rendering/RenderButton.h:
* rendering/RenderCombineText.cpp:
* rendering/RenderCombineText.h:
* rendering/RenderCounter.cpp:
* rendering/RenderCounter.h:
* rendering/RenderDeprecatedFlexibleBox.cpp:
* rendering/RenderDeprecatedFlexibleBox.h:
* rendering/RenderDetailsMarker.cpp:
* rendering/RenderDetailsMarker.h:
* rendering/RenderElement.cpp:
* rendering/RenderElement.h:
* rendering/RenderEmbeddedObject.cpp:
* rendering/RenderEmbeddedObject.h:
* rendering/RenderFileUploadControl.cpp:
* rendering/RenderFileUploadControl.h:
* rendering/RenderFlexibleBox.cpp:
* rendering/RenderFlexibleBox.h:
* rendering/RenderFragmentContainer.cpp:
* rendering/RenderFragmentContainer.h:
* rendering/RenderFragmentContainerSet.cpp:
* rendering/RenderFragmentContainerSet.h:
* rendering/RenderFragmentedFlow.cpp:
* rendering/RenderFragmentedFlow.h:
* rendering/RenderFrameBase.cpp:
* rendering/RenderFrameBase.h:
* rendering/RenderFrameSet.cpp:
* rendering/RenderFrameSet.h:
* rendering/RenderFullScreen.cpp:
* rendering/RenderFullScreen.h:
* rendering/RenderGrid.cpp:
* rendering/RenderGrid.h:
* rendering/RenderHTMLCanvas.cpp:
* rendering/RenderHTMLCanvas.h:
* rendering/RenderImage.cpp:
* rendering/RenderImage.h:
* rendering/RenderImageResourceStyleImage.cpp:
* rendering/RenderImageResourceStyleImage.h:
* rendering/RenderInline.cpp:
* rendering/RenderInline.h:
* rendering/RenderLayerModelObject.cpp:
* rendering/RenderLayerModelObject.h:
* rendering/RenderLineBreak.cpp:
* rendering/RenderLineBreak.h:
* rendering/RenderListBox.cpp:
* rendering/RenderListBox.h:
* rendering/RenderListItem.cpp:
* rendering/RenderListItem.h:
* rendering/RenderListMarker.cpp:
* rendering/RenderListMarker.h:
* rendering/RenderMedia.cpp:
* rendering/RenderMedia.h:
* rendering/RenderMediaControlElements.cpp:
* rendering/RenderMediaControlElements.h:
* rendering/RenderMenuList.cpp:
* rendering/RenderMenuList.h:
* rendering/RenderMeter.cpp:
* rendering/RenderMeter.h:
* rendering/RenderMultiColumnFlow.cpp:
* rendering/RenderMultiColumnFlow.h:
* rendering/RenderMultiColumnSet.cpp:
* rendering/RenderMultiColumnSet.h:
* rendering/RenderMultiColumnSpannerPlaceholder.cpp:
* rendering/RenderMultiColumnSpannerPlaceholder.h:
* rendering/RenderObject.cpp:
* rendering/RenderObject.h:
* rendering/RenderProgress.cpp:
* rendering/RenderProgress.h:
* rendering/RenderQuote.cpp:
* rendering/RenderQuote.h:
* rendering/RenderReplaced.cpp:
* rendering/RenderReplaced.h:
* rendering/RenderReplica.cpp:
* rendering/RenderReplica.h:
* rendering/RenderRuby.cpp:
* rendering/RenderRuby.h:
* rendering/RenderRubyBase.cpp:
* rendering/RenderRubyBase.h:
* rendering/RenderRubyRun.cpp:
* rendering/RenderRubyRun.h:
* rendering/RenderRubyText.cpp:
* rendering/RenderRubyText.h:
* rendering/RenderScrollbarPart.cpp:
* rendering/RenderScrollbarPart.h:
* rendering/RenderSearchField.cpp:
* rendering/RenderSearchField.h:
* rendering/RenderSlider.cpp:
* rendering/RenderSlider.h:
* rendering/RenderTable.cpp:
* rendering/RenderTable.h:
* rendering/RenderTableCaption.cpp:
* rendering/RenderTableCaption.h:
* rendering/RenderTableCell.cpp:
* rendering/RenderTableCell.h:
* rendering/RenderTableCol.cpp:
* rendering/RenderTableCol.h:
* rendering/RenderTableRow.cpp:
* rendering/RenderTableRow.h:
* rendering/RenderTableSection.cpp:
* rendering/RenderTableSection.h:
* rendering/RenderText.cpp:
* rendering/RenderText.h:
* rendering/RenderTextControl.cpp:
* rendering/RenderTextControl.h:
* rendering/RenderTextControlMultiLine.cpp:
* rendering/RenderTextControlMultiLine.h:
* rendering/RenderTextControlSingleLine.cpp:
* rendering/RenderTextControlSingleLine.h:
* rendering/RenderTextFragment.cpp:
* rendering/RenderTextFragment.h:
* rendering/RenderVTTCue.cpp:
* rendering/RenderVTTCue.h:
* rendering/RenderVideo.cpp:
* rendering/RenderVideo.h:
* rendering/RenderView.cpp:
* rendering/RenderView.h:
* rendering/RenderWidget.cpp:
* rendering/RenderWidget.h:
* rendering/mathml/RenderMathMLBlock.cpp:
* rendering/mathml/RenderMathMLBlock.h:
* rendering/mathml/RenderMathMLFenced.cpp:
* rendering/mathml/RenderMathMLFenced.h:
* rendering/mathml/RenderMathMLFencedOperator.cpp:
* rendering/mathml/RenderMathMLFencedOperator.h:
* rendering/mathml/RenderMathMLFraction.cpp:
* rendering/mathml/RenderMathMLFraction.h:
* rendering/mathml/RenderMathMLMath.cpp:
* rendering/mathml/RenderMathMLMath.h:
* rendering/mathml/RenderMathMLMenclose.cpp:
* rendering/mathml/RenderMathMLMenclose.h:
* rendering/mathml/RenderMathMLOperator.cpp:
* rendering/mathml/RenderMathMLOperator.h:
* rendering/mathml/RenderMathMLPadded.cpp:
* rendering/mathml/RenderMathMLPadded.h:
* rendering/mathml/RenderMathMLRoot.cpp:
* rendering/mathml/RenderMathMLRoot.h:
* rendering/mathml/RenderMathMLRow.cpp:
* rendering/mathml/RenderMathMLRow.h:
* rendering/mathml/RenderMathMLScripts.cpp:
* rendering/mathml/RenderMathMLScripts.h:
* rendering/mathml/RenderMathMLSpace.cpp:
* rendering/mathml/RenderMathMLSpace.h:
* rendering/mathml/RenderMathMLToken.cpp:
* rendering/mathml/RenderMathMLToken.h:
* rendering/mathml/RenderMathMLUnderOver.cpp:
* rendering/mathml/RenderMathMLUnderOver.h:
* rendering/svg/RenderSVGBlock.cpp:
* rendering/svg/RenderSVGBlock.h:
* rendering/svg/RenderSVGContainer.cpp:
* rendering/svg/RenderSVGContainer.h:
* rendering/svg/RenderSVGEllipse.cpp:
* rendering/svg/RenderSVGEllipse.h:
* rendering/svg/RenderSVGForeignObject.cpp:
* rendering/svg/RenderSVGForeignObject.h:
* rendering/svg/RenderSVGGradientStop.cpp:
* rendering/svg/RenderSVGGradientStop.h:
* rendering/svg/RenderSVGHiddenContainer.cpp:
* rendering/svg/RenderSVGHiddenContainer.h:
* rendering/svg/RenderSVGImage.cpp:
* rendering/svg/RenderSVGImage.h:
* rendering/svg/RenderSVGInline.cpp:
* rendering/svg/RenderSVGInline.h:
* rendering/svg/RenderSVGInlineText.cpp:
* rendering/svg/RenderSVGInlineText.h:
* rendering/svg/RenderSVGModelObject.cpp:
* rendering/svg/RenderSVGModelObject.h:
* rendering/svg/RenderSVGPath.cpp:
* rendering/svg/RenderSVGPath.h:
* rendering/svg/RenderSVGRect.cpp:
* rendering/svg/RenderSVGRect.h:
* rendering/svg/RenderSVGResourceClipper.cpp:
* rendering/svg/RenderSVGResourceClipper.h:
* rendering/svg/RenderSVGResourceContainer.cpp:
* rendering/svg/RenderSVGResourceContainer.h:
* rendering/svg/RenderSVGResourceFilter.cpp:
* rendering/svg/RenderSVGResourceFilter.h:
* rendering/svg/RenderSVGResourceFilterPrimitive.cpp:
* rendering/svg/RenderSVGResourceFilterPrimitive.h:
* rendering/svg/RenderSVGResourceGradient.cpp:
* rendering/svg/RenderSVGResourceGradient.h:
* rendering/svg/RenderSVGResourceLinearGradient.cpp:
* rendering/svg/RenderSVGResourceLinearGradient.h:
* rendering/svg/RenderSVGResourceMarker.cpp:
* rendering/svg/RenderSVGResourceMarker.h:
* rendering/svg/RenderSVGResourceMasker.cpp:
* rendering/svg/RenderSVGResourceMasker.h:
* rendering/svg/RenderSVGResourcePattern.cpp:
* rendering/svg/RenderSVGResourcePattern.h:
* rendering/svg/RenderSVGResourceRadialGradient.cpp:
* rendering/svg/RenderSVGResourceRadialGradient.h:
* rendering/svg/RenderSVGRoot.cpp:
* rendering/svg/RenderSVGRoot.h:
* rendering/svg/RenderSVGShape.cpp:
* rendering/svg/RenderSVGShape.h:
* rendering/svg/RenderSVGTSpan.cpp: Added.
* rendering/svg/RenderSVGTSpan.h:
* rendering/svg/RenderSVGText.cpp:
* rendering/svg/RenderSVGText.h:
* rendering/svg/RenderSVGTextPath.cpp:
* rendering/svg/RenderSVGTextPath.h:
* rendering/svg/RenderSVGTransformableContainer.cpp:
* rendering/svg/RenderSVGTransformableContainer.h:
* rendering/svg/RenderSVGViewportContainer.cpp:
* rendering/svg/RenderSVGViewportContainer.h:
Source/WTF:
This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not
enabled, these macros will just make the object FastMalloced.
* WTF.xcodeproj/project.pbxproj:
* wtf/FastTLS.h:
* wtf/IsoMalloc.h: Added.
* wtf/IsoMallocInlines.h: Added.
Canonical link: https://commits.webkit.org/195445@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
|
|
|
{
|
2018-04-10 23:34:42 +00:00
|
|
|
BulkDecommit decommitter;
|
|
|
|
|
|
|
|
{
|
|
|
|
PrintTime printTime("\nfull scavenge under lock time");
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
size_t deferredDecommits = 0;
|
2020-05-14 01:09:19 +00:00
|
|
|
UniqueLockHolder lock(Heap::mutex());
|
2018-04-10 23:34:42 +00:00
|
|
|
for (unsigned i = numHeaps; i--;) {
|
|
|
|
if (!isActiveHeapKind(static_cast<HeapKind>(i)))
|
|
|
|
continue;
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
PerProcess<PerHeapKind<Heap>>::get()->at(i).scavenge(lock, decommitter, deferredDecommits);
|
2018-04-10 23:34:42 +00:00
|
|
|
}
|
|
|
|
decommitter.processEager();
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
|
|
|
|
if (deferredDecommits)
|
|
|
|
m_state = State::RunSoon;
|
2018-04-10 23:34:42 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
{
|
|
|
|
PrintTime printTime("full scavenge lazy decommit time");
|
|
|
|
decommitter.processLazy();
|
|
|
|
}
|
|
|
|
|
|
|
|
{
|
|
|
|
PrintTime printTime("full scavenge mark all as eligible time");
|
2020-01-18 00:43:00 +00:00
|
|
|
LockHolder lock(Heap::mutex());
|
2018-04-10 23:34:42 +00:00
|
|
|
for (unsigned i = numHeaps; i--;) {
|
|
|
|
if (!isActiveHeapKind(static_cast<HeapKind>(i)))
|
|
|
|
continue;
|
|
|
|
PerProcess<PerHeapKind<Heap>>::get()->at(i).markAllLargeAsEligibile(lock);
|
|
|
|
}
|
bmalloc should support strictly type-segregated isolated heaps
https://bugs.webkit.org/show_bug.cgi?id=178108
Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa.
Source/bmalloc:
This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by
type and created in static storage. When unused, it takes only a few words. When you do use
it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs
in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will
point to the same object even though they should not have.
IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide
(the JSC GC):
Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls
pages. Pages are collected into directories. Directories track pages using bitvectors, so
that it's easy to quickly find a completely free page or one that has at least one free
object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and
line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit
discipline. However, the real reason why I wrote it that was is that this is what I'm more
familiar with. This is a part of the design I want to revisit (bug 179278).
Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data
structures can be locked with a coarse-grained lock, since the deallocator only grabs it when
flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop
FreeList.
This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's
RenderObject.
Note that despite the use of GC concepts, it's not a goal to make this code directly sharable
with GC. The GC will probably have to do isolated heaps its own way (likely a special
Subspace or something like that).
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Algorithm.h:
(bmalloc::findBitInWord):
* bmalloc/AllIsoHeaps.cpp: Added.
(bmalloc::AllIsoHeaps::AllIsoHeaps):
(bmalloc::AllIsoHeaps::add):
(bmalloc::AllIsoHeaps::head):
* bmalloc/AllIsoHeaps.h: Added.
* bmalloc/AllIsoHeapsInlines.h: Added.
(bmalloc::AllIsoHeaps::forEach):
* bmalloc/BMalloced.h: Added.
* bmalloc/Bits.h: Added.
(bmalloc::bitsArrayLength):
(bmalloc::BitsWordView::BitsWordView):
(bmalloc::BitsWordView::numBits const):
(bmalloc::BitsWordView::word const):
(bmalloc::BitsWordOwner::BitsWordOwner):
(bmalloc::BitsWordOwner::view const):
(bmalloc::BitsWordOwner::operator=):
(bmalloc::BitsWordOwner::setAll):
(bmalloc::BitsWordOwner::clearAll):
(bmalloc::BitsWordOwner::set):
(bmalloc::BitsWordOwner::numBits const):
(bmalloc::BitsWordOwner::arrayLength const):
(bmalloc::BitsWordOwner::word const):
(bmalloc::BitsWordOwner::word):
(bmalloc::BitsWordOwner::words const):
(bmalloc::BitsWordOwner::words):
(bmalloc::BitsAndWords::BitsAndWords):
(bmalloc::BitsAndWords::view const):
(bmalloc::BitsAndWords::numBits const):
(bmalloc::BitsAndWords::word const):
(bmalloc::BitsOrWords::BitsOrWords):
(bmalloc::BitsOrWords::view const):
(bmalloc::BitsOrWords::numBits const):
(bmalloc::BitsOrWords::word const):
(bmalloc::BitsNotWords::BitsNotWords):
(bmalloc::BitsNotWords::view const):
(bmalloc::BitsNotWords::numBits const):
(bmalloc::BitsNotWords::word const):
(bmalloc::BitsImpl::BitsImpl):
(bmalloc::BitsImpl::numBits const):
(bmalloc::BitsImpl::size const):
(bmalloc::BitsImpl::arrayLength const):
(bmalloc::BitsImpl::operator== const):
(bmalloc::BitsImpl::operator!= const):
(bmalloc::BitsImpl::at const):
(bmalloc::BitsImpl::operator[] const):
(bmalloc::BitsImpl::isEmpty const):
(bmalloc::BitsImpl::operator& const):
(bmalloc::BitsImpl::operator| const):
(bmalloc::BitsImpl::operator~ const):
(bmalloc::BitsImpl::forEachSetBit const):
(bmalloc::BitsImpl::forEachClearBit const):
(bmalloc::BitsImpl::forEachBit const):
(bmalloc::BitsImpl::findBit const):
(bmalloc::BitsImpl::findSetBit const):
(bmalloc::BitsImpl::findClearBit const):
(bmalloc::BitsImpl::wordView const):
(bmalloc::BitsImpl::atImpl const):
(bmalloc::Bits::Bits):
(bmalloc::Bits::operator=):
(bmalloc::Bits::resize):
(bmalloc::Bits::setAll):
(bmalloc::Bits::clearAll):
(bmalloc::Bits::setAndCheck):
(bmalloc::Bits::operator|=):
(bmalloc::Bits::operator&=):
(bmalloc::Bits::at const):
(bmalloc::Bits::operator[] const):
(bmalloc::Bits::BitReference::BitReference):
(bmalloc::Bits::BitReference::operator bool const):
(bmalloc::Bits::BitReference::operator=):
(bmalloc::Bits::at):
(bmalloc::Bits::operator[]):
* bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp.
(bmalloc::cryptoRandom):
* bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h.
* bmalloc/DeferredDecommit.h: Added.
* bmalloc/DeferredDecommitInlines.h: Added.
(bmalloc::DeferredDecommit::DeferredDecommit):
* bmalloc/DeferredTrigger.h: Added.
(bmalloc::DeferredTrigger::DeferredTrigger):
* bmalloc/DeferredTriggerInlines.h: Added.
(bmalloc::DeferredTrigger<trigger>::didBecome):
(bmalloc::DeferredTrigger<trigger>::handleDeferral):
* bmalloc/EligibilityResult.h: Added.
(bmalloc::EligibilityResult::EligibilityResult):
* bmalloc/EligibilityResultInlines.h: Added.
(bmalloc::EligibilityResult<Config>::EligibilityResult):
* bmalloc/FixedVector.h:
* bmalloc/FreeList.cpp: Added.
(bmalloc::FreeList::FreeList):
(bmalloc::FreeList::~FreeList):
(bmalloc::FreeList::clear):
(bmalloc::FreeList::initializeList):
(bmalloc::FreeList::initializeBump):
(bmalloc::FreeList::contains const):
* bmalloc/FreeList.h: Added.
(bmalloc::FreeCell::scramble):
(bmalloc::FreeCell::descramble):
(bmalloc::FreeCell::setNext):
(bmalloc::FreeCell::next const):
(bmalloc::FreeList::allocationWillFail const):
(bmalloc::FreeList::allocationWillSucceed const):
(bmalloc::FreeList::originalSize const):
(bmalloc::FreeList::head const):
* bmalloc/FreeListInlines.h: Added.
(bmalloc::FreeList::allocate):
(bmalloc::FreeList::forEach const):
* bmalloc/IsoAllocator.h: Added.
* bmalloc/IsoAllocatorInlines.h: Added.
(bmalloc::IsoAllocator<Config>::IsoAllocator):
(bmalloc::IsoAllocator<Config>::~IsoAllocator):
(bmalloc::IsoAllocator<Config>::allocate):
(bmalloc::IsoAllocator<Config>::allocateSlow):
(bmalloc::IsoAllocator<Config>::scavenge):
* bmalloc/IsoConfig.h: Added.
* bmalloc/IsoDeallocator.h: Added.
* bmalloc/IsoDeallocatorInlines.h: Added.
(bmalloc::IsoDeallocator<Config>::IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::~IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::deallocate):
(bmalloc::IsoDeallocator<Config>::scavenge):
* bmalloc/IsoDirectory.h: Added.
(bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBase::heap):
* bmalloc/IsoDirectoryInlines.h: Added.
(bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase):
(bmalloc::passedNumPages>::IsoDirectory):
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
(bmalloc::passedNumPages>::didDecommit):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::forEachCommittedPage):
* bmalloc/IsoDirectoryPage.h: Added.
(bmalloc::IsoDirectoryPage::index const):
* bmalloc/IsoDirectoryPageInlines.h: Added.
(bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage):
(bmalloc::IsoDirectoryPage<Config>::pageFor):
* bmalloc/IsoHeap.h: Added.
(bmalloc::api::IsoHeap::allocatorOffset):
(bmalloc::api::IsoHeap::setAllocatorOffset):
(bmalloc::api::IsoHeap::deallocatorOffset):
(bmalloc::api::IsoHeap::setDeallocatorOffset):
* bmalloc/IsoHeapImpl.cpp: Added.
(bmalloc::IsoHeapImplBase::IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::~IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::scavengeNow):
(bmalloc::IsoHeapImplBase::finishScavenging):
* bmalloc/IsoHeapImpl.h: Added.
* bmalloc/IsoHeapImplInlines.h: Added.
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
(bmalloc::IsoHeapImpl<Config>::takeFirstEligible):
(bmalloc::IsoHeapImpl<Config>::didBecomeEligible):
(bmalloc::IsoHeapImpl<Config>::scavenge):
(bmalloc::IsoHeapImpl<Config>::allocatorOffset):
(bmalloc::IsoHeapImpl<Config>::deallocatorOffset):
(bmalloc::IsoHeapImpl<Config>::numLiveObjects):
(bmalloc::IsoHeapImpl<Config>::numCommittedPages):
(bmalloc::IsoHeapImpl<Config>::forEachDirectory):
(bmalloc::IsoHeapImpl<Config>::forEachCommittedPage):
(bmalloc::IsoHeapImpl<Config>::forEachLiveObject):
* bmalloc/IsoHeapInlines.h: Added.
(bmalloc::api::IsoHeap<Type>::allocate):
(bmalloc::api::IsoHeap<Type>::tryAllocate):
(bmalloc::api::IsoHeap<Type>::deallocate):
(bmalloc::api::IsoHeap<Type>::scavenge):
(bmalloc::api::IsoHeap<Type>::isInitialized):
(bmalloc::api::IsoHeap<Type>::impl):
* bmalloc/IsoPage.h: Added.
(bmalloc::IsoPage::index const):
(bmalloc::IsoPage::directory):
(bmalloc::IsoPage::isInUseForAllocation const):
(bmalloc::IsoPage::indexOfFirstObject):
* bmalloc/IsoPageInlines.h: Added.
(bmalloc::IsoPage<Config>::tryCreate):
(bmalloc::IsoPage<Config>::IsoPage):
(bmalloc::IsoPage<Config>::free):
(bmalloc::IsoPage<Config>::startAllocating):
(bmalloc::IsoPage<Config>::stopAllocating):
(bmalloc::IsoPage<Config>::forEachLiveObject):
* bmalloc/IsoPageTrigger.h: Added.
* bmalloc/IsoTLS.cpp: Added.
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::destructor):
(bmalloc::IsoTLS::sizeForCapacity):
(bmalloc::IsoTLS::capacityForSize):
(bmalloc::IsoTLS::size):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLS.h: Added.
* bmalloc/IsoTLSAllocatorEntry.h: Added.
* bmalloc/IsoTLSAllocatorEntryInlines.h: Added.
(bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::construct):
* bmalloc/IsoTLSDeallocatorEntry.h: Added.
* bmalloc/IsoTLSDeallocatorEntryInlines.h: Added.
(bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::construct):
* bmalloc/IsoTLSEntry.cpp: Added.
(bmalloc::IsoTLSEntry::IsoTLSEntry):
(bmalloc::IsoTLSEntry::~IsoTLSEntry):
* bmalloc/IsoTLSEntry.h: Added.
(bmalloc::IsoTLSEntry::offset const):
(bmalloc::IsoTLSEntry::alignment const):
(bmalloc::IsoTLSEntry::size const):
(bmalloc::IsoTLSEntry::extent const):
* bmalloc/IsoTLSEntryInlines.h: Added.
(bmalloc::IsoTLSEntry::walkUpToInclusive):
(bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::move):
(bmalloc::DefaultIsoTLSEntry<EntryType>::destruct):
(bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge):
* bmalloc/IsoTLSInlines.h: Added.
(bmalloc::IsoTLS::allocate):
(bmalloc::IsoTLS::deallocate):
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::allocator):
(bmalloc::IsoTLS::deallocator):
(bmalloc::IsoTLS::get):
(bmalloc::IsoTLS::set):
(bmalloc::IsoTLS::ensureHeap):
(bmalloc::IsoTLS::ensureHeapAndEntries):
* bmalloc/IsoTLSLayout.cpp: Added.
(bmalloc::IsoTLSLayout::IsoTLSLayout):
(bmalloc::IsoTLSLayout::add):
* bmalloc/IsoTLSLayout.h: Added.
(bmalloc::IsoTLSLayout::head const):
* bmalloc/PerHeapKind.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h:
* bmalloc/bmalloc.h:
(bmalloc::api::scavengeThisThread):
* test: Added.
* test/testbmalloc.cpp: Added.
(hiddenTruthBecauseNoReturnIsStupid):
(usage):
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
(testIsoSimple):
(testIsoSimpleScavengeBeforeDealloc):
(testIsoFlipFlopFragmentedPages):
(testIsoFlipFlopFragmentedPagesScavengeInMiddle):
(BisoMalloced::BisoMalloced):
(testBisoMalloced):
(BisoMallocedInline::BisoMallocedInline):
(testBisoMallocedInline):
(run):
(main):
Source/WebCore:
No new tests because no new change in behavior. Though, the bmalloc change has a unit test.
Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we
opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never
have to do any kind of switch or dynamic lookup to find the right allocator for a type.
This change is perf-neutral on MotionMark, PLT3, and membuster.
* Sources.txt:
* html/shadow/SliderThumbElement.cpp:
* html/shadow/SliderThumbElement.h:
* html/shadow/mac/ImageControlsButtonElementMac.cpp:
* html/shadow/mac/ImageControlsRootElementMac.cpp:
* rendering/RenderAttachment.cpp:
* rendering/RenderAttachment.h:
* rendering/RenderBlock.cpp:
* rendering/RenderBlock.h:
* rendering/RenderBlockFlow.cpp:
* rendering/RenderBlockFlow.h:
* rendering/RenderBox.cpp:
* rendering/RenderBox.h:
* rendering/RenderBoxModelObject.cpp:
* rendering/RenderBoxModelObject.h:
* rendering/RenderButton.cpp:
* rendering/RenderButton.h:
* rendering/RenderCombineText.cpp:
* rendering/RenderCombineText.h:
* rendering/RenderCounter.cpp:
* rendering/RenderCounter.h:
* rendering/RenderDeprecatedFlexibleBox.cpp:
* rendering/RenderDeprecatedFlexibleBox.h:
* rendering/RenderDetailsMarker.cpp:
* rendering/RenderDetailsMarker.h:
* rendering/RenderElement.cpp:
* rendering/RenderElement.h:
* rendering/RenderEmbeddedObject.cpp:
* rendering/RenderEmbeddedObject.h:
* rendering/RenderFileUploadControl.cpp:
* rendering/RenderFileUploadControl.h:
* rendering/RenderFlexibleBox.cpp:
* rendering/RenderFlexibleBox.h:
* rendering/RenderFragmentContainer.cpp:
* rendering/RenderFragmentContainer.h:
* rendering/RenderFragmentContainerSet.cpp:
* rendering/RenderFragmentContainerSet.h:
* rendering/RenderFragmentedFlow.cpp:
* rendering/RenderFragmentedFlow.h:
* rendering/RenderFrameBase.cpp:
* rendering/RenderFrameBase.h:
* rendering/RenderFrameSet.cpp:
* rendering/RenderFrameSet.h:
* rendering/RenderFullScreen.cpp:
* rendering/RenderFullScreen.h:
* rendering/RenderGrid.cpp:
* rendering/RenderGrid.h:
* rendering/RenderHTMLCanvas.cpp:
* rendering/RenderHTMLCanvas.h:
* rendering/RenderImage.cpp:
* rendering/RenderImage.h:
* rendering/RenderImageResourceStyleImage.cpp:
* rendering/RenderImageResourceStyleImage.h:
* rendering/RenderInline.cpp:
* rendering/RenderInline.h:
* rendering/RenderLayerModelObject.cpp:
* rendering/RenderLayerModelObject.h:
* rendering/RenderLineBreak.cpp:
* rendering/RenderLineBreak.h:
* rendering/RenderListBox.cpp:
* rendering/RenderListBox.h:
* rendering/RenderListItem.cpp:
* rendering/RenderListItem.h:
* rendering/RenderListMarker.cpp:
* rendering/RenderListMarker.h:
* rendering/RenderMedia.cpp:
* rendering/RenderMedia.h:
* rendering/RenderMediaControlElements.cpp:
* rendering/RenderMediaControlElements.h:
* rendering/RenderMenuList.cpp:
* rendering/RenderMenuList.h:
* rendering/RenderMeter.cpp:
* rendering/RenderMeter.h:
* rendering/RenderMultiColumnFlow.cpp:
* rendering/RenderMultiColumnFlow.h:
* rendering/RenderMultiColumnSet.cpp:
* rendering/RenderMultiColumnSet.h:
* rendering/RenderMultiColumnSpannerPlaceholder.cpp:
* rendering/RenderMultiColumnSpannerPlaceholder.h:
* rendering/RenderObject.cpp:
* rendering/RenderObject.h:
* rendering/RenderProgress.cpp:
* rendering/RenderProgress.h:
* rendering/RenderQuote.cpp:
* rendering/RenderQuote.h:
* rendering/RenderReplaced.cpp:
* rendering/RenderReplaced.h:
* rendering/RenderReplica.cpp:
* rendering/RenderReplica.h:
* rendering/RenderRuby.cpp:
* rendering/RenderRuby.h:
* rendering/RenderRubyBase.cpp:
* rendering/RenderRubyBase.h:
* rendering/RenderRubyRun.cpp:
* rendering/RenderRubyRun.h:
* rendering/RenderRubyText.cpp:
* rendering/RenderRubyText.h:
* rendering/RenderScrollbarPart.cpp:
* rendering/RenderScrollbarPart.h:
* rendering/RenderSearchField.cpp:
* rendering/RenderSearchField.h:
* rendering/RenderSlider.cpp:
* rendering/RenderSlider.h:
* rendering/RenderTable.cpp:
* rendering/RenderTable.h:
* rendering/RenderTableCaption.cpp:
* rendering/RenderTableCaption.h:
* rendering/RenderTableCell.cpp:
* rendering/RenderTableCell.h:
* rendering/RenderTableCol.cpp:
* rendering/RenderTableCol.h:
* rendering/RenderTableRow.cpp:
* rendering/RenderTableRow.h:
* rendering/RenderTableSection.cpp:
* rendering/RenderTableSection.h:
* rendering/RenderText.cpp:
* rendering/RenderText.h:
* rendering/RenderTextControl.cpp:
* rendering/RenderTextControl.h:
* rendering/RenderTextControlMultiLine.cpp:
* rendering/RenderTextControlMultiLine.h:
* rendering/RenderTextControlSingleLine.cpp:
* rendering/RenderTextControlSingleLine.h:
* rendering/RenderTextFragment.cpp:
* rendering/RenderTextFragment.h:
* rendering/RenderVTTCue.cpp:
* rendering/RenderVTTCue.h:
* rendering/RenderVideo.cpp:
* rendering/RenderVideo.h:
* rendering/RenderView.cpp:
* rendering/RenderView.h:
* rendering/RenderWidget.cpp:
* rendering/RenderWidget.h:
* rendering/mathml/RenderMathMLBlock.cpp:
* rendering/mathml/RenderMathMLBlock.h:
* rendering/mathml/RenderMathMLFenced.cpp:
* rendering/mathml/RenderMathMLFenced.h:
* rendering/mathml/RenderMathMLFencedOperator.cpp:
* rendering/mathml/RenderMathMLFencedOperator.h:
* rendering/mathml/RenderMathMLFraction.cpp:
* rendering/mathml/RenderMathMLFraction.h:
* rendering/mathml/RenderMathMLMath.cpp:
* rendering/mathml/RenderMathMLMath.h:
* rendering/mathml/RenderMathMLMenclose.cpp:
* rendering/mathml/RenderMathMLMenclose.h:
* rendering/mathml/RenderMathMLOperator.cpp:
* rendering/mathml/RenderMathMLOperator.h:
* rendering/mathml/RenderMathMLPadded.cpp:
* rendering/mathml/RenderMathMLPadded.h:
* rendering/mathml/RenderMathMLRoot.cpp:
* rendering/mathml/RenderMathMLRoot.h:
* rendering/mathml/RenderMathMLRow.cpp:
* rendering/mathml/RenderMathMLRow.h:
* rendering/mathml/RenderMathMLScripts.cpp:
* rendering/mathml/RenderMathMLScripts.h:
* rendering/mathml/RenderMathMLSpace.cpp:
* rendering/mathml/RenderMathMLSpace.h:
* rendering/mathml/RenderMathMLToken.cpp:
* rendering/mathml/RenderMathMLToken.h:
* rendering/mathml/RenderMathMLUnderOver.cpp:
* rendering/mathml/RenderMathMLUnderOver.h:
* rendering/svg/RenderSVGBlock.cpp:
* rendering/svg/RenderSVGBlock.h:
* rendering/svg/RenderSVGContainer.cpp:
* rendering/svg/RenderSVGContainer.h:
* rendering/svg/RenderSVGEllipse.cpp:
* rendering/svg/RenderSVGEllipse.h:
* rendering/svg/RenderSVGForeignObject.cpp:
* rendering/svg/RenderSVGForeignObject.h:
* rendering/svg/RenderSVGGradientStop.cpp:
* rendering/svg/RenderSVGGradientStop.h:
* rendering/svg/RenderSVGHiddenContainer.cpp:
* rendering/svg/RenderSVGHiddenContainer.h:
* rendering/svg/RenderSVGImage.cpp:
* rendering/svg/RenderSVGImage.h:
* rendering/svg/RenderSVGInline.cpp:
* rendering/svg/RenderSVGInline.h:
* rendering/svg/RenderSVGInlineText.cpp:
* rendering/svg/RenderSVGInlineText.h:
* rendering/svg/RenderSVGModelObject.cpp:
* rendering/svg/RenderSVGModelObject.h:
* rendering/svg/RenderSVGPath.cpp:
* rendering/svg/RenderSVGPath.h:
* rendering/svg/RenderSVGRect.cpp:
* rendering/svg/RenderSVGRect.h:
* rendering/svg/RenderSVGResourceClipper.cpp:
* rendering/svg/RenderSVGResourceClipper.h:
* rendering/svg/RenderSVGResourceContainer.cpp:
* rendering/svg/RenderSVGResourceContainer.h:
* rendering/svg/RenderSVGResourceFilter.cpp:
* rendering/svg/RenderSVGResourceFilter.h:
* rendering/svg/RenderSVGResourceFilterPrimitive.cpp:
* rendering/svg/RenderSVGResourceFilterPrimitive.h:
* rendering/svg/RenderSVGResourceGradient.cpp:
* rendering/svg/RenderSVGResourceGradient.h:
* rendering/svg/RenderSVGResourceLinearGradient.cpp:
* rendering/svg/RenderSVGResourceLinearGradient.h:
* rendering/svg/RenderSVGResourceMarker.cpp:
* rendering/svg/RenderSVGResourceMarker.h:
* rendering/svg/RenderSVGResourceMasker.cpp:
* rendering/svg/RenderSVGResourceMasker.h:
* rendering/svg/RenderSVGResourcePattern.cpp:
* rendering/svg/RenderSVGResourcePattern.h:
* rendering/svg/RenderSVGResourceRadialGradient.cpp:
* rendering/svg/RenderSVGResourceRadialGradient.h:
* rendering/svg/RenderSVGRoot.cpp:
* rendering/svg/RenderSVGRoot.h:
* rendering/svg/RenderSVGShape.cpp:
* rendering/svg/RenderSVGShape.h:
* rendering/svg/RenderSVGTSpan.cpp: Added.
* rendering/svg/RenderSVGTSpan.h:
* rendering/svg/RenderSVGText.cpp:
* rendering/svg/RenderSVGText.h:
* rendering/svg/RenderSVGTextPath.cpp:
* rendering/svg/RenderSVGTextPath.h:
* rendering/svg/RenderSVGTransformableContainer.cpp:
* rendering/svg/RenderSVGTransformableContainer.h:
* rendering/svg/RenderSVGViewportContainer.cpp:
* rendering/svg/RenderSVGViewportContainer.h:
Source/WTF:
This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not
enabled, these macros will just make the object FastMalloced.
* WTF.xcodeproj/project.pbxproj:
* wtf/FastTLS.h:
* wtf/IsoMalloc.h: Added.
* wtf/IsoMallocInlines.h: Added.
Canonical link: https://commits.webkit.org/195445@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
|
|
|
}
|
2017-10-12 16:02:45 +00:00
|
|
|
}
|
2018-04-10 23:34:42 +00:00
|
|
|
|
2018-04-02 21:09:45 +00:00
|
|
|
{
|
|
|
|
RELEASE_BASSERT(!m_deferredDecommits.size());
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
AllIsoHeaps::get()->forEach(
|
2018-04-02 21:09:45 +00:00
|
|
|
[&] (IsoHeapImplBase& heap) {
|
|
|
|
heap.scavenge(m_deferredDecommits);
|
|
|
|
});
|
|
|
|
IsoHeapImplBase::finishScavenging(m_deferredDecommits);
|
|
|
|
m_deferredDecommits.shrink(0);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (verbose) {
|
|
|
|
fprintf(stderr, "--after scavenging--\n");
|
|
|
|
dumpStats();
|
|
|
|
fprintf(stderr, "--------------------------------\n");
|
|
|
|
}
|
2018-04-10 23:34:42 +00:00
|
|
|
|
|
|
|
{
|
2020-01-18 00:43:00 +00:00
|
|
|
UniqueLockHolder lock(mutex());
|
2018-04-10 23:34:42 +00:00
|
|
|
m_lastFullScavengeTime = std::chrono::steady_clock::now();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-04-02 21:09:45 +00:00
|
|
|
size_t Scavenger::freeableMemory()
|
|
|
|
{
|
|
|
|
size_t result = 0;
|
|
|
|
{
|
2020-05-14 01:09:19 +00:00
|
|
|
UniqueLockHolder lock(Heap::mutex());
|
2018-04-02 21:09:45 +00:00
|
|
|
for (unsigned i = numHeaps; i--;) {
|
|
|
|
if (!isActiveHeapKind(static_cast<HeapKind>(i)))
|
|
|
|
continue;
|
|
|
|
result += PerProcess<PerHeapKind<Heap>>::get()->at(i).freeableMemory(lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
AllIsoHeaps::get()->forEach(
|
bmalloc should support strictly type-segregated isolated heaps
https://bugs.webkit.org/show_bug.cgi?id=178108
Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa.
Source/bmalloc:
This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by
type and created in static storage. When unused, it takes only a few words. When you do use
it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs
in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will
point to the same object even though they should not have.
IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide
(the JSC GC):
Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls
pages. Pages are collected into directories. Directories track pages using bitvectors, so
that it's easy to quickly find a completely free page or one that has at least one free
object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and
line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit
discipline. However, the real reason why I wrote it that was is that this is what I'm more
familiar with. This is a part of the design I want to revisit (bug 179278).
Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data
structures can be locked with a coarse-grained lock, since the deallocator only grabs it when
flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop
FreeList.
This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's
RenderObject.
Note that despite the use of GC concepts, it's not a goal to make this code directly sharable
with GC. The GC will probably have to do isolated heaps its own way (likely a special
Subspace or something like that).
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Algorithm.h:
(bmalloc::findBitInWord):
* bmalloc/AllIsoHeaps.cpp: Added.
(bmalloc::AllIsoHeaps::AllIsoHeaps):
(bmalloc::AllIsoHeaps::add):
(bmalloc::AllIsoHeaps::head):
* bmalloc/AllIsoHeaps.h: Added.
* bmalloc/AllIsoHeapsInlines.h: Added.
(bmalloc::AllIsoHeaps::forEach):
* bmalloc/BMalloced.h: Added.
* bmalloc/Bits.h: Added.
(bmalloc::bitsArrayLength):
(bmalloc::BitsWordView::BitsWordView):
(bmalloc::BitsWordView::numBits const):
(bmalloc::BitsWordView::word const):
(bmalloc::BitsWordOwner::BitsWordOwner):
(bmalloc::BitsWordOwner::view const):
(bmalloc::BitsWordOwner::operator=):
(bmalloc::BitsWordOwner::setAll):
(bmalloc::BitsWordOwner::clearAll):
(bmalloc::BitsWordOwner::set):
(bmalloc::BitsWordOwner::numBits const):
(bmalloc::BitsWordOwner::arrayLength const):
(bmalloc::BitsWordOwner::word const):
(bmalloc::BitsWordOwner::word):
(bmalloc::BitsWordOwner::words const):
(bmalloc::BitsWordOwner::words):
(bmalloc::BitsAndWords::BitsAndWords):
(bmalloc::BitsAndWords::view const):
(bmalloc::BitsAndWords::numBits const):
(bmalloc::BitsAndWords::word const):
(bmalloc::BitsOrWords::BitsOrWords):
(bmalloc::BitsOrWords::view const):
(bmalloc::BitsOrWords::numBits const):
(bmalloc::BitsOrWords::word const):
(bmalloc::BitsNotWords::BitsNotWords):
(bmalloc::BitsNotWords::view const):
(bmalloc::BitsNotWords::numBits const):
(bmalloc::BitsNotWords::word const):
(bmalloc::BitsImpl::BitsImpl):
(bmalloc::BitsImpl::numBits const):
(bmalloc::BitsImpl::size const):
(bmalloc::BitsImpl::arrayLength const):
(bmalloc::BitsImpl::operator== const):
(bmalloc::BitsImpl::operator!= const):
(bmalloc::BitsImpl::at const):
(bmalloc::BitsImpl::operator[] const):
(bmalloc::BitsImpl::isEmpty const):
(bmalloc::BitsImpl::operator& const):
(bmalloc::BitsImpl::operator| const):
(bmalloc::BitsImpl::operator~ const):
(bmalloc::BitsImpl::forEachSetBit const):
(bmalloc::BitsImpl::forEachClearBit const):
(bmalloc::BitsImpl::forEachBit const):
(bmalloc::BitsImpl::findBit const):
(bmalloc::BitsImpl::findSetBit const):
(bmalloc::BitsImpl::findClearBit const):
(bmalloc::BitsImpl::wordView const):
(bmalloc::BitsImpl::atImpl const):
(bmalloc::Bits::Bits):
(bmalloc::Bits::operator=):
(bmalloc::Bits::resize):
(bmalloc::Bits::setAll):
(bmalloc::Bits::clearAll):
(bmalloc::Bits::setAndCheck):
(bmalloc::Bits::operator|=):
(bmalloc::Bits::operator&=):
(bmalloc::Bits::at const):
(bmalloc::Bits::operator[] const):
(bmalloc::Bits::BitReference::BitReference):
(bmalloc::Bits::BitReference::operator bool const):
(bmalloc::Bits::BitReference::operator=):
(bmalloc::Bits::at):
(bmalloc::Bits::operator[]):
* bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp.
(bmalloc::cryptoRandom):
* bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h.
* bmalloc/DeferredDecommit.h: Added.
* bmalloc/DeferredDecommitInlines.h: Added.
(bmalloc::DeferredDecommit::DeferredDecommit):
* bmalloc/DeferredTrigger.h: Added.
(bmalloc::DeferredTrigger::DeferredTrigger):
* bmalloc/DeferredTriggerInlines.h: Added.
(bmalloc::DeferredTrigger<trigger>::didBecome):
(bmalloc::DeferredTrigger<trigger>::handleDeferral):
* bmalloc/EligibilityResult.h: Added.
(bmalloc::EligibilityResult::EligibilityResult):
* bmalloc/EligibilityResultInlines.h: Added.
(bmalloc::EligibilityResult<Config>::EligibilityResult):
* bmalloc/FixedVector.h:
* bmalloc/FreeList.cpp: Added.
(bmalloc::FreeList::FreeList):
(bmalloc::FreeList::~FreeList):
(bmalloc::FreeList::clear):
(bmalloc::FreeList::initializeList):
(bmalloc::FreeList::initializeBump):
(bmalloc::FreeList::contains const):
* bmalloc/FreeList.h: Added.
(bmalloc::FreeCell::scramble):
(bmalloc::FreeCell::descramble):
(bmalloc::FreeCell::setNext):
(bmalloc::FreeCell::next const):
(bmalloc::FreeList::allocationWillFail const):
(bmalloc::FreeList::allocationWillSucceed const):
(bmalloc::FreeList::originalSize const):
(bmalloc::FreeList::head const):
* bmalloc/FreeListInlines.h: Added.
(bmalloc::FreeList::allocate):
(bmalloc::FreeList::forEach const):
* bmalloc/IsoAllocator.h: Added.
* bmalloc/IsoAllocatorInlines.h: Added.
(bmalloc::IsoAllocator<Config>::IsoAllocator):
(bmalloc::IsoAllocator<Config>::~IsoAllocator):
(bmalloc::IsoAllocator<Config>::allocate):
(bmalloc::IsoAllocator<Config>::allocateSlow):
(bmalloc::IsoAllocator<Config>::scavenge):
* bmalloc/IsoConfig.h: Added.
* bmalloc/IsoDeallocator.h: Added.
* bmalloc/IsoDeallocatorInlines.h: Added.
(bmalloc::IsoDeallocator<Config>::IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::~IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::deallocate):
(bmalloc::IsoDeallocator<Config>::scavenge):
* bmalloc/IsoDirectory.h: Added.
(bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBase::heap):
* bmalloc/IsoDirectoryInlines.h: Added.
(bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase):
(bmalloc::passedNumPages>::IsoDirectory):
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
(bmalloc::passedNumPages>::didDecommit):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::forEachCommittedPage):
* bmalloc/IsoDirectoryPage.h: Added.
(bmalloc::IsoDirectoryPage::index const):
* bmalloc/IsoDirectoryPageInlines.h: Added.
(bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage):
(bmalloc::IsoDirectoryPage<Config>::pageFor):
* bmalloc/IsoHeap.h: Added.
(bmalloc::api::IsoHeap::allocatorOffset):
(bmalloc::api::IsoHeap::setAllocatorOffset):
(bmalloc::api::IsoHeap::deallocatorOffset):
(bmalloc::api::IsoHeap::setDeallocatorOffset):
* bmalloc/IsoHeapImpl.cpp: Added.
(bmalloc::IsoHeapImplBase::IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::~IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::scavengeNow):
(bmalloc::IsoHeapImplBase::finishScavenging):
* bmalloc/IsoHeapImpl.h: Added.
* bmalloc/IsoHeapImplInlines.h: Added.
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
(bmalloc::IsoHeapImpl<Config>::takeFirstEligible):
(bmalloc::IsoHeapImpl<Config>::didBecomeEligible):
(bmalloc::IsoHeapImpl<Config>::scavenge):
(bmalloc::IsoHeapImpl<Config>::allocatorOffset):
(bmalloc::IsoHeapImpl<Config>::deallocatorOffset):
(bmalloc::IsoHeapImpl<Config>::numLiveObjects):
(bmalloc::IsoHeapImpl<Config>::numCommittedPages):
(bmalloc::IsoHeapImpl<Config>::forEachDirectory):
(bmalloc::IsoHeapImpl<Config>::forEachCommittedPage):
(bmalloc::IsoHeapImpl<Config>::forEachLiveObject):
* bmalloc/IsoHeapInlines.h: Added.
(bmalloc::api::IsoHeap<Type>::allocate):
(bmalloc::api::IsoHeap<Type>::tryAllocate):
(bmalloc::api::IsoHeap<Type>::deallocate):
(bmalloc::api::IsoHeap<Type>::scavenge):
(bmalloc::api::IsoHeap<Type>::isInitialized):
(bmalloc::api::IsoHeap<Type>::impl):
* bmalloc/IsoPage.h: Added.
(bmalloc::IsoPage::index const):
(bmalloc::IsoPage::directory):
(bmalloc::IsoPage::isInUseForAllocation const):
(bmalloc::IsoPage::indexOfFirstObject):
* bmalloc/IsoPageInlines.h: Added.
(bmalloc::IsoPage<Config>::tryCreate):
(bmalloc::IsoPage<Config>::IsoPage):
(bmalloc::IsoPage<Config>::free):
(bmalloc::IsoPage<Config>::startAllocating):
(bmalloc::IsoPage<Config>::stopAllocating):
(bmalloc::IsoPage<Config>::forEachLiveObject):
* bmalloc/IsoPageTrigger.h: Added.
* bmalloc/IsoTLS.cpp: Added.
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::destructor):
(bmalloc::IsoTLS::sizeForCapacity):
(bmalloc::IsoTLS::capacityForSize):
(bmalloc::IsoTLS::size):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLS.h: Added.
* bmalloc/IsoTLSAllocatorEntry.h: Added.
* bmalloc/IsoTLSAllocatorEntryInlines.h: Added.
(bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::construct):
* bmalloc/IsoTLSDeallocatorEntry.h: Added.
* bmalloc/IsoTLSDeallocatorEntryInlines.h: Added.
(bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::construct):
* bmalloc/IsoTLSEntry.cpp: Added.
(bmalloc::IsoTLSEntry::IsoTLSEntry):
(bmalloc::IsoTLSEntry::~IsoTLSEntry):
* bmalloc/IsoTLSEntry.h: Added.
(bmalloc::IsoTLSEntry::offset const):
(bmalloc::IsoTLSEntry::alignment const):
(bmalloc::IsoTLSEntry::size const):
(bmalloc::IsoTLSEntry::extent const):
* bmalloc/IsoTLSEntryInlines.h: Added.
(bmalloc::IsoTLSEntry::walkUpToInclusive):
(bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::move):
(bmalloc::DefaultIsoTLSEntry<EntryType>::destruct):
(bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge):
* bmalloc/IsoTLSInlines.h: Added.
(bmalloc::IsoTLS::allocate):
(bmalloc::IsoTLS::deallocate):
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::allocator):
(bmalloc::IsoTLS::deallocator):
(bmalloc::IsoTLS::get):
(bmalloc::IsoTLS::set):
(bmalloc::IsoTLS::ensureHeap):
(bmalloc::IsoTLS::ensureHeapAndEntries):
* bmalloc/IsoTLSLayout.cpp: Added.
(bmalloc::IsoTLSLayout::IsoTLSLayout):
(bmalloc::IsoTLSLayout::add):
* bmalloc/IsoTLSLayout.h: Added.
(bmalloc::IsoTLSLayout::head const):
* bmalloc/PerHeapKind.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h:
* bmalloc/bmalloc.h:
(bmalloc::api::scavengeThisThread):
* test: Added.
* test/testbmalloc.cpp: Added.
(hiddenTruthBecauseNoReturnIsStupid):
(usage):
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
(testIsoSimple):
(testIsoSimpleScavengeBeforeDealloc):
(testIsoFlipFlopFragmentedPages):
(testIsoFlipFlopFragmentedPagesScavengeInMiddle):
(BisoMalloced::BisoMalloced):
(testBisoMalloced):
(BisoMallocedInline::BisoMallocedInline):
(testBisoMallocedInline):
(run):
(main):
Source/WebCore:
No new tests because no new change in behavior. Though, the bmalloc change has a unit test.
Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we
opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never
have to do any kind of switch or dynamic lookup to find the right allocator for a type.
This change is perf-neutral on MotionMark, PLT3, and membuster.
* Sources.txt:
* html/shadow/SliderThumbElement.cpp:
* html/shadow/SliderThumbElement.h:
* html/shadow/mac/ImageControlsButtonElementMac.cpp:
* html/shadow/mac/ImageControlsRootElementMac.cpp:
* rendering/RenderAttachment.cpp:
* rendering/RenderAttachment.h:
* rendering/RenderBlock.cpp:
* rendering/RenderBlock.h:
* rendering/RenderBlockFlow.cpp:
* rendering/RenderBlockFlow.h:
* rendering/RenderBox.cpp:
* rendering/RenderBox.h:
* rendering/RenderBoxModelObject.cpp:
* rendering/RenderBoxModelObject.h:
* rendering/RenderButton.cpp:
* rendering/RenderButton.h:
* rendering/RenderCombineText.cpp:
* rendering/RenderCombineText.h:
* rendering/RenderCounter.cpp:
* rendering/RenderCounter.h:
* rendering/RenderDeprecatedFlexibleBox.cpp:
* rendering/RenderDeprecatedFlexibleBox.h:
* rendering/RenderDetailsMarker.cpp:
* rendering/RenderDetailsMarker.h:
* rendering/RenderElement.cpp:
* rendering/RenderElement.h:
* rendering/RenderEmbeddedObject.cpp:
* rendering/RenderEmbeddedObject.h:
* rendering/RenderFileUploadControl.cpp:
* rendering/RenderFileUploadControl.h:
* rendering/RenderFlexibleBox.cpp:
* rendering/RenderFlexibleBox.h:
* rendering/RenderFragmentContainer.cpp:
* rendering/RenderFragmentContainer.h:
* rendering/RenderFragmentContainerSet.cpp:
* rendering/RenderFragmentContainerSet.h:
* rendering/RenderFragmentedFlow.cpp:
* rendering/RenderFragmentedFlow.h:
* rendering/RenderFrameBase.cpp:
* rendering/RenderFrameBase.h:
* rendering/RenderFrameSet.cpp:
* rendering/RenderFrameSet.h:
* rendering/RenderFullScreen.cpp:
* rendering/RenderFullScreen.h:
* rendering/RenderGrid.cpp:
* rendering/RenderGrid.h:
* rendering/RenderHTMLCanvas.cpp:
* rendering/RenderHTMLCanvas.h:
* rendering/RenderImage.cpp:
* rendering/RenderImage.h:
* rendering/RenderImageResourceStyleImage.cpp:
* rendering/RenderImageResourceStyleImage.h:
* rendering/RenderInline.cpp:
* rendering/RenderInline.h:
* rendering/RenderLayerModelObject.cpp:
* rendering/RenderLayerModelObject.h:
* rendering/RenderLineBreak.cpp:
* rendering/RenderLineBreak.h:
* rendering/RenderListBox.cpp:
* rendering/RenderListBox.h:
* rendering/RenderListItem.cpp:
* rendering/RenderListItem.h:
* rendering/RenderListMarker.cpp:
* rendering/RenderListMarker.h:
* rendering/RenderMedia.cpp:
* rendering/RenderMedia.h:
* rendering/RenderMediaControlElements.cpp:
* rendering/RenderMediaControlElements.h:
* rendering/RenderMenuList.cpp:
* rendering/RenderMenuList.h:
* rendering/RenderMeter.cpp:
* rendering/RenderMeter.h:
* rendering/RenderMultiColumnFlow.cpp:
* rendering/RenderMultiColumnFlow.h:
* rendering/RenderMultiColumnSet.cpp:
* rendering/RenderMultiColumnSet.h:
* rendering/RenderMultiColumnSpannerPlaceholder.cpp:
* rendering/RenderMultiColumnSpannerPlaceholder.h:
* rendering/RenderObject.cpp:
* rendering/RenderObject.h:
* rendering/RenderProgress.cpp:
* rendering/RenderProgress.h:
* rendering/RenderQuote.cpp:
* rendering/RenderQuote.h:
* rendering/RenderReplaced.cpp:
* rendering/RenderReplaced.h:
* rendering/RenderReplica.cpp:
* rendering/RenderReplica.h:
* rendering/RenderRuby.cpp:
* rendering/RenderRuby.h:
* rendering/RenderRubyBase.cpp:
* rendering/RenderRubyBase.h:
* rendering/RenderRubyRun.cpp:
* rendering/RenderRubyRun.h:
* rendering/RenderRubyText.cpp:
* rendering/RenderRubyText.h:
* rendering/RenderScrollbarPart.cpp:
* rendering/RenderScrollbarPart.h:
* rendering/RenderSearchField.cpp:
* rendering/RenderSearchField.h:
* rendering/RenderSlider.cpp:
* rendering/RenderSlider.h:
* rendering/RenderTable.cpp:
* rendering/RenderTable.h:
* rendering/RenderTableCaption.cpp:
* rendering/RenderTableCaption.h:
* rendering/RenderTableCell.cpp:
* rendering/RenderTableCell.h:
* rendering/RenderTableCol.cpp:
* rendering/RenderTableCol.h:
* rendering/RenderTableRow.cpp:
* rendering/RenderTableRow.h:
* rendering/RenderTableSection.cpp:
* rendering/RenderTableSection.h:
* rendering/RenderText.cpp:
* rendering/RenderText.h:
* rendering/RenderTextControl.cpp:
* rendering/RenderTextControl.h:
* rendering/RenderTextControlMultiLine.cpp:
* rendering/RenderTextControlMultiLine.h:
* rendering/RenderTextControlSingleLine.cpp:
* rendering/RenderTextControlSingleLine.h:
* rendering/RenderTextFragment.cpp:
* rendering/RenderTextFragment.h:
* rendering/RenderVTTCue.cpp:
* rendering/RenderVTTCue.h:
* rendering/RenderVideo.cpp:
* rendering/RenderVideo.h:
* rendering/RenderView.cpp:
* rendering/RenderView.h:
* rendering/RenderWidget.cpp:
* rendering/RenderWidget.h:
* rendering/mathml/RenderMathMLBlock.cpp:
* rendering/mathml/RenderMathMLBlock.h:
* rendering/mathml/RenderMathMLFenced.cpp:
* rendering/mathml/RenderMathMLFenced.h:
* rendering/mathml/RenderMathMLFencedOperator.cpp:
* rendering/mathml/RenderMathMLFencedOperator.h:
* rendering/mathml/RenderMathMLFraction.cpp:
* rendering/mathml/RenderMathMLFraction.h:
* rendering/mathml/RenderMathMLMath.cpp:
* rendering/mathml/RenderMathMLMath.h:
* rendering/mathml/RenderMathMLMenclose.cpp:
* rendering/mathml/RenderMathMLMenclose.h:
* rendering/mathml/RenderMathMLOperator.cpp:
* rendering/mathml/RenderMathMLOperator.h:
* rendering/mathml/RenderMathMLPadded.cpp:
* rendering/mathml/RenderMathMLPadded.h:
* rendering/mathml/RenderMathMLRoot.cpp:
* rendering/mathml/RenderMathMLRoot.h:
* rendering/mathml/RenderMathMLRow.cpp:
* rendering/mathml/RenderMathMLRow.h:
* rendering/mathml/RenderMathMLScripts.cpp:
* rendering/mathml/RenderMathMLScripts.h:
* rendering/mathml/RenderMathMLSpace.cpp:
* rendering/mathml/RenderMathMLSpace.h:
* rendering/mathml/RenderMathMLToken.cpp:
* rendering/mathml/RenderMathMLToken.h:
* rendering/mathml/RenderMathMLUnderOver.cpp:
* rendering/mathml/RenderMathMLUnderOver.h:
* rendering/svg/RenderSVGBlock.cpp:
* rendering/svg/RenderSVGBlock.h:
* rendering/svg/RenderSVGContainer.cpp:
* rendering/svg/RenderSVGContainer.h:
* rendering/svg/RenderSVGEllipse.cpp:
* rendering/svg/RenderSVGEllipse.h:
* rendering/svg/RenderSVGForeignObject.cpp:
* rendering/svg/RenderSVGForeignObject.h:
* rendering/svg/RenderSVGGradientStop.cpp:
* rendering/svg/RenderSVGGradientStop.h:
* rendering/svg/RenderSVGHiddenContainer.cpp:
* rendering/svg/RenderSVGHiddenContainer.h:
* rendering/svg/RenderSVGImage.cpp:
* rendering/svg/RenderSVGImage.h:
* rendering/svg/RenderSVGInline.cpp:
* rendering/svg/RenderSVGInline.h:
* rendering/svg/RenderSVGInlineText.cpp:
* rendering/svg/RenderSVGInlineText.h:
* rendering/svg/RenderSVGModelObject.cpp:
* rendering/svg/RenderSVGModelObject.h:
* rendering/svg/RenderSVGPath.cpp:
* rendering/svg/RenderSVGPath.h:
* rendering/svg/RenderSVGRect.cpp:
* rendering/svg/RenderSVGRect.h:
* rendering/svg/RenderSVGResourceClipper.cpp:
* rendering/svg/RenderSVGResourceClipper.h:
* rendering/svg/RenderSVGResourceContainer.cpp:
* rendering/svg/RenderSVGResourceContainer.h:
* rendering/svg/RenderSVGResourceFilter.cpp:
* rendering/svg/RenderSVGResourceFilter.h:
* rendering/svg/RenderSVGResourceFilterPrimitive.cpp:
* rendering/svg/RenderSVGResourceFilterPrimitive.h:
* rendering/svg/RenderSVGResourceGradient.cpp:
* rendering/svg/RenderSVGResourceGradient.h:
* rendering/svg/RenderSVGResourceLinearGradient.cpp:
* rendering/svg/RenderSVGResourceLinearGradient.h:
* rendering/svg/RenderSVGResourceMarker.cpp:
* rendering/svg/RenderSVGResourceMarker.h:
* rendering/svg/RenderSVGResourceMasker.cpp:
* rendering/svg/RenderSVGResourceMasker.h:
* rendering/svg/RenderSVGResourcePattern.cpp:
* rendering/svg/RenderSVGResourcePattern.h:
* rendering/svg/RenderSVGResourceRadialGradient.cpp:
* rendering/svg/RenderSVGResourceRadialGradient.h:
* rendering/svg/RenderSVGRoot.cpp:
* rendering/svg/RenderSVGRoot.h:
* rendering/svg/RenderSVGShape.cpp:
* rendering/svg/RenderSVGShape.h:
* rendering/svg/RenderSVGTSpan.cpp: Added.
* rendering/svg/RenderSVGTSpan.h:
* rendering/svg/RenderSVGText.cpp:
* rendering/svg/RenderSVGText.h:
* rendering/svg/RenderSVGTextPath.cpp:
* rendering/svg/RenderSVGTextPath.h:
* rendering/svg/RenderSVGTransformableContainer.cpp:
* rendering/svg/RenderSVGTransformableContainer.h:
* rendering/svg/RenderSVGViewportContainer.cpp:
* rendering/svg/RenderSVGViewportContainer.h:
Source/WTF:
This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not
enabled, these macros will just make the object FastMalloced.
* WTF.xcodeproj/project.pbxproj:
* wtf/FastTLS.h:
* wtf/IsoMalloc.h: Added.
* wtf/IsoMallocInlines.h: Added.
Canonical link: https://commits.webkit.org/195445@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
|
|
|
[&] (IsoHeapImplBase& heap) {
|
2018-04-02 21:09:45 +00:00
|
|
|
result += heap.freeableMemory();
|
bmalloc should support strictly type-segregated isolated heaps
https://bugs.webkit.org/show_bug.cgi?id=178108
Reviewed by Saam Barati, Simon Fraser, and Ryosuke Niwa.
Source/bmalloc:
This introduces a new allocation API in bmalloc called IsoHeap. An IsoHeap is templatized by
type and created in static storage. When unused, it takes only a few words. When you do use
it, each IsoHeap gets a bag of virtual pages unique to it. This prevents use-after-free bugs
in one IsoHeap from affecting any other memory. At worst, two pointers of the same type will
point to the same object even though they should not have.
IsoHeaps allocate using a first-fit discipline that combines ideas from bmalloc and Riptide
(the JSC GC):
Like Riptide, it uses a bump'n'pop allocator. What Riptide calls blocks, IsoHeaps calls
pages. Pages are collected into directories. Directories track pages using bitvectors, so
that it's easy to quickly find a completely free page or one that has at least one free
object. I think that the bump'n'pop allocator is as fast as the bmalloc Immix-style (page and
line) allocator, but is better at allocating in holes. It's guaranteed to follow a first-fit
discipline. However, the real reason why I wrote it that was is that this is what I'm more
familiar with. This is a part of the design I want to revisit (bug 179278).
Like bmalloc, it uses a deallocation log. This means that the internal IsoHeap data
structures can be locked with a coarse-grained lock, since the deallocator only grabs it when
flushing the log. Similarly, the allocator only grabs it when refilling the bump'n'pop
FreeList.
This adds a unit test for IsoHeaps. In this change, IsoHeaps are adopted only by WebCore's
RenderObject.
Note that despite the use of GC concepts, it's not a goal to make this code directly sharable
with GC. The GC will probably have to do isolated heaps its own way (likely a special
Subspace or something like that).
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/Algorithm.h:
(bmalloc::findBitInWord):
* bmalloc/AllIsoHeaps.cpp: Added.
(bmalloc::AllIsoHeaps::AllIsoHeaps):
(bmalloc::AllIsoHeaps::add):
(bmalloc::AllIsoHeaps::head):
* bmalloc/AllIsoHeaps.h: Added.
* bmalloc/AllIsoHeapsInlines.h: Added.
(bmalloc::AllIsoHeaps::forEach):
* bmalloc/BMalloced.h: Added.
* bmalloc/Bits.h: Added.
(bmalloc::bitsArrayLength):
(bmalloc::BitsWordView::BitsWordView):
(bmalloc::BitsWordView::numBits const):
(bmalloc::BitsWordView::word const):
(bmalloc::BitsWordOwner::BitsWordOwner):
(bmalloc::BitsWordOwner::view const):
(bmalloc::BitsWordOwner::operator=):
(bmalloc::BitsWordOwner::setAll):
(bmalloc::BitsWordOwner::clearAll):
(bmalloc::BitsWordOwner::set):
(bmalloc::BitsWordOwner::numBits const):
(bmalloc::BitsWordOwner::arrayLength const):
(bmalloc::BitsWordOwner::word const):
(bmalloc::BitsWordOwner::word):
(bmalloc::BitsWordOwner::words const):
(bmalloc::BitsWordOwner::words):
(bmalloc::BitsAndWords::BitsAndWords):
(bmalloc::BitsAndWords::view const):
(bmalloc::BitsAndWords::numBits const):
(bmalloc::BitsAndWords::word const):
(bmalloc::BitsOrWords::BitsOrWords):
(bmalloc::BitsOrWords::view const):
(bmalloc::BitsOrWords::numBits const):
(bmalloc::BitsOrWords::word const):
(bmalloc::BitsNotWords::BitsNotWords):
(bmalloc::BitsNotWords::view const):
(bmalloc::BitsNotWords::numBits const):
(bmalloc::BitsNotWords::word const):
(bmalloc::BitsImpl::BitsImpl):
(bmalloc::BitsImpl::numBits const):
(bmalloc::BitsImpl::size const):
(bmalloc::BitsImpl::arrayLength const):
(bmalloc::BitsImpl::operator== const):
(bmalloc::BitsImpl::operator!= const):
(bmalloc::BitsImpl::at const):
(bmalloc::BitsImpl::operator[] const):
(bmalloc::BitsImpl::isEmpty const):
(bmalloc::BitsImpl::operator& const):
(bmalloc::BitsImpl::operator| const):
(bmalloc::BitsImpl::operator~ const):
(bmalloc::BitsImpl::forEachSetBit const):
(bmalloc::BitsImpl::forEachClearBit const):
(bmalloc::BitsImpl::forEachBit const):
(bmalloc::BitsImpl::findBit const):
(bmalloc::BitsImpl::findSetBit const):
(bmalloc::BitsImpl::findClearBit const):
(bmalloc::BitsImpl::wordView const):
(bmalloc::BitsImpl::atImpl const):
(bmalloc::Bits::Bits):
(bmalloc::Bits::operator=):
(bmalloc::Bits::resize):
(bmalloc::Bits::setAll):
(bmalloc::Bits::clearAll):
(bmalloc::Bits::setAndCheck):
(bmalloc::Bits::operator|=):
(bmalloc::Bits::operator&=):
(bmalloc::Bits::at const):
(bmalloc::Bits::operator[] const):
(bmalloc::Bits::BitReference::BitReference):
(bmalloc::Bits::BitReference::operator bool const):
(bmalloc::Bits::BitReference::operator=):
(bmalloc::Bits::at):
(bmalloc::Bits::operator[]):
* bmalloc/CryptoRandom.cpp: Replaced with Source/bmalloc/bmalloc/CryptoRandom.cpp.
(bmalloc::cryptoRandom):
* bmalloc/CryptoRandom.h: Replaced with Source/bmalloc/bmalloc/CryptoRandom.h.
* bmalloc/DeferredDecommit.h: Added.
* bmalloc/DeferredDecommitInlines.h: Added.
(bmalloc::DeferredDecommit::DeferredDecommit):
* bmalloc/DeferredTrigger.h: Added.
(bmalloc::DeferredTrigger::DeferredTrigger):
* bmalloc/DeferredTriggerInlines.h: Added.
(bmalloc::DeferredTrigger<trigger>::didBecome):
(bmalloc::DeferredTrigger<trigger>::handleDeferral):
* bmalloc/EligibilityResult.h: Added.
(bmalloc::EligibilityResult::EligibilityResult):
* bmalloc/EligibilityResultInlines.h: Added.
(bmalloc::EligibilityResult<Config>::EligibilityResult):
* bmalloc/FixedVector.h:
* bmalloc/FreeList.cpp: Added.
(bmalloc::FreeList::FreeList):
(bmalloc::FreeList::~FreeList):
(bmalloc::FreeList::clear):
(bmalloc::FreeList::initializeList):
(bmalloc::FreeList::initializeBump):
(bmalloc::FreeList::contains const):
* bmalloc/FreeList.h: Added.
(bmalloc::FreeCell::scramble):
(bmalloc::FreeCell::descramble):
(bmalloc::FreeCell::setNext):
(bmalloc::FreeCell::next const):
(bmalloc::FreeList::allocationWillFail const):
(bmalloc::FreeList::allocationWillSucceed const):
(bmalloc::FreeList::originalSize const):
(bmalloc::FreeList::head const):
* bmalloc/FreeListInlines.h: Added.
(bmalloc::FreeList::allocate):
(bmalloc::FreeList::forEach const):
* bmalloc/IsoAllocator.h: Added.
* bmalloc/IsoAllocatorInlines.h: Added.
(bmalloc::IsoAllocator<Config>::IsoAllocator):
(bmalloc::IsoAllocator<Config>::~IsoAllocator):
(bmalloc::IsoAllocator<Config>::allocate):
(bmalloc::IsoAllocator<Config>::allocateSlow):
(bmalloc::IsoAllocator<Config>::scavenge):
* bmalloc/IsoConfig.h: Added.
* bmalloc/IsoDeallocator.h: Added.
* bmalloc/IsoDeallocatorInlines.h: Added.
(bmalloc::IsoDeallocator<Config>::IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::~IsoDeallocator):
(bmalloc::IsoDeallocator<Config>::deallocate):
(bmalloc::IsoDeallocator<Config>::scavenge):
* bmalloc/IsoDirectory.h: Added.
(bmalloc::IsoDirectoryBaseBase::IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBaseBase::~IsoDirectoryBaseBase):
(bmalloc::IsoDirectoryBase::heap):
* bmalloc/IsoDirectoryInlines.h: Added.
(bmalloc::IsoDirectoryBase<Config>::IsoDirectoryBase):
(bmalloc::passedNumPages>::IsoDirectory):
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
(bmalloc::passedNumPages>::didDecommit):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::forEachCommittedPage):
* bmalloc/IsoDirectoryPage.h: Added.
(bmalloc::IsoDirectoryPage::index const):
* bmalloc/IsoDirectoryPageInlines.h: Added.
(bmalloc::IsoDirectoryPage<Config>::IsoDirectoryPage):
(bmalloc::IsoDirectoryPage<Config>::pageFor):
* bmalloc/IsoHeap.h: Added.
(bmalloc::api::IsoHeap::allocatorOffset):
(bmalloc::api::IsoHeap::setAllocatorOffset):
(bmalloc::api::IsoHeap::deallocatorOffset):
(bmalloc::api::IsoHeap::setDeallocatorOffset):
* bmalloc/IsoHeapImpl.cpp: Added.
(bmalloc::IsoHeapImplBase::IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::~IsoHeapImplBase):
(bmalloc::IsoHeapImplBase::scavengeNow):
(bmalloc::IsoHeapImplBase::finishScavenging):
* bmalloc/IsoHeapImpl.h: Added.
* bmalloc/IsoHeapImplInlines.h: Added.
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
(bmalloc::IsoHeapImpl<Config>::takeFirstEligible):
(bmalloc::IsoHeapImpl<Config>::didBecomeEligible):
(bmalloc::IsoHeapImpl<Config>::scavenge):
(bmalloc::IsoHeapImpl<Config>::allocatorOffset):
(bmalloc::IsoHeapImpl<Config>::deallocatorOffset):
(bmalloc::IsoHeapImpl<Config>::numLiveObjects):
(bmalloc::IsoHeapImpl<Config>::numCommittedPages):
(bmalloc::IsoHeapImpl<Config>::forEachDirectory):
(bmalloc::IsoHeapImpl<Config>::forEachCommittedPage):
(bmalloc::IsoHeapImpl<Config>::forEachLiveObject):
* bmalloc/IsoHeapInlines.h: Added.
(bmalloc::api::IsoHeap<Type>::allocate):
(bmalloc::api::IsoHeap<Type>::tryAllocate):
(bmalloc::api::IsoHeap<Type>::deallocate):
(bmalloc::api::IsoHeap<Type>::scavenge):
(bmalloc::api::IsoHeap<Type>::isInitialized):
(bmalloc::api::IsoHeap<Type>::impl):
* bmalloc/IsoPage.h: Added.
(bmalloc::IsoPage::index const):
(bmalloc::IsoPage::directory):
(bmalloc::IsoPage::isInUseForAllocation const):
(bmalloc::IsoPage::indexOfFirstObject):
* bmalloc/IsoPageInlines.h: Added.
(bmalloc::IsoPage<Config>::tryCreate):
(bmalloc::IsoPage<Config>::IsoPage):
(bmalloc::IsoPage<Config>::free):
(bmalloc::IsoPage<Config>::startAllocating):
(bmalloc::IsoPage<Config>::stopAllocating):
(bmalloc::IsoPage<Config>::forEachLiveObject):
* bmalloc/IsoPageTrigger.h: Added.
* bmalloc/IsoTLS.cpp: Added.
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::destructor):
(bmalloc::IsoTLS::sizeForCapacity):
(bmalloc::IsoTLS::capacityForSize):
(bmalloc::IsoTLS::size):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLS.h: Added.
* bmalloc/IsoTLSAllocatorEntry.h: Added.
* bmalloc/IsoTLSAllocatorEntryInlines.h: Added.
(bmalloc::IsoTLSAllocatorEntry<Config>::IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::~IsoTLSAllocatorEntry):
(bmalloc::IsoTLSAllocatorEntry<Config>::construct):
* bmalloc/IsoTLSDeallocatorEntry.h: Added.
* bmalloc/IsoTLSDeallocatorEntryInlines.h: Added.
(bmalloc::IsoTLSDeallocatorEntry<Config>::IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::~IsoTLSDeallocatorEntry):
(bmalloc::IsoTLSDeallocatorEntry<Config>::construct):
* bmalloc/IsoTLSEntry.cpp: Added.
(bmalloc::IsoTLSEntry::IsoTLSEntry):
(bmalloc::IsoTLSEntry::~IsoTLSEntry):
* bmalloc/IsoTLSEntry.h: Added.
(bmalloc::IsoTLSEntry::offset const):
(bmalloc::IsoTLSEntry::alignment const):
(bmalloc::IsoTLSEntry::size const):
(bmalloc::IsoTLSEntry::extent const):
* bmalloc/IsoTLSEntryInlines.h: Added.
(bmalloc::IsoTLSEntry::walkUpToInclusive):
(bmalloc::DefaultIsoTLSEntry<EntryType>::DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::~DefaultIsoTLSEntry):
(bmalloc::DefaultIsoTLSEntry<EntryType>::move):
(bmalloc::DefaultIsoTLSEntry<EntryType>::destruct):
(bmalloc::DefaultIsoTLSEntry<EntryType>::scavenge):
* bmalloc/IsoTLSInlines.h: Added.
(bmalloc::IsoTLS::allocate):
(bmalloc::IsoTLS::deallocate):
(bmalloc::IsoTLS::scavenge):
(bmalloc::IsoTLS::allocator):
(bmalloc::IsoTLS::deallocator):
(bmalloc::IsoTLS::get):
(bmalloc::IsoTLS::set):
(bmalloc::IsoTLS::ensureHeap):
(bmalloc::IsoTLS::ensureHeapAndEntries):
* bmalloc/IsoTLSLayout.cpp: Added.
(bmalloc::IsoTLSLayout::IsoTLSLayout):
(bmalloc::IsoTLSLayout::add):
* bmalloc/IsoTLSLayout.h: Added.
(bmalloc::IsoTLSLayout::head const):
* bmalloc/PerHeapKind.h:
* bmalloc/PerProcess.h:
(bmalloc::PerProcess<T>::getFastCase):
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h:
* bmalloc/bmalloc.h:
(bmalloc::api::scavengeThisThread):
* test: Added.
* test/testbmalloc.cpp: Added.
(hiddenTruthBecauseNoReturnIsStupid):
(usage):
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
(testIsoSimple):
(testIsoSimpleScavengeBeforeDealloc):
(testIsoFlipFlopFragmentedPages):
(testIsoFlipFlopFragmentedPagesScavengeInMiddle):
(BisoMalloced::BisoMalloced):
(testBisoMalloced):
(BisoMallocedInline::BisoMallocedInline):
(testBisoMallocedInline):
(run):
(main):
Source/WebCore:
No new tests because no new change in behavior. Though, the bmalloc change has a unit test.
Adopting IsoHeap means dropping in macros in both the .h and .cpp file of each class that we
opt in. It's not pretty, but it helps ensure speedy allocation since it means that we never
have to do any kind of switch or dynamic lookup to find the right allocator for a type.
This change is perf-neutral on MotionMark, PLT3, and membuster.
* Sources.txt:
* html/shadow/SliderThumbElement.cpp:
* html/shadow/SliderThumbElement.h:
* html/shadow/mac/ImageControlsButtonElementMac.cpp:
* html/shadow/mac/ImageControlsRootElementMac.cpp:
* rendering/RenderAttachment.cpp:
* rendering/RenderAttachment.h:
* rendering/RenderBlock.cpp:
* rendering/RenderBlock.h:
* rendering/RenderBlockFlow.cpp:
* rendering/RenderBlockFlow.h:
* rendering/RenderBox.cpp:
* rendering/RenderBox.h:
* rendering/RenderBoxModelObject.cpp:
* rendering/RenderBoxModelObject.h:
* rendering/RenderButton.cpp:
* rendering/RenderButton.h:
* rendering/RenderCombineText.cpp:
* rendering/RenderCombineText.h:
* rendering/RenderCounter.cpp:
* rendering/RenderCounter.h:
* rendering/RenderDeprecatedFlexibleBox.cpp:
* rendering/RenderDeprecatedFlexibleBox.h:
* rendering/RenderDetailsMarker.cpp:
* rendering/RenderDetailsMarker.h:
* rendering/RenderElement.cpp:
* rendering/RenderElement.h:
* rendering/RenderEmbeddedObject.cpp:
* rendering/RenderEmbeddedObject.h:
* rendering/RenderFileUploadControl.cpp:
* rendering/RenderFileUploadControl.h:
* rendering/RenderFlexibleBox.cpp:
* rendering/RenderFlexibleBox.h:
* rendering/RenderFragmentContainer.cpp:
* rendering/RenderFragmentContainer.h:
* rendering/RenderFragmentContainerSet.cpp:
* rendering/RenderFragmentContainerSet.h:
* rendering/RenderFragmentedFlow.cpp:
* rendering/RenderFragmentedFlow.h:
* rendering/RenderFrameBase.cpp:
* rendering/RenderFrameBase.h:
* rendering/RenderFrameSet.cpp:
* rendering/RenderFrameSet.h:
* rendering/RenderFullScreen.cpp:
* rendering/RenderFullScreen.h:
* rendering/RenderGrid.cpp:
* rendering/RenderGrid.h:
* rendering/RenderHTMLCanvas.cpp:
* rendering/RenderHTMLCanvas.h:
* rendering/RenderImage.cpp:
* rendering/RenderImage.h:
* rendering/RenderImageResourceStyleImage.cpp:
* rendering/RenderImageResourceStyleImage.h:
* rendering/RenderInline.cpp:
* rendering/RenderInline.h:
* rendering/RenderLayerModelObject.cpp:
* rendering/RenderLayerModelObject.h:
* rendering/RenderLineBreak.cpp:
* rendering/RenderLineBreak.h:
* rendering/RenderListBox.cpp:
* rendering/RenderListBox.h:
* rendering/RenderListItem.cpp:
* rendering/RenderListItem.h:
* rendering/RenderListMarker.cpp:
* rendering/RenderListMarker.h:
* rendering/RenderMedia.cpp:
* rendering/RenderMedia.h:
* rendering/RenderMediaControlElements.cpp:
* rendering/RenderMediaControlElements.h:
* rendering/RenderMenuList.cpp:
* rendering/RenderMenuList.h:
* rendering/RenderMeter.cpp:
* rendering/RenderMeter.h:
* rendering/RenderMultiColumnFlow.cpp:
* rendering/RenderMultiColumnFlow.h:
* rendering/RenderMultiColumnSet.cpp:
* rendering/RenderMultiColumnSet.h:
* rendering/RenderMultiColumnSpannerPlaceholder.cpp:
* rendering/RenderMultiColumnSpannerPlaceholder.h:
* rendering/RenderObject.cpp:
* rendering/RenderObject.h:
* rendering/RenderProgress.cpp:
* rendering/RenderProgress.h:
* rendering/RenderQuote.cpp:
* rendering/RenderQuote.h:
* rendering/RenderReplaced.cpp:
* rendering/RenderReplaced.h:
* rendering/RenderReplica.cpp:
* rendering/RenderReplica.h:
* rendering/RenderRuby.cpp:
* rendering/RenderRuby.h:
* rendering/RenderRubyBase.cpp:
* rendering/RenderRubyBase.h:
* rendering/RenderRubyRun.cpp:
* rendering/RenderRubyRun.h:
* rendering/RenderRubyText.cpp:
* rendering/RenderRubyText.h:
* rendering/RenderScrollbarPart.cpp:
* rendering/RenderScrollbarPart.h:
* rendering/RenderSearchField.cpp:
* rendering/RenderSearchField.h:
* rendering/RenderSlider.cpp:
* rendering/RenderSlider.h:
* rendering/RenderTable.cpp:
* rendering/RenderTable.h:
* rendering/RenderTableCaption.cpp:
* rendering/RenderTableCaption.h:
* rendering/RenderTableCell.cpp:
* rendering/RenderTableCell.h:
* rendering/RenderTableCol.cpp:
* rendering/RenderTableCol.h:
* rendering/RenderTableRow.cpp:
* rendering/RenderTableRow.h:
* rendering/RenderTableSection.cpp:
* rendering/RenderTableSection.h:
* rendering/RenderText.cpp:
* rendering/RenderText.h:
* rendering/RenderTextControl.cpp:
* rendering/RenderTextControl.h:
* rendering/RenderTextControlMultiLine.cpp:
* rendering/RenderTextControlMultiLine.h:
* rendering/RenderTextControlSingleLine.cpp:
* rendering/RenderTextControlSingleLine.h:
* rendering/RenderTextFragment.cpp:
* rendering/RenderTextFragment.h:
* rendering/RenderVTTCue.cpp:
* rendering/RenderVTTCue.h:
* rendering/RenderVideo.cpp:
* rendering/RenderVideo.h:
* rendering/RenderView.cpp:
* rendering/RenderView.h:
* rendering/RenderWidget.cpp:
* rendering/RenderWidget.h:
* rendering/mathml/RenderMathMLBlock.cpp:
* rendering/mathml/RenderMathMLBlock.h:
* rendering/mathml/RenderMathMLFenced.cpp:
* rendering/mathml/RenderMathMLFenced.h:
* rendering/mathml/RenderMathMLFencedOperator.cpp:
* rendering/mathml/RenderMathMLFencedOperator.h:
* rendering/mathml/RenderMathMLFraction.cpp:
* rendering/mathml/RenderMathMLFraction.h:
* rendering/mathml/RenderMathMLMath.cpp:
* rendering/mathml/RenderMathMLMath.h:
* rendering/mathml/RenderMathMLMenclose.cpp:
* rendering/mathml/RenderMathMLMenclose.h:
* rendering/mathml/RenderMathMLOperator.cpp:
* rendering/mathml/RenderMathMLOperator.h:
* rendering/mathml/RenderMathMLPadded.cpp:
* rendering/mathml/RenderMathMLPadded.h:
* rendering/mathml/RenderMathMLRoot.cpp:
* rendering/mathml/RenderMathMLRoot.h:
* rendering/mathml/RenderMathMLRow.cpp:
* rendering/mathml/RenderMathMLRow.h:
* rendering/mathml/RenderMathMLScripts.cpp:
* rendering/mathml/RenderMathMLScripts.h:
* rendering/mathml/RenderMathMLSpace.cpp:
* rendering/mathml/RenderMathMLSpace.h:
* rendering/mathml/RenderMathMLToken.cpp:
* rendering/mathml/RenderMathMLToken.h:
* rendering/mathml/RenderMathMLUnderOver.cpp:
* rendering/mathml/RenderMathMLUnderOver.h:
* rendering/svg/RenderSVGBlock.cpp:
* rendering/svg/RenderSVGBlock.h:
* rendering/svg/RenderSVGContainer.cpp:
* rendering/svg/RenderSVGContainer.h:
* rendering/svg/RenderSVGEllipse.cpp:
* rendering/svg/RenderSVGEllipse.h:
* rendering/svg/RenderSVGForeignObject.cpp:
* rendering/svg/RenderSVGForeignObject.h:
* rendering/svg/RenderSVGGradientStop.cpp:
* rendering/svg/RenderSVGGradientStop.h:
* rendering/svg/RenderSVGHiddenContainer.cpp:
* rendering/svg/RenderSVGHiddenContainer.h:
* rendering/svg/RenderSVGImage.cpp:
* rendering/svg/RenderSVGImage.h:
* rendering/svg/RenderSVGInline.cpp:
* rendering/svg/RenderSVGInline.h:
* rendering/svg/RenderSVGInlineText.cpp:
* rendering/svg/RenderSVGInlineText.h:
* rendering/svg/RenderSVGModelObject.cpp:
* rendering/svg/RenderSVGModelObject.h:
* rendering/svg/RenderSVGPath.cpp:
* rendering/svg/RenderSVGPath.h:
* rendering/svg/RenderSVGRect.cpp:
* rendering/svg/RenderSVGRect.h:
* rendering/svg/RenderSVGResourceClipper.cpp:
* rendering/svg/RenderSVGResourceClipper.h:
* rendering/svg/RenderSVGResourceContainer.cpp:
* rendering/svg/RenderSVGResourceContainer.h:
* rendering/svg/RenderSVGResourceFilter.cpp:
* rendering/svg/RenderSVGResourceFilter.h:
* rendering/svg/RenderSVGResourceFilterPrimitive.cpp:
* rendering/svg/RenderSVGResourceFilterPrimitive.h:
* rendering/svg/RenderSVGResourceGradient.cpp:
* rendering/svg/RenderSVGResourceGradient.h:
* rendering/svg/RenderSVGResourceLinearGradient.cpp:
* rendering/svg/RenderSVGResourceLinearGradient.h:
* rendering/svg/RenderSVGResourceMarker.cpp:
* rendering/svg/RenderSVGResourceMarker.h:
* rendering/svg/RenderSVGResourceMasker.cpp:
* rendering/svg/RenderSVGResourceMasker.h:
* rendering/svg/RenderSVGResourcePattern.cpp:
* rendering/svg/RenderSVGResourcePattern.h:
* rendering/svg/RenderSVGResourceRadialGradient.cpp:
* rendering/svg/RenderSVGResourceRadialGradient.h:
* rendering/svg/RenderSVGRoot.cpp:
* rendering/svg/RenderSVGRoot.h:
* rendering/svg/RenderSVGShape.cpp:
* rendering/svg/RenderSVGShape.h:
* rendering/svg/RenderSVGTSpan.cpp: Added.
* rendering/svg/RenderSVGTSpan.h:
* rendering/svg/RenderSVGText.cpp:
* rendering/svg/RenderSVGText.h:
* rendering/svg/RenderSVGTextPath.cpp:
* rendering/svg/RenderSVGTextPath.h:
* rendering/svg/RenderSVGTransformableContainer.cpp:
* rendering/svg/RenderSVGTransformableContainer.h:
* rendering/svg/RenderSVGViewportContainer.cpp:
* rendering/svg/RenderSVGViewportContainer.h:
Source/WTF:
This just adds an easy way of using the bmalloc IsoHeap API in WebKit. If bmalloc is not
enabled, these macros will just make the object FastMalloced.
* WTF.xcodeproj/project.pbxproj:
* wtf/FastTLS.h:
* wtf/IsoMalloc.h: Added.
* wtf/IsoMallocInlines.h: Added.
Canonical link: https://commits.webkit.org/195445@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@224537 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-07 19:21:52 +00:00
|
|
|
});
|
2018-04-02 21:09:45 +00:00
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
|
|
|
size_t Scavenger::footprint()
|
|
|
|
{
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
RELEASE_BASSERT(!Environment::get()->isDebugHeapEnabled());
|
2018-04-02 21:09:45 +00:00
|
|
|
|
|
|
|
size_t result = 0;
|
|
|
|
for (unsigned i = numHeaps; i--;) {
|
|
|
|
if (!isActiveHeapKind(static_cast<HeapKind>(i)))
|
|
|
|
continue;
|
|
|
|
result += PerProcess<PerHeapKind<Heap>>::get()->at(i).footprint();
|
|
|
|
}
|
|
|
|
|
[bmalloc] Add StaticPerProcess for known types to save pages
https://bugs.webkit.org/show_bug.cgi?id=195691
Reviewed by Mark Lam.
As initial memory footprint of VM + JSGlobalObject becomes 488KB dirty size in fast malloc memory (w/ JSC_useJIT=0 and Malloc=1), pages for PerProcess is costly.
For example, under Malloc=1 mode, we still need to allocate PerProcess<DebugHeap> and PerProcess<Environment>. And sizeof(Environment) is only 1 (bool flag), and
sizeof(DebugHeap) is 120. But we are allocating 1 pages for them. Since page size in iOS is 16KB, this 121B consumes 16KB dirty memory, and it is not negligible
size if we keep in mind that the current fast malloc heap size is 488KB. Putting them into the __DATA section, close to the other mutable data, we can avoid allocating
this page.
This patch revives the SafePerProcess concept in r228107. We add "StaticPerProcess<T>", which allocates underlying storage statically in the __DATA section instead of
allocating it at runtime. And we use this StaticPerProcess<T> for types where (1) T is known a priori, and (2) sizeof(T) is not huge.
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllIsoHeaps.cpp:
* bmalloc/AllIsoHeaps.h:
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
* bmalloc/Cache.cpp:
(bmalloc::Cache::Cache):
* bmalloc/CryptoRandom.cpp:
(bmalloc::cryptoRandom):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
* bmalloc/DebugHeap.cpp:
* bmalloc/DebugHeap.h:
(bmalloc::DebugHeap::tryGet):
* bmalloc/Environment.cpp:
* bmalloc/Environment.h:
* bmalloc/Gigacage.cpp:
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::shouldBeEnabled):
(Gigacage::bmalloc::Callback::Callback): Deleted.
(Gigacage::bmalloc::Callback::function): Deleted.
(Gigacage::bmalloc::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): Deleted.
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::tryAllocateLarge):
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::didBecome):
* bmalloc/IsoHeapImpl.cpp:
(bmalloc::IsoHeapImplBase::addToAllIsoHeaps):
* bmalloc/IsoPage.cpp:
(bmalloc::IsoPageBase::allocatePageMemory):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::IsoTLS):
(bmalloc::IsoTLS::ensureEntries):
(bmalloc::IsoTLS::forEachEntry):
* bmalloc/IsoTLSEntry.cpp:
(bmalloc::IsoTLSEntry::IsoTLSEntry):
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/IsoTLSLayout.cpp:
* bmalloc/IsoTLSLayout.h:
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::dumpStats):
(bmalloc::Scavenger::scavenge):
(bmalloc::Scavenger::partialScavenge):
(bmalloc::Scavenger::freeableMemory):
(bmalloc::Scavenger::footprint):
* bmalloc/Scavenger.h:
* bmalloc/StaticPerProcess.h: Added.
* bmalloc/VMHeap.cpp:
* bmalloc/VMHeap.h:
* bmalloc/Zone.h:
* bmalloc/bmalloc.cpp:
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::enableMiniMode):
* test/testbmalloc.cpp:
(assertEmptyPointerSet):
(assertHasObjects):
(assertHasOnlyObjects):
(assertClean):
Canonical link: https://commits.webkit.org/210028@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@242938 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-14 08:02:00 +00:00
|
|
|
AllIsoHeaps::get()->forEach(
|
2018-04-02 21:09:45 +00:00
|
|
|
[&] (IsoHeapImplBase& heap) {
|
|
|
|
result += heap.footprint();
|
|
|
|
});
|
|
|
|
|
|
|
|
return result;
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
}
|
|
|
|
|
2017-10-06 16:34:41 +00:00
|
|
|
void Scavenger::threadEntryPoint(Scavenger* scavenger)
|
|
|
|
{
|
|
|
|
scavenger->threadRunLoop();
|
|
|
|
}
|
|
|
|
|
|
|
|
void Scavenger::threadRunLoop()
|
|
|
|
{
|
|
|
|
setSelfQOSClass();
|
2018-04-21 21:58:51 +00:00
|
|
|
#if BOS(DARWIN)
|
|
|
|
setThreadName("JavaScriptCore bmalloc scavenger");
|
|
|
|
#else
|
|
|
|
setThreadName("BMScavenger");
|
|
|
|
#endif
|
2017-10-06 16:34:41 +00:00
|
|
|
|
|
|
|
// This loop ratchets downward from most active to least active state. While
|
|
|
|
// we ratchet downward, any other thread may reset our state.
|
|
|
|
|
|
|
|
// We require any state change while we are sleeping to signal to our
|
|
|
|
// condition variable and wake us up.
|
|
|
|
|
2018-06-07 21:05:29 +00:00
|
|
|
while (true) {
|
2017-10-06 16:34:41 +00:00
|
|
|
if (m_state == State::Sleep) {
|
2020-01-18 00:43:00 +00:00
|
|
|
UniqueLockHolder lock(mutex());
|
2017-10-06 16:34:41 +00:00
|
|
|
m_condition.wait(lock, [&]() { return m_state != State::Sleep; });
|
|
|
|
}
|
|
|
|
|
|
|
|
if (m_state == State::RunSoon) {
|
2020-01-18 00:43:00 +00:00
|
|
|
UniqueLockHolder lock(mutex());
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
m_condition.wait_for(lock, m_waitTime, [&]() { return m_state != State::RunSoon; });
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
m_state = State::Sleep;
|
|
|
|
|
|
|
|
setSelfQOSClass();
|
|
|
|
|
2018-04-10 23:34:42 +00:00
|
|
|
if (verbose) {
|
|
|
|
fprintf(stderr, "--------------------------------\n");
|
|
|
|
fprintf(stderr, "considering running scavenger\n");
|
|
|
|
dumpStats();
|
|
|
|
fprintf(stderr, "--------------------------------\n");
|
|
|
|
}
|
|
|
|
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
std::chrono::steady_clock::time_point start { std::chrono::steady_clock::now() };
|
|
|
|
|
|
|
|
scavenge();
|
2018-05-29 18:56:05 +00:00
|
|
|
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
auto timeSpentScavenging = std::chrono::steady_clock::now() - start;
|
2018-04-10 23:34:42 +00:00
|
|
|
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
if (verbose) {
|
|
|
|
fprintf(stderr, "time spent scavenging %lfms\n",
|
|
|
|
static_cast<double>(std::chrono::duration_cast<std::chrono::microseconds>(timeSpentScavenging).count()) / 1000);
|
|
|
|
}
|
2018-05-29 18:56:05 +00:00
|
|
|
|
2019-11-08 01:29:07 +00:00
|
|
|
// FIXME: We need to investigate mini-mode's adjustment.
|
|
|
|
// https://bugs.webkit.org/show_bug.cgi?id=203987
|
|
|
|
if (!m_isInMiniMode) {
|
2021-07-06 21:20:53 +00:00
|
|
|
timeSpentScavenging *= s_newWaitMultiplier;
|
2019-11-08 01:29:07 +00:00
|
|
|
std::chrono::milliseconds newWaitTime = std::chrono::duration_cast<std::chrono::milliseconds>(timeSpentScavenging);
|
2021-07-06 21:20:53 +00:00
|
|
|
m_waitTime = std::min(std::max(newWaitTime, std::chrono::milliseconds(s_minWaitTimeMilliseconds)), std::chrono::milliseconds(s_maxWaitTimeMilliseconds));
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
[BMalloc] Scavenger should react to recent memory activity
https://bugs.webkit.org/show_bug.cgi?id=195895
Reviewed by Geoffrey Garen.
This change adds a recently used bit to objects that are scavenged. When an object is allocated, that bit is set.
When we scavenge, if the bit is set, we clear it. If the bit was already clear, we decommit the object. The timing
to scavenging has been changed as well. We perform our first scavne almost immediately after bmalloc is initialized
(10ms later). Subsequent scavenging is done as a multiple of the time it took to scavenge. We bound this computed
time between a minimum and maximum. Through empirical testing, the multiplier, minimum and maximum are
150x, 100ms and 10,000ms respectively. For mini-mode, when the JIT is disabled, we use much more aggressive values of
50x, 25ms and 500ms.
Eliminated partial scavenging since this change allows for any scavenge to be partial or full based on recent use of
the objects on the various free lists.
* bmalloc/Chunk.h:
(bmalloc::Chunk::usedSinceLastScavenge):
(bmalloc::Chunk::clearUsedSinceLastScavenge):
(bmalloc::Chunk::setUsedSinceLastScavenge):
* bmalloc/Heap.cpp:
(bmalloc::Heap::scavenge):
(bmalloc::Heap::allocateSmallChunk):
(bmalloc::Heap::allocateSmallPage):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::scavengeToHighWatermark): Deleted.
* bmalloc/Heap.h:
* bmalloc/IsoDirectory.h:
* bmalloc/IsoDirectoryInlines.h:
(bmalloc::passedNumPages>::takeFirstEligible):
(bmalloc::passedNumPages>::scavenge):
(bmalloc::passedNumPages>::scavengeToHighWatermark): Deleted.
* bmalloc/IsoHeapImpl.h:
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::scavengeToHighWatermark): Deleted.
* bmalloc/LargeRange.h:
(bmalloc::LargeRange::LargeRange):
(bmalloc::LargeRange::usedSinceLastScavenge):
(bmalloc::LargeRange::clearUsedSinceLastScavenge):
(bmalloc::LargeRange::setUsedSinceLastScavenge):
(): Deleted.
* bmalloc/Scavenger.cpp:
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::threadRunLoop):
(bmalloc::Scavenger::timeSinceLastPartialScavenge): Deleted.
(bmalloc::Scavenger::partialScavenge): Deleted.
* bmalloc/Scavenger.h:
* bmalloc/SmallPage.h:
(bmalloc::SmallPage::usedSinceLastScavenge):
(bmalloc::SmallPage::clearUsedSinceLastScavenge):
(bmalloc::SmallPage::setUsedSinceLastScavenge):
Canonical link: https://commits.webkit.org/210213@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243144 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-03-19 17:31:01 +00:00
|
|
|
|
|
|
|
if (verbose)
|
2019-03-19 20:07:51 +00:00
|
|
|
fprintf(stderr, "new wait time %lldms\n", static_cast<long long int>(m_waitTime.count()));
|
2017-10-06 16:34:41 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-04-10 07:58:19 +00:00
|
|
|
void Scavenger::setThreadName(const char* name)
|
|
|
|
{
|
|
|
|
BUNUSED(name);
|
2020-02-27 04:56:58 +00:00
|
|
|
#if BOS(DARWIN) || BPLATFORM(PLAYSTATION)
|
2018-04-10 07:58:19 +00:00
|
|
|
pthread_setname_np(name);
|
|
|
|
#elif BOS(LINUX)
|
|
|
|
// Truncate the given name since Linux limits the size of the thread name 16 including null terminator.
|
|
|
|
std::array<char, 16> buf;
|
|
|
|
strncpy(buf.data(), name, buf.size() - 1);
|
|
|
|
buf[buf.size() - 1] = '\0';
|
|
|
|
pthread_setname_np(pthread_self(), buf.data());
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2017-10-06 16:34:41 +00:00
|
|
|
void Scavenger::setSelfQOSClass()
|
|
|
|
{
|
|
|
|
#if BOS(DARWIN)
|
|
|
|
pthread_set_qos_class_self_np(requestedScavengerThreadQOSClass(), 0);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
https://bugs.webkit.org/show_bug.cgi?id=174727
Reviewed by Mark Lam.
Source/bmalloc:
This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:
- You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.
- Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a
Cache for each heap kind.
- Each isoheap gets its own Heap.
- Each Heap gets a scavenger thread.
- Some things, like Zone/VMHeap/Scavenger, are per-process.
Most of the per-HeapKind functionality is handled by PerHeapKind<>.
This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.
* CMakeLists.txt:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/AllocationKind.h: Added.
* bmalloc/Allocator.cpp:
(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):
* bmalloc/Allocator.h:
* bmalloc/BExport.h: Added.
* bmalloc/Cache.cpp:
(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.
* bmalloc/Cache.h:
(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):
* bmalloc/Deallocator.cpp:
(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):
* bmalloc/Deallocator.h:
* bmalloc/Gigacage.cpp: Added.
(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
* bmalloc/Gigacage.h: Added.
(Gigacage::caged):
(Gigacage::isCaged):
* bmalloc/Heap.cpp:
(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):
* bmalloc/Heap.h:
(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.
* bmalloc/HeapKind.h: Added.
* bmalloc/ObjectType.cpp:
(bmalloc::objectType):
* bmalloc/ObjectType.h:
* bmalloc/PerHeapKind.h: Added.
(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):
* bmalloc/PerThread.h:
(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.
* bmalloc/Scavenger.cpp: Added.
(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):
* bmalloc/Scavenger.h: Added.
(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):
* bmalloc/VMHeap.cpp:
(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):
* bmalloc/VMHeap.h:
* bmalloc/Zone.cpp:
(bmalloc::Zone::Zone):
* bmalloc/Zone.h:
* bmalloc/bmalloc.h:
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):
* bmalloc/mbmalloc.cpp:
Source/JavaScriptCore:
This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.
This is neutral on JetStream.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* b3/B3InsertionSet.cpp:
(JSC::B3::InsertionSet::execute):
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGArgumentsEliminationPhase.cpp:
* dfg/DFGClobberize.cpp:
(JSC::DFG::readsOverlap):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.
(JSC::DFG::performFixedButterflyAccessUncaging):
* dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
* dfg/DFGHeapLocation.cpp:
(WTF::printInternal):
* dfg/DFGHeapLocation.h:
* dfg/DFGNodeType.h:
* dfg/DFGPlan.cpp:
(JSC::DFG::Plan::compileInThreadImpl):
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compileGetButterfly):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGTypeCheckHoistingPhase.cpp:
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):
* heap/GigacageSubspace.cpp: Added.
(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):
* heap/GigacageSubspace.h: Added.
* heap/Heap.cpp:
(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.
* heap/Heap.h:
* heap/LargeAllocation.cpp:
(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.
* heap/MarkedBlock.h:
(JSC::MarkedBlock::Handle::subspace const):
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.
* heap/MarkedSpace.h:
(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.
* heap/Subspace.cpp:
(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):
* heap/Subspace.h:
(JSC::Subspace::didCreateFirstAllocator):
* heap/SubspaceInlines.h:
(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):
* jit/JITPropertyAccess.cpp:
(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):
* jsc.cpp:
(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):
* llint/LowLevelInterpreter64.asm:
* runtime/ArrayBuffer.cpp:
(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):
* runtime/IndexingHeader.h:
* runtime/InitializeThreading.cpp:
(JSC::initializeThreading):
* runtime/JSArrayBuffer.cpp:
* runtime/JSArrayBufferView.cpp:
(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):
* runtime/JSLock.cpp:
(JSC::JSLock::didAcquireLock):
* runtime/JSObject.h:
* runtime/Options.cpp:
(JSC::recomputeDependentOptions):
* runtime/Options.h:
* runtime/ScopedArgumentsTable.h:
* runtime/VM.cpp:
(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):
* runtime/VM.h:
(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):
* wasm/WasmB3IRGenerator.cpp:
(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):
* wasm/WasmCodeBlock.cpp:
(JSC::Wasm::CodeBlock::isSafeToRun):
* wasm/WasmMemory.cpp:
(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.
* wasm/WasmMemory.h:
* wasm/js/JSWebAssemblyInstance.cpp:
(JSC::JSWebAssemblyInstance::create):
* wasm/js/JSWebAssemblyMemory.cpp:
(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):
* wasm/js/JSWebAssemblyMemory.h:
(JSC::JSWebAssemblyMemory::subspaceFor):
Source/WebCore:
No new tests because no change in behavior.
Needed to teach Metal how to allocate in the Gigacage.
* platform/graphics/cocoa/GPUBufferMetal.mm:
(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):
Source/WebKit:
The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.
* WebProcess/WebProcess.cpp:
(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
Source/WTF:
For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.
Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.
* WTF.xcodeproj/project.pbxproj:
* wtf/CMakeLists.txt:
* wtf/FastMalloc.cpp:
* wtf/Gigacage.cpp: Added.
(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
* wtf/Gigacage.h: Added.
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
Canonical link: https://commits.webkit.org/191825@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
|
|
|
} // namespace bmalloc
|
|
|
|
|