haikuwebkit/Source/WTF/wtf/Gigacage.cpp

161 lines
4.5 KiB
C++
Raw Permalink Normal View History

Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
/*
Refactor the Gigacage code to require less pointer casting. https://bugs.webkit.org/show_bug.cgi?id=201521 Reviewed by Saam Barati. Source/bmalloc: 1. Define a Gigacage::Config struct instead of hacking around a g_gigacageBasePtrs array of bytes. 2. Change Gigacage::basePtr() to return a pointer instead of a reference to the requested basePtr. Instead, make it explicit when the client is trying to take the address of the basePtr, or setting it. 3. Renamed wasEnabled() to isEnabled() because it returns the present state of the flag, not some past state. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::primitiveGigacageDisabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::Config::basePtr const): (Gigacage::Config::setBasePtr): (Gigacage::isEnabled): (Gigacage::basePtr): (Gigacage::addressOfBasePtr): (Gigacage::size): (Gigacage::caged): (Gigacage::wasEnabled): Deleted. (Gigacage::setWasEnabled): Deleted. (Gigacage::basePtrs): Deleted. * bmalloc/HeapKind.h: (bmalloc::heapKind): (bmalloc::isActiveHeapKindAfterEnsuringGigacage): (bmalloc::mapToActiveHeapKindAfterEnsuringGigacage): Source/JavaScriptCore: Change LLInt's loadCagedJSValue() to skip the caging if Gigacage is not enabled in the build. This allows us to remove the unneeded stubs in WTF Gigacage.h. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cageConditionally): * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter64.asm: * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): Source/WTF: Remove some unneeded stubs in WTF Gigacage.h. * wtf/Gigacage.cpp: * wtf/Gigacage.h: (Gigacage::name): (Gigacage::isEnabled): (Gigacage::basePtr): Deleted. (Gigacage::basePtrs): Deleted. Canonical link: https://commits.webkit.org/215156@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@249556 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-09-06 00:03:52 +00:00
* Copyright (C) 2017-2019 Apple Inc. All rights reserved.
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
*/
#include "config.h"
Use pragma once in WTF https://bugs.webkit.org/show_bug.cgi?id=190527 Reviewed by Chris Dumez. Source/WTF: We also need to consistently include wtf headers from within wtf so we can build wtf without symbol redefinition errors from including the copy in Source and the copy in the build directory. * wtf/ASCIICType.h: * wtf/Assertions.cpp: * wtf/Assertions.h: * wtf/Atomics.h: * wtf/AutomaticThread.cpp: * wtf/AutomaticThread.h: * wtf/BackwardsGraph.h: * wtf/Bag.h: * wtf/BagToHashMap.h: * wtf/BitVector.cpp: * wtf/BitVector.h: * wtf/Bitmap.h: * wtf/BloomFilter.h: * wtf/Box.h: * wtf/BubbleSort.h: * wtf/BumpPointerAllocator.h: * wtf/ByteOrder.h: * wtf/CPUTime.cpp: * wtf/CallbackAggregator.h: * wtf/CheckedArithmetic.h: * wtf/CheckedBoolean.h: * wtf/ClockType.cpp: * wtf/ClockType.h: * wtf/CommaPrinter.h: * wtf/CompilationThread.cpp: * wtf/CompilationThread.h: * wtf/Compiler.h: * wtf/ConcurrentPtrHashSet.cpp: * wtf/ConcurrentVector.h: * wtf/Condition.h: * wtf/CountingLock.cpp: * wtf/CrossThreadTaskHandler.cpp: * wtf/CryptographicUtilities.cpp: * wtf/CryptographicUtilities.h: * wtf/CryptographicallyRandomNumber.cpp: * wtf/CryptographicallyRandomNumber.h: * wtf/CurrentTime.cpp: * wtf/DataLog.cpp: * wtf/DataLog.h: * wtf/DateMath.cpp: * wtf/DateMath.h: * wtf/DecimalNumber.cpp: * wtf/DecimalNumber.h: * wtf/Deque.h: * wtf/DisallowCType.h: * wtf/Dominators.h: * wtf/DoublyLinkedList.h: * wtf/FastBitVector.cpp: * wtf/FastMalloc.cpp: * wtf/FastMalloc.h: * wtf/FeatureDefines.h: * wtf/FilePrintStream.cpp: * wtf/FilePrintStream.h: * wtf/FlipBytes.h: * wtf/FunctionDispatcher.cpp: * wtf/FunctionDispatcher.h: * wtf/GetPtr.h: * wtf/Gigacage.cpp: * wtf/GlobalVersion.cpp: * wtf/GraphNodeWorklist.h: * wtf/GregorianDateTime.cpp: * wtf/GregorianDateTime.h: * wtf/HashFunctions.h: * wtf/HashMap.h: * wtf/HashMethod.h: * wtf/HashSet.h: * wtf/HashTable.cpp: * wtf/HashTraits.h: * wtf/Indenter.h: * wtf/IndexSparseSet.h: * wtf/InlineASM.h: * wtf/Insertion.h: * wtf/IteratorAdaptors.h: * wtf/IteratorRange.h: * wtf/JSONValues.cpp: * wtf/JSValueMalloc.cpp: * wtf/LEBDecoder.h: * wtf/Language.cpp: * wtf/ListDump.h: * wtf/Lock.cpp: * wtf/Lock.h: * wtf/LockAlgorithm.h: * wtf/LockedPrintStream.cpp: * wtf/Locker.h: * wtf/MD5.cpp: * wtf/MD5.h: * wtf/MainThread.cpp: * wtf/MainThread.h: * wtf/MallocPtr.h: * wtf/MathExtras.h: * wtf/MediaTime.cpp: * wtf/MediaTime.h: * wtf/MemoryPressureHandler.cpp: * wtf/MessageQueue.h: * wtf/MetaAllocator.cpp: * wtf/MetaAllocator.h: * wtf/MetaAllocatorHandle.h: * wtf/MonotonicTime.cpp: * wtf/MonotonicTime.h: * wtf/NakedPtr.h: * wtf/NoLock.h: * wtf/NoTailCalls.h: * wtf/Noncopyable.h: * wtf/NumberOfCores.cpp: * wtf/NumberOfCores.h: * wtf/OSAllocator.h: * wtf/OSAllocatorPosix.cpp: * wtf/OSRandomSource.cpp: * wtf/OSRandomSource.h: * wtf/ObjcRuntimeExtras.h: * wtf/OrderMaker.h: * wtf/PackedIntVector.h: * wtf/PageAllocation.h: * wtf/PageBlock.cpp: * wtf/PageBlock.h: * wtf/PageReservation.h: * wtf/ParallelHelperPool.cpp: * wtf/ParallelHelperPool.h: * wtf/ParallelJobs.h: * wtf/ParallelJobsLibdispatch.h: * wtf/ParallelVectorIterator.h: * wtf/ParkingLot.cpp: * wtf/ParkingLot.h: * wtf/Platform.h: * wtf/PointerComparison.h: * wtf/Poisoned.cpp: * wtf/PrintStream.cpp: * wtf/PrintStream.h: * wtf/ProcessID.h: * wtf/ProcessPrivilege.cpp: * wtf/RAMSize.cpp: * wtf/RAMSize.h: * wtf/RandomDevice.cpp: * wtf/RandomNumber.cpp: * wtf/RandomNumber.h: * wtf/RandomNumberSeed.h: * wtf/RangeSet.h: * wtf/RawPointer.h: * wtf/ReadWriteLock.cpp: * wtf/RedBlackTree.h: * wtf/Ref.h: * wtf/RefCountedArray.h: * wtf/RefCountedLeakCounter.cpp: * wtf/RefCountedLeakCounter.h: * wtf/RefCounter.h: * wtf/RefPtr.h: * wtf/RetainPtr.h: * wtf/RunLoop.cpp: * wtf/RunLoop.h: * wtf/RunLoopTimer.h: * wtf/RunLoopTimerCF.cpp: * wtf/SHA1.cpp: * wtf/SHA1.h: * wtf/SaturatedArithmetic.h: (saturatedSubtraction): * wtf/SchedulePair.h: * wtf/SchedulePairCF.cpp: * wtf/SchedulePairMac.mm: * wtf/ScopedLambda.h: * wtf/Seconds.cpp: * wtf/Seconds.h: * wtf/SegmentedVector.h: * wtf/SentinelLinkedList.h: * wtf/SharedTask.h: * wtf/SimpleStats.h: * wtf/SingleRootGraph.h: * wtf/SinglyLinkedList.h: * wtf/SixCharacterHash.cpp: * wtf/SixCharacterHash.h: * wtf/SmallPtrSet.h: * wtf/Spectrum.h: * wtf/StackBounds.cpp: * wtf/StackBounds.h: * wtf/StackStats.cpp: * wtf/StackStats.h: * wtf/StackTrace.cpp: * wtf/StdLibExtras.h: * wtf/StreamBuffer.h: * wtf/StringHashDumpContext.h: * wtf/StringPrintStream.cpp: * wtf/StringPrintStream.h: * wtf/ThreadGroup.cpp: * wtf/ThreadMessage.cpp: * wtf/ThreadSpecific.h: * wtf/Threading.cpp: * wtf/Threading.h: * wtf/ThreadingPrimitives.h: * wtf/ThreadingPthreads.cpp: * wtf/TimeWithDynamicClockType.cpp: * wtf/TimeWithDynamicClockType.h: * wtf/TimingScope.cpp: * wtf/TinyLRUCache.h: * wtf/TinyPtrSet.h: * wtf/TriState.h: * wtf/TypeCasts.h: * wtf/UUID.cpp: * wtf/UnionFind.h: * wtf/VMTags.h: * wtf/ValueCheck.h: * wtf/Vector.h: * wtf/VectorTraits.h: * wtf/WallTime.cpp: * wtf/WallTime.h: * wtf/WeakPtr.h: * wtf/WeakRandom.h: * wtf/WordLock.cpp: * wtf/WordLock.h: * wtf/WorkQueue.cpp: * wtf/WorkQueue.h: * wtf/WorkerPool.cpp: * wtf/cf/LanguageCF.cpp: * wtf/cf/RunLoopCF.cpp: * wtf/cocoa/Entitlements.mm: * wtf/cocoa/MachSendRight.cpp: * wtf/cocoa/MainThreadCocoa.mm: * wtf/cocoa/MemoryFootprintCocoa.cpp: * wtf/cocoa/WorkQueueCocoa.cpp: * wtf/dtoa.cpp: * wtf/dtoa.h: * wtf/ios/WebCoreThread.cpp: * wtf/ios/WebCoreThread.h: * wtf/mac/AppKitCompatibilityDeclarations.h: * wtf/mac/DeprecatedSymbolsUsedBySafari.mm: * wtf/mbmalloc.cpp: * wtf/persistence/PersistentCoders.cpp: * wtf/persistence/PersistentDecoder.cpp: * wtf/persistence/PersistentEncoder.cpp: * wtf/spi/cf/CFBundleSPI.h: * wtf/spi/darwin/CommonCryptoSPI.h: * wtf/text/ASCIIFastPath.h: * wtf/text/ASCIILiteral.cpp: * wtf/text/AtomicString.cpp: * wtf/text/AtomicString.h: * wtf/text/AtomicStringHash.h: * wtf/text/AtomicStringImpl.cpp: * wtf/text/AtomicStringImpl.h: * wtf/text/AtomicStringTable.cpp: * wtf/text/AtomicStringTable.h: * wtf/text/Base64.cpp: * wtf/text/CString.cpp: * wtf/text/CString.h: * wtf/text/ConversionMode.h: * wtf/text/ExternalStringImpl.cpp: * wtf/text/IntegerToStringConversion.h: * wtf/text/LChar.h: * wtf/text/LineEnding.cpp: * wtf/text/StringBuffer.h: * wtf/text/StringBuilder.cpp: * wtf/text/StringBuilder.h: * wtf/text/StringBuilderJSON.cpp: * wtf/text/StringCommon.h: * wtf/text/StringConcatenate.h: * wtf/text/StringHash.h: * wtf/text/StringImpl.cpp: * wtf/text/StringImpl.h: * wtf/text/StringOperators.h: * wtf/text/StringView.cpp: * wtf/text/StringView.h: * wtf/text/SymbolImpl.cpp: * wtf/text/SymbolRegistry.cpp: * wtf/text/SymbolRegistry.h: * wtf/text/TextBreakIterator.cpp: * wtf/text/TextBreakIterator.h: * wtf/text/TextBreakIteratorInternalICU.h: * wtf/text/TextPosition.h: * wtf/text/TextStream.cpp: * wtf/text/UniquedStringImpl.h: * wtf/text/WTFString.cpp: * wtf/text/WTFString.h: * wtf/text/cocoa/StringCocoa.mm: * wtf/text/cocoa/StringViewCocoa.mm: * wtf/text/cocoa/TextBreakIteratorInternalICUCocoa.cpp: * wtf/text/icu/UTextProvider.cpp: * wtf/text/icu/UTextProvider.h: * wtf/text/icu/UTextProviderLatin1.cpp: * wtf/text/icu/UTextProviderLatin1.h: * wtf/text/icu/UTextProviderUTF16.cpp: * wtf/text/icu/UTextProviderUTF16.h: * wtf/threads/BinarySemaphore.cpp: * wtf/threads/BinarySemaphore.h: * wtf/threads/Signals.cpp: * wtf/unicode/CharacterNames.h: * wtf/unicode/Collator.h: * wtf/unicode/CollatorDefault.cpp: * wtf/unicode/UTF8.cpp: * wtf/unicode/UTF8.h: Tools: Put WorkQueue in namespace DRT so it does not conflict with WTF::WorkQueue. * DumpRenderTree/TestRunner.cpp: (TestRunner::queueLoadHTMLString): (TestRunner::queueLoadAlternateHTMLString): (TestRunner::queueBackNavigation): (TestRunner::queueForwardNavigation): (TestRunner::queueLoadingScript): (TestRunner::queueNonLoadingScript): (TestRunner::queueReload): * DumpRenderTree/WorkQueue.cpp: (WorkQueue::singleton): Deleted. (WorkQueue::WorkQueue): Deleted. (WorkQueue::queue): Deleted. (WorkQueue::dequeue): Deleted. (WorkQueue::count): Deleted. (WorkQueue::clear): Deleted. (WorkQueue::processWork): Deleted. * DumpRenderTree/WorkQueue.h: (WorkQueue::setFrozen): Deleted. * DumpRenderTree/WorkQueueItem.h: * DumpRenderTree/mac/DumpRenderTree.mm: (runTest): * DumpRenderTree/mac/FrameLoadDelegate.mm: (-[FrameLoadDelegate processWork:]): (-[FrameLoadDelegate webView:locationChangeDone:forDataSource:]): * DumpRenderTree/mac/TestRunnerMac.mm: (TestRunner::notifyDone): (TestRunner::forceImmediateCompletion): (TestRunner::queueLoad): * DumpRenderTree/win/DumpRenderTree.cpp: (runTest): * DumpRenderTree/win/FrameLoadDelegate.cpp: (FrameLoadDelegate::processWork): (FrameLoadDelegate::locationChangeDone): * DumpRenderTree/win/TestRunnerWin.cpp: (TestRunner::notifyDone): (TestRunner::forceImmediateCompletion): (TestRunner::queueLoad): Canonical link: https://commits.webkit.org/205473@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@237099 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-10-15 14:24:49 +00:00
#include <wtf/Gigacage.h>
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
#include <wtf/Atomics.h>
#include <wtf/PageBlock.h>
#if defined(USE_SYSTEM_MALLOC) && USE_SYSTEM_MALLOC
[IWYU] Try removing redundant includes in WTF implementation files https://bugs.webkit.org/show_bug.cgi?id=211821 Reviewed by Simon Fraser. * wtf/Assertions.cpp: * wtf/Bag.cpp: * wtf/BitVector.cpp: * wtf/ConcurrentBuffer.cpp: * wtf/ConcurrentPtrHashSet.cpp: * wtf/CrossThreadCopier.cpp: * wtf/CurrentTime.cpp: * wtf/DataLog.cpp: * wtf/DateMath.cpp: * wtf/FastBitVector.cpp: * wtf/FastMalloc.cpp: * wtf/FileSystem.cpp: * wtf/Gigacage.cpp: * wtf/HashTable.cpp: * wtf/JSValueMalloc.cpp: * wtf/Language.cpp: * wtf/MainThread.cpp: * wtf/MetaAllocator.cpp: * wtf/MonotonicTime.cpp: * wtf/NumberOfCores.cpp: * wtf/OSLogPrintStream.cpp: * wtf/ParallelHelperPool.cpp: * wtf/ParkingLot.cpp: * wtf/PrintStream.cpp: * wtf/RAMSize.cpp: * wtf/RandomDevice.cpp: * wtf/RandomNumber.cpp: * wtf/RefCountedArray.cpp: * wtf/RefCountedLeakCounter.cpp: * wtf/SegmentedVector.cpp: * wtf/SmallPtrSet.cpp: * wtf/StackBounds.cpp: * wtf/ThreadGroup.cpp: * wtf/Threading.cpp: * wtf/TimeWithDynamicClockType.cpp: * wtf/URL.cpp: * wtf/URLParser.cpp: * wtf/UUID.cpp: * wtf/UniqueArray.cpp: * wtf/Vector.cpp: * wtf/WTFConfig.cpp: * wtf/WallTime.cpp: * wtf/WordLock.cpp: * wtf/WorkQueue.cpp: * wtf/WorkerPool.cpp: * wtf/generic/WorkQueueGeneric.cpp: * wtf/posix/OSAllocatorPOSIX.cpp: * wtf/posix/ThreadingPOSIX.cpp: * wtf/text/AtomString.cpp: * wtf/text/AtomStringImpl.cpp: * wtf/text/AtomStringTable.cpp: * wtf/text/CString.cpp: * wtf/text/StringBuffer.cpp: * wtf/text/StringBuilder.cpp: * wtf/text/StringImpl.cpp: * wtf/text/StringView.cpp: * wtf/text/TextBreakIterator.cpp: * wtf/text/TextStream.cpp: * wtf/text/WTFString.cpp: * wtf/threads/Signals.cpp: Canonical link: https://commits.webkit.org/224759@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@261661 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-05-13 22:33:20 +00:00
#include <wtf/OSAllocator.h>
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
namespace Gigacage {
Gigacages should start allocations from a slide https://bugs.webkit.org/show_bug.cgi?id=193523 Reviewed by Mark Lam. Source/bmalloc: This patch makes it so that Gigacage Heaps slide the start of the cage by some random amount. We still ensure that there is always at least 4/2GB, on MacOS/iOS respectively, of VA space available for allocation. Also, this patch changes some macros into constants since macros are the devil. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::shouldBeEnabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::gigacageSizeToMask): (Gigacage::size): (Gigacage::mask): (Gigacage::basePtr): (Gigacage::ensureGigacage): (Gigacage::wasEnabled): (Gigacage::isCaged): (Gigacage::caged): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): * bmalloc/Sizes.h: (bmalloc::Sizes::maskSizeClass): (bmalloc::Sizes::maskObjectSize): (bmalloc::Sizes::logSizeClass): (bmalloc::Sizes::logObjectSize): (bmalloc::Sizes::sizeClass): (bmalloc::Sizes::objectSize): (bmalloc::Sizes::pageSize): Source/JavaScriptCore: This patch changes some macros into constants since macros are the devil. * llint/LowLevelInterpreter64.asm: Source/WTF: This patch changes some macros into constants since macros are the devil. * wtf/Gigacage.cpp: * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/208086@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@240160 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-01-18 19:38:56 +00:00
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void* tryMalloc(Kind, size_t size)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
JSTests: Unreviewed, skipping slow tests. These tests are now timing out. They would have always been slow. The timeouts are probably because OOMs work differently now. * stress/regexp-prototype-exec-on-too-long-rope.js: * stress/string-prototype-charCodeAt-on-too-long-rope.js: Source/bmalloc: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. This adds a StringGigacage. * bmalloc/Gigacage.cpp: * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): Source/JavaScriptCore: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. * runtime/JSString.cpp: (JSC::JSRopeString::resolveRopeToAtomicString const): (JSC::JSRopeString::resolveRope const): * runtime/JSString.h: (JSC::JSString::create): (JSC::JSString::createHasOtherOwner): * runtime/JSStringBuilder.h: * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): Source/WebCore: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. No new tests because no new behavior. * html/canvas/CanvasRenderingContext2D.cpp: (WebCore::normalizeSpaces): Source/WTF: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. This makes all strings allocations come from the string gigacage. Because we expect string allocation to be a hot path, I created specialized allocation paths for the string gigacage. These paths are accessible via <wtf/text/StringMalloc.h>. However, those paths are equivalent to saying Gigacage::malloc and friends with the Gigacage::String kind. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Deque.h: * wtf/FastMalloc.cpp: (WTF::fastFree): * wtf/FastMalloc.h: (WTF::FastMalloc::malloc): (WTF::FastMalloc::tryMalloc): (WTF::FastMalloc::realloc): (WTF::FastMalloc::free): * wtf/Forward.h: * wtf/Gigacage.cpp: (Gigacage::tryMalloc): * wtf/Gigacage.h: (Gigacage::name): * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. * wtf/text/AtomicStringImpl.h: * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): (WTF::StringImpl::releaseAssertCaged const): * wtf/text/StringImpl.h: (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): (WTF::StringImpl::bufferOwnership const): (WTF::StringImpl::assertCaged const): * wtf/text/StringMalloc.cpp: Added. (WTF::tryStringMalloc): (WTF::stringMalloc): (WTF::stringRealloc): (WTF::stringFree): * wtf/text/StringMalloc.h: Added. (WTF::StringMalloc::malloc): (WTF::StringMalloc::tryMalloc): (WTF::StringMalloc::realloc): (WTF::StringMalloc::free): * wtf/text/StringVector.h: Added. * wtf/text/SymbolImpl.h: * wtf/text/UniquedStringImpl.h: * wtf/text/WTFString.h: (WTF::String::adopt): (WTF::String::assertCaged const): (WTF::String::releaseAssertCaged const): Canonical link: https://commits.webkit.org/192824@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@221422 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-31 16:38:16 +00:00
return FastMalloc::tryMalloc(size);
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
}
[JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active https://bugs.webkit.org/show_bug.cgi?id=196160 Reviewed by Saam Barati. Source/JavaScriptCore: "realloc" can be effective in terms of peak/current memory footprint when realloc succeeds because, 1. It does not allocate additional memory while expanding a vector 2. It does not deallocate an old memory, just reusing the current memory by expanding, so that memory footprint is tight even before scavenging We found that we can "realloc" large butterflies in certain conditions are met because, 1. If it goes to LargeAllocation, this memory region is never reused until GC sweeps it. 2. Butterflies are owned by owner JSObjects, so we know the lifetime of Butterflies. This patch attempts to use "realloc" onto butterflies if, 1. Butterflies are allocated in LargeAllocation kind 2. Concurrent collector is not active 3. Butterflies do not have property storage The condition (2) is required to avoid deallocating butterflies while the concurrent collector looks into it. The condition (3) is also required to avoid deallocating butterflies while the concurrent compiler looks into it. We also change LargeAllocation mechanism to using "malloc" and "free" instead of "posix_memalign". This allows us to use "realloc" safely in all the platforms. Since LargeAllocation uses alignment to distinguish LargeAllocation and MarkedBlock, we manually adjust 16B alignment by allocating 8B more memory in "malloc". Speedometer2 and JetStream2 are neutral. RAMification shows about 1% progression (even in some of JIT tests). * heap/AlignedMemoryAllocator.h: * heap/CompleteSubspace.cpp: (JSC::CompleteSubspace::tryAllocateSlow): (JSC::CompleteSubspace::reallocateLargeAllocationNonVirtual): * heap/CompleteSubspace.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): (JSC::IsoAlignedMemoryAllocator::tryReallocateMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/LargeAllocation.cpp: (JSC::isAlignedForLargeAllocation): (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::tryReallocate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::destroy): * heap/LargeAllocation.h: (JSC::LargeAllocation::indexInSpace): (JSC::LargeAllocation::setIndexInSpace): (JSC::LargeAllocation::basePointer const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::prepareForConservativeScan): * heap/WeakSet.h: (JSC::WeakSet::isTriviallyDestructible const): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::reallocArrayRightIfPossible): * runtime/JSObject.cpp: (JSC::JSObject::ensureLengthSlow): Source/WTF: * wtf/FastMalloc.h: (WTF::FastMalloc::tryRealloc): * wtf/Gigacage.cpp: (Gigacage::tryRealloc): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/210704@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243688 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-04-01 06:51:11 +00:00
void* tryRealloc(Kind, void* pointer, size_t size)
{
return FastMalloc::tryRealloc(pointer, size);
}
void* tryAllocateZeroedVirtualPages(Kind, size_t requestedSize)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
size_t size = roundUpToMultipleOf(WTF::pageSize(), requestedSize);
RELEASE_ASSERT(size >= requestedSize);
void* result = OSAllocator::reserveUncommitted(size);
PerformanceTests: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * JetStream2/wasm/HashSet.cpp: * StitchMarker/wtf/Assertions.h: * StitchMarker/wtf/DateMath.cpp: (WTF::initializeDates): * StitchMarker/wtf/HashTable.h: * StitchMarker/wtf/Hasher.h: (WTF::StringHasher::addCharacters): * StitchMarker/wtf/NeverDestroyed.h: (WTF::LazyNeverDestroyed::construct): * StitchMarker/wtf/StackBounds.h: (WTF::StackBounds::checkConsistency const): * StitchMarker/wtf/ValueCheck.h: * StitchMarker/wtf/Vector.h: (WTF::minCapacity>::checkConsistency): * StitchMarker/wtf/text/AtomicStringImpl.cpp: * StitchMarker/wtf/text/AtomicStringImpl.h: * StitchMarker/wtf/text/StringCommon.h: (WTF::hasPrefixWithLettersIgnoringASCIICaseCommon): * StitchMarker/wtf/text/StringImpl.h: * StitchMarker/wtf/text/SymbolImpl.h: * StitchMarker/wtf/text/UniquedStringImpl.h: Source/JavaScriptCore: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * API/tests/testapi.c: * assembler/ARM64Assembler.h: (JSC::ARM64Assembler::replaceWithLoad): (JSC::ARM64Assembler::replaceWithAddressComputation): * assembler/AssemblerBuffer.h: (JSC::AssemblerBuffer::LocalWriter::LocalWriter): * assembler/LinkBuffer.cpp: (JSC::LinkBuffer::copyCompactAndLinkCode): * assembler/ProbeStack.cpp: (JSC::Probe::Stack::Stack): * assembler/ProbeStack.h: * b3/B3FoldPathConstants.cpp: * b3/B3LowerToAir.cpp: * b3/B3MemoryValue.cpp: (JSC::B3::MemoryValue::MemoryValue): * b3/B3Opcode.cpp: * b3/B3Type.h: * b3/B3TypeMap.h: * b3/B3Width.h: * b3/air/AirAllocateRegistersAndStackAndGenerateCode.cpp: (JSC::B3::Air::GenerateAndAllocateRegisters::prepareForGeneration): (JSC::B3::Air::GenerateAndAllocateRegisters::generate): * b3/air/AirAllocateRegistersAndStackAndGenerateCode.h: * b3/air/AirAllocateRegistersByGraphColoring.cpp: * b3/air/AirArg.cpp: * b3/air/AirArg.h: * b3/air/AirCode.h: * b3/air/AirEmitShuffle.cpp: (JSC::B3::Air::emitShuffle): * builtins/BuiltinExecutables.cpp: (JSC::BuiltinExecutables::createExecutable): * bytecode/AccessCase.cpp: * bytecode/AccessCase.h: * bytecode/CallVariant.cpp: (JSC::variantListWithVariant): * bytecode/CodeBlock.cpp: (JSC::CodeBlock::ensureCatchLivenessIsComputedForBytecodeIndex): * bytecode/CodeBlockHash.cpp: (JSC::CodeBlockHash::dump const): * bytecode/StructureStubInfo.cpp: * bytecode/StructureStubInfo.h: * bytecompiler/NodesCodegen.cpp: (JSC::FunctionCallResolveNode::emitBytecode): * bytecompiler/RegisterID.h: (JSC::RegisterID::RegisterID): (JSC::RegisterID::setIndex): * debugger/Debugger.cpp: (JSC::Debugger::removeBreakpoint): * debugger/DebuggerEvalEnabler.h: (JSC::DebuggerEvalEnabler::DebuggerEvalEnabler): (JSC::DebuggerEvalEnabler::~DebuggerEvalEnabler): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::observeTransitions): * dfg/DFGAbstractValue.cpp: * dfg/DFGAbstractValue.h: (JSC::DFG::AbstractValue::merge): (JSC::DFG::AbstractValue::checkConsistency const): (JSC::DFG::AbstractValue::assertIsRegistered const): * dfg/DFGArithMode.h: (JSC::DFG::doesOverflow): * dfg/DFGBasicBlock.cpp: (JSC::DFG::BasicBlock::BasicBlock): * dfg/DFGBasicBlock.h: (JSC::DFG::BasicBlock::didLink): * dfg/DFGCFAPhase.cpp: (JSC::DFG::CFAPhase::performBlockCFA): * dfg/DFGCommon.h: (JSC::DFG::validationEnabled): * dfg/DFGCommonData.cpp: (JSC::DFG::CommonData::finalizeCatchEntrypoints): * dfg/DFGDesiredWatchpoints.h: * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGEdge.h: (JSC::DFG::Edge::makeWord): * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGJITCode.cpp: (JSC::DFG::JITCode::finalizeOSREntrypoints): * dfg/DFGObjectAllocationSinkingPhase.cpp: * dfg/DFGSSAConversionPhase.cpp: (JSC::DFG::SSAConversionPhase::run): * dfg/DFGScoreBoard.h: (JSC::DFG::ScoreBoard::assertClear): * dfg/DFGSlowPathGenerator.h: (JSC::DFG::SlowPathGenerator::generate): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileCurrentBlock): (JSC::DFG::SpeculativeJIT::emitBinarySwitchStringRecurse): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): (JSC::DFG::SpeculativeJIT::compileAllocateNewArrayWithSize): (JSC::DFG::SpeculativeJIT::compileMakeRope): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::fillSpeculateCell): * dfg/DFGStructureAbstractValue.cpp: * dfg/DFGStructureAbstractValue.h: (JSC::DFG::StructureAbstractValue::assertIsRegistered const): * dfg/DFGVarargsForwardingPhase.cpp: * dfg/DFGVirtualRegisterAllocationPhase.cpp: (JSC::DFG::VirtualRegisterAllocationPhase::run): * ftl/FTLLink.cpp: (JSC::FTL::link): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::callPreflight): (JSC::FTL::DFG::LowerDFGToB3::callCheck): (JSC::FTL::DFG::LowerDFGToB3::crash): * ftl/FTLOperations.cpp: (JSC::FTL::operationMaterializeObjectInOSR): * heap/BlockDirectory.cpp: (JSC::BlockDirectory::assertNoUnswept): * heap/GCSegmentedArray.h: (JSC::GCArraySegment::GCArraySegment): * heap/GCSegmentedArrayInlines.h: (JSC::GCSegmentedArray<T>::clear): (JSC::GCSegmentedArray<T>::expand): (JSC::GCSegmentedArray<T>::validatePrevious): * heap/HandleSet.cpp: * heap/HandleSet.h: * heap/Heap.cpp: (JSC::Heap::updateAllocationLimits): * heap/Heap.h: * heap/MarkedBlock.cpp: * heap/MarkedBlock.h: (JSC::MarkedBlock::assertValidCell const): (JSC::MarkedBlock::assertMarksNotStale): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::beginMarking): (JSC::MarkedSpace::endMarking): (JSC::MarkedSpace::assertNoUnswept): * heap/PreciseAllocation.cpp: * heap/PreciseAllocation.h: (JSC::PreciseAllocation::assertValidCell const): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::SlotVisitor): (JSC::SlotVisitor::appendJSCellOrAuxiliary): * heap/SlotVisitor.h: * inspector/InspectorProtocolTypes.h: (Inspector::Protocol::BindingTraits<JSON::ArrayOf<T>>::assertValueHasExpectedType): * inspector/scripts/codegen/generate_cpp_protocol_types_implementation.py: (CppProtocolTypesImplementationGenerator._generate_assertion_for_object_declaration): (CppProtocolTypesImplementationGenerator): (CppProtocolTypesImplementationGenerator._generate_assertion_for_enum): * inspector/scripts/tests/generic/expected/type-requiring-runtime-casts.json-result: * interpreter/FrameTracers.h: (JSC::JITOperationPrologueCallFrameTracer::JITOperationPrologueCallFrameTracer): * interpreter/Interpreter.cpp: (JSC::Interpreter::Interpreter): * interpreter/Interpreter.h: * jit/AssemblyHelpers.cpp: (JSC::AssemblyHelpers::emitStoreStructureWithTypeInfo): * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::prepareCallOperation): * jit/BinarySwitch.cpp: (JSC::BinarySwitch::BinarySwitch): * jit/CCallHelpers.h: (JSC::CCallHelpers::setupStubArgs): * jit/CallFrameShuffler.cpp: (JSC::CallFrameShuffler::emitDeltaCheck): (JSC::CallFrameShuffler::prepareAny): * jit/JIT.cpp: (JSC::JIT::assertStackPointerOffset): (JSC::JIT::compileWithoutLinking): * jit/JITOpcodes.cpp: (JSC::JIT::emitSlow_op_loop_hint): * jit/JITPropertyAccess.cpp: (JSC::JIT::emit_op_get_from_scope): * jit/JITPropertyAccess32_64.cpp: (JSC::JIT::emit_op_get_from_scope): * jit/Repatch.cpp: (JSC::linkPolymorphicCall): * jit/ThunkGenerators.cpp: (JSC::emitPointerValidation): * llint/LLIntData.cpp: (JSC::LLInt::Data::performAssertions): * llint/LLIntOfflineAsmConfig.h: * parser/Lexer.cpp: * parser/Lexer.h: (JSC::isSafeBuiltinIdentifier): (JSC::Lexer<T>::lexExpectIdentifier): * runtime/ArgList.h: (JSC::MarkedArgumentBuffer::setNeedsOverflowCheck): (JSC::MarkedArgumentBuffer::clearNeedsOverflowCheck): * runtime/Butterfly.h: (JSC::ContiguousData::ContiguousData): (JSC::ContiguousData::Data::Data): * runtime/HashMapImpl.h: (JSC::HashMapImpl::checkConsistency const): (JSC::HashMapImpl::assertBufferIsEmpty const): * runtime/JSCellInlines.h: (JSC::JSCell::methodTable const): * runtime/JSFunction.cpp: * runtime/JSFunction.h: (JSC::JSFunction::assertTypeInfoFlagInvariants): * runtime/JSGlobalObject.cpp: (JSC::JSGlobalObject::init): * runtime/JSGlobalObject.h: * runtime/JSObject.cpp: (JSC::JSObject::visitChildren): (JSC::JSFinalObject::visitChildren): * runtime/JSObjectInlines.h: (JSC::JSObject::validatePutOwnDataProperty): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::assertVariableIsInThisObject): * runtime/LiteralParser.cpp: (JSC::LiteralParser<CharType>::Lexer::lex): * runtime/LiteralParser.h: * runtime/Operations.h: (JSC::scribbleFreeCells): * runtime/OptionsList.h: * runtime/VM.cpp: (JSC::VM::computeCanUseJIT): * runtime/VM.h: (JSC::VM::canUseJIT): * runtime/VarOffset.h: (JSC::VarOffset::checkSanity const): * runtime/WeakMapImpl.h: (JSC::WeakMapImpl::checkConsistency const): (JSC::WeakMapImpl::assertBufferIsEmpty const): * wasm/WasmAirIRGenerator.cpp: (JSC::Wasm::AirIRGenerator::validateInst): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::parseAndCompile): * wasm/WasmFunctionParser.h: (JSC::Wasm::FunctionParser::validationFail const): * wasm/WasmLLIntGenerator.cpp: (JSC::Wasm::LLIntGenerator::checkConsistency): * wasm/WasmPlan.cpp: (JSC::Wasm::Plan::tryRemoveContextAndCancelIfLast): * wasm/WasmSectionParser.h: * wasm/WasmSections.h: * wasm/WasmSignatureInlines.h: (JSC::Wasm::SignatureInformation::get): * wasm/WasmWorklist.cpp: (JSC::Wasm::Worklist::enqueue): * wasm/js/JSToWasm.cpp: (JSC::Wasm::createJSToWasmWrapper): * wasm/js/WebAssemblyFunction.cpp: (JSC::WebAssemblyFunction::previousInstanceOffset const): Source/WebCore: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * Modules/fetch/FetchBodySource.cpp: (WebCore::FetchBodySource::close): * Modules/fetch/FetchBodySource.h: * Modules/webdatabase/DatabaseDetails.h: (WebCore::DatabaseDetails::DatabaseDetails): (WebCore::DatabaseDetails::operator=): * Modules/webdatabase/DatabaseTask.cpp: (WebCore::DatabaseTask::performTask): * Modules/webdatabase/DatabaseTask.h: * Modules/webdatabase/DatabaseThread.cpp: (WebCore::DatabaseThread::terminationRequested const): * Modules/webgpu/WHLSL/AST/WHLSLAddressSpace.h: (WebCore::WHLSL::AST::TypeAnnotation::TypeAnnotation): * Modules/webgpu/WHLSL/WHLSLHighZombieFinder.cpp: (WebCore::WHLSL::findHighZombies): * Modules/webgpu/WHLSL/WHLSLInferTypes.cpp: (WebCore::WHLSL::matches): * Modules/webgpu/WHLSL/WHLSLLiteralTypeChecker.cpp: (WebCore::WHLSL::checkLiteralTypes): * Modules/webgpu/WHLSL/WHLSLSynthesizeConstructors.cpp: (WebCore::WHLSL::FindAllTypes::appendNamedType): * bindings/js/JSCallbackData.h: * bindings/js/JSLazyEventListener.cpp: * bindings/js/JSLazyEventListener.h: * contentextensions/ContentExtensionCompiler.cpp: (WebCore::ContentExtensions::compileRuleList): * css/CSSCalculationValue.cpp: (WebCore::CSSCalcOperationNode::primitiveType const): * css/CSSComputedStyleDeclaration.cpp: (WebCore::ComputedStyleExtractor::valueForPropertyInStyle): * css/CSSPrimitiveValue.cpp: * css/CSSSelector.cpp: (WebCore::CSSSelector::selectorText const): * css/CSSStyleSheet.cpp: * dom/ActiveDOMObject.cpp: (WebCore::ActiveDOMObject::suspendIfNeeded): (WebCore::ActiveDOMObject::assertSuspendIfNeededWasCalled const): * dom/ActiveDOMObject.h: * dom/ContainerNode.cpp: * dom/ContainerNodeAlgorithms.cpp: * dom/ContainerNodeAlgorithms.h: * dom/CustomElementReactionQueue.cpp: * dom/CustomElementReactionQueue.h: (WebCore::CustomElementReactionDisallowedScope::CustomElementReactionDisallowedScope): (WebCore::CustomElementReactionDisallowedScope::~CustomElementReactionDisallowedScope): * dom/Document.cpp: (WebCore::Document::hitTest): * dom/Document.h: (WebCore::Document::decrementReferencingNodeCount): * dom/Element.cpp: (WebCore::Element::addShadowRoot): (WebCore::Element::getURLAttribute const): (WebCore::Element::getNonEmptyURLAttribute const): * dom/Element.h: * dom/ElementAndTextDescendantIterator.h: (WebCore::ElementAndTextDescendantIterator::ElementAndTextDescendantIterator): (WebCore::ElementAndTextDescendantIterator::dropAssertions): (WebCore::ElementAndTextDescendantIterator::popAncestorSiblingStack): (WebCore::ElementAndTextDescendantIterator::traverseNextSibling): (WebCore::ElementAndTextDescendantIterator::traversePreviousSibling): * dom/ElementDescendantIterator.h: (WebCore::ElementDescendantIterator::ElementDescendantIterator): (WebCore::ElementDescendantIterator::dropAssertions): (WebCore::ElementDescendantIterator::operator++): (WebCore::ElementDescendantIterator::operator--): (WebCore::ElementDescendantConstIterator::ElementDescendantConstIterator): (WebCore::ElementDescendantConstIterator::dropAssertions): (WebCore::ElementDescendantConstIterator::operator++): * dom/ElementIterator.h: (WebCore::ElementIterator<ElementType>::ElementIterator): (WebCore::ElementIterator<ElementType>::traverseNext): (WebCore::ElementIterator<ElementType>::traversePrevious): (WebCore::ElementIterator<ElementType>::traverseNextSibling): (WebCore::ElementIterator<ElementType>::traversePreviousSibling): (WebCore::ElementIterator<ElementType>::traverseNextSkippingChildren): (WebCore::ElementIterator<ElementType>::dropAssertions): (WebCore::ElementIterator<ElementType>::traverseAncestor): (WebCore::ElementConstIterator<ElementType>::ElementConstIterator): (WebCore::ElementConstIterator<ElementType>::traverseNext): (WebCore::ElementConstIterator<ElementType>::traversePrevious): (WebCore::ElementConstIterator<ElementType>::traverseNextSibling): (WebCore::ElementConstIterator<ElementType>::traversePreviousSibling): (WebCore::ElementConstIterator<ElementType>::traverseNextSkippingChildren): (WebCore::ElementConstIterator<ElementType>::traverseAncestor): (WebCore::ElementConstIterator<ElementType>::dropAssertions): * dom/EventContext.cpp: * dom/EventContext.h: * dom/EventListener.h: * dom/EventPath.cpp: * dom/EventSender.h: * dom/EventTarget.cpp: (WebCore::EventTarget::addEventListener): (WebCore::EventTarget::setAttributeEventListener): (WebCore::EventTarget::innerInvokeEventListeners): * dom/Node.cpp: (WebCore::Node::~Node): (WebCore::Node::moveNodeToNewDocument): (WebCore::Node::removedLastRef): * dom/Node.h: (WebCore::Node::deref const): * dom/ScriptDisallowedScope.h: (WebCore::ScriptDisallowedScope::InMainThread::isEventDispatchAllowedInSubtree): * dom/ScriptExecutionContext.cpp: (WebCore::ScriptExecutionContext::~ScriptExecutionContext): * dom/ScriptExecutionContext.h: * dom/SelectorQuery.cpp: (WebCore::SelectorDataList::execute const): * dom/SlotAssignment.cpp: (WebCore::SlotAssignment::addSlotElementByName): (WebCore::SlotAssignment::removeSlotElementByName): (WebCore::SlotAssignment::resolveSlotsAfterSlotMutation): (WebCore::SlotAssignment::findFirstSlotElement): * dom/SlotAssignment.h: * dom/TreeScopeOrderedMap.cpp: (WebCore::TreeScopeOrderedMap::add): (WebCore::TreeScopeOrderedMap::get const): * dom/TreeScopeOrderedMap.h: * fileapi/Blob.cpp: * fileapi/Blob.h: * history/BackForwardCache.cpp: (WebCore::BackForwardCache::removeAllItemsForPage): * history/BackForwardCache.h: * html/CanvasBase.cpp: (WebCore::CanvasBase::notifyObserversCanvasDestroyed): * html/CanvasBase.h: * html/HTMLCollection.h: (WebCore::CollectionNamedElementCache::didPopulate): * html/HTMLSelectElement.cpp: (WebCore:: const): * html/HTMLTableRowsCollection.cpp: (WebCore::assertRowIsInTable): * html/HTMLTextFormControlElement.cpp: (WebCore::HTMLTextFormControlElement::indexForPosition const): * html/canvas/CanvasRenderingContext2DBase.cpp: (WebCore::CanvasRenderingContext2DBase::~CanvasRenderingContext2DBase): * html/parser/HTMLParserScheduler.cpp: (WebCore::HTMLParserScheduler::HTMLParserScheduler): (WebCore::HTMLParserScheduler::suspend): (WebCore::HTMLParserScheduler::resume): * html/parser/HTMLParserScheduler.h: * html/parser/HTMLToken.h: (WebCore::HTMLToken::beginStartTag): (WebCore::HTMLToken::beginEndTag): (WebCore::HTMLToken::endAttribute): * html/parser/HTMLTreeBuilder.cpp: (WebCore::HTMLTreeBuilder::HTMLTreeBuilder): (WebCore::HTMLTreeBuilder::constructTree): * html/parser/HTMLTreeBuilder.h: (WebCore::HTMLTreeBuilder::~HTMLTreeBuilder): * layout/FormattingContext.cpp: (WebCore::Layout::FormattingContext::geometryForBox const): * layout/blockformatting/BlockFormattingContext.cpp: (WebCore::Layout::BlockFormattingContext::computeEstimatedVerticalPosition): * layout/blockformatting/BlockFormattingContext.h: * layout/displaytree/DisplayBox.cpp: (WebCore::Display::Box::Box): * layout/displaytree/DisplayBox.h: (WebCore::Display::Box::setTopLeft): (WebCore::Display::Box::setTop): (WebCore::Display::Box::setLeft): (WebCore::Display::Box::setContentBoxHeight): (WebCore::Display::Box::setContentBoxWidth): (WebCore::Display::Box::setHorizontalMargin): (WebCore::Display::Box::setVerticalMargin): (WebCore::Display::Box::setHorizontalComputedMargin): (WebCore::Display::Box::setBorder): (WebCore::Display::Box::setPadding): * layout/displaytree/DisplayInlineRect.h: (WebCore::Display::InlineRect::InlineRect): (WebCore::Display::InlineRect::setTopLeft): (WebCore::Display::InlineRect::setTop): (WebCore::Display::InlineRect::setBottom): (WebCore::Display::InlineRect::setLeft): (WebCore::Display::InlineRect::setWidth): (WebCore::Display::InlineRect::setHeight): * layout/displaytree/DisplayLineBox.h: (WebCore::Display::LineBox::LineBox): (WebCore::Display::LineBox::setBaselineOffsetIfGreater): (WebCore::Display::LineBox::resetBaseline): (WebCore::Display::LineBox::Baseline::Baseline): (WebCore::Display::LineBox::Baseline::setAscent): (WebCore::Display::LineBox::Baseline::setDescent): (WebCore::Display::LineBox::Baseline::reset): * layout/displaytree/DisplayRect.h: (WebCore::Display::Rect::Rect): (WebCore::Display::Rect::setTopLeft): (WebCore::Display::Rect::setTop): (WebCore::Display::Rect::setLeft): (WebCore::Display::Rect::setWidth): (WebCore::Display::Rect::setHeight): (WebCore::Display::Rect::setSize): (WebCore::Display::Rect::clone const): * layout/floats/FloatingContext.cpp: * layout/inlineformatting/InlineLineBuilder.cpp: (WebCore::Layout::LineBuilder::CollapsibleContent::collapse): * layout/tableformatting/TableGrid.cpp: (WebCore::Layout::TableGrid::Column::setWidthConstraints): (WebCore::Layout::TableGrid::Column::setLogicalWidth): (WebCore::Layout::TableGrid::Column::setLogicalLeft): * layout/tableformatting/TableGrid.h: * loader/DocumentLoader.cpp: (WebCore::DocumentLoader::continueAfterContentPolicy): (WebCore::DocumentLoader::attachToFrame): (WebCore::DocumentLoader::detachFromFrame): (WebCore::DocumentLoader::addSubresourceLoader): * loader/DocumentLoader.h: * loader/ImageLoader.cpp: * loader/cache/CachedResource.h: * loader/cache/MemoryCache.cpp: (WebCore::MemoryCache::lruListFor): (WebCore::MemoryCache::removeFromLRUList): * page/FrameView.cpp: (WebCore::FrameView::updateLayoutAndStyleIfNeededRecursive): * page/FrameViewLayoutContext.cpp: * page/FrameViewLayoutContext.h: * page/Page.cpp: * page/Page.h: * page/ViewportConfiguration.cpp: * page/ViewportConfiguration.h: * page/mac/EventHandlerMac.mm: (WebCore::CurrentEventScope::CurrentEventScope): * platform/DateComponents.cpp: (WebCore::DateComponents::toStringForTime const): * platform/ScrollableArea.cpp: * platform/SharedBuffer.cpp: (WebCore::SharedBuffer::combineIntoOneSegment const): * platform/SharedBuffer.h: * platform/Supplementable.h: * platform/Timer.cpp: (WebCore::TimerBase::checkHeapIndex const): (WebCore::TimerBase::updateHeapIfNeeded): * platform/graphics/BitmapImage.cpp: * platform/graphics/BitmapImage.h: * platform/graphics/Image.h: * platform/graphics/ShadowBlur.cpp: (WebCore::ScratchBuffer::ScratchBuffer): (WebCore::ScratchBuffer::getScratchBuffer): (WebCore::ScratchBuffer::scheduleScratchBufferPurge): * platform/graphics/ca/win/CACFLayerTreeHost.cpp: (WebCore::CACFLayerTreeHost::setWindow): * platform/graphics/ca/win/CACFLayerTreeHost.h: * platform/graphics/cg/ImageBufferDataCG.cpp: (WebCore::ImageBufferData::putData): * platform/graphics/cocoa/FontCacheCoreText.cpp: * platform/graphics/gstreamer/GstAllocatorFastMalloc.cpp: (gstAllocatorFastMallocFree): * platform/graphics/nicosia/cairo/NicosiaPaintingContextCairo.cpp: (Nicosia::PaintingContextCairo::ForPainting::ForPainting): * platform/graphics/nicosia/texmap/NicosiaBackingStoreTextureMapperImpl.cpp: (Nicosia::BackingStoreTextureMapperImpl::createTile): * platform/graphics/nicosia/texmap/NicosiaContentLayerTextureMapperImpl.cpp: (Nicosia::ContentLayerTextureMapperImpl::~ContentLayerTextureMapperImpl): * platform/graphics/win/GradientDirect2D.cpp: (WebCore::Gradient::fill): * platform/graphics/win/ImageBufferDataDirect2D.cpp: (WebCore::ImageBufferData::putData): * platform/graphics/win/PathDirect2D.cpp: (WebCore::Path::appendGeometry): (WebCore::Path::Path): (WebCore::Path::operator=): (WebCore::Path::strokeContains const): (WebCore::Path::transform): * platform/graphics/win/PlatformContextDirect2D.cpp: (WebCore::PlatformContextDirect2D::setTags): * platform/mediastream/MediaStreamTrackPrivate.h: * platform/mediastream/RealtimeOutgoingAudioSource.cpp: (WebCore::RealtimeOutgoingAudioSource::~RealtimeOutgoingAudioSource): * platform/mediastream/RealtimeOutgoingVideoSource.cpp: (WebCore::RealtimeOutgoingVideoSource::~RealtimeOutgoingVideoSource): * platform/network/HTTPParsers.cpp: (WebCore::isCrossOriginSafeHeader): * platform/sql/SQLiteDatabase.cpp: * platform/sql/SQLiteDatabase.h: * platform/sql/SQLiteStatement.cpp: (WebCore::SQLiteStatement::SQLiteStatement): (WebCore::SQLiteStatement::prepare): (WebCore::SQLiteStatement::finalize): * platform/sql/SQLiteStatement.h: * platform/win/COMPtr.h: * rendering/ComplexLineLayout.cpp: (WebCore::ComplexLineLayout::removeInlineBox const): * rendering/FloatingObjects.cpp: (WebCore::FloatingObject::FloatingObject): (WebCore::FloatingObjects::addPlacedObject): (WebCore::FloatingObjects::removePlacedObject): * rendering/FloatingObjects.h: * rendering/GridTrackSizingAlgorithm.cpp: * rendering/GridTrackSizingAlgorithm.h: * rendering/LayoutDisallowedScope.cpp: * rendering/LayoutDisallowedScope.h: * rendering/RenderBlock.cpp: * rendering/RenderBlock.h: * rendering/RenderBlockFlow.cpp: (WebCore::RenderBlockFlow::layoutBlockChild): (WebCore::RenderBlockFlow::removeFloatingObject): (WebCore::RenderBlockFlow::ensureLineBoxes): * rendering/RenderBoxModelObject.cpp: * rendering/RenderDeprecatedFlexibleBox.cpp: (WebCore::RenderDeprecatedFlexibleBox::layoutBlock): * rendering/RenderElement.cpp: * rendering/RenderGeometryMap.cpp: (WebCore::RenderGeometryMap::mapToContainer const): * rendering/RenderGrid.cpp: (WebCore::RenderGrid::placeItemsOnGrid const): (WebCore::RenderGrid::baselinePosition const): * rendering/RenderInline.cpp: (WebCore::RenderInline::willBeDestroyed): * rendering/RenderLayer.cpp: (WebCore::ClipRectsCache::ClipRectsCache): (WebCore::RenderLayer::RenderLayer): (WebCore::RenderLayer::paintList): (WebCore::RenderLayer::hitTestLayer): (WebCore::RenderLayer::updateClipRects): (WebCore::RenderLayer::calculateClipRects const): * rendering/RenderLayer.h: * rendering/RenderLayerBacking.cpp: (WebCore::traverseVisibleNonCompositedDescendantLayers): * rendering/RenderLayerCompositor.cpp: (WebCore::RenderLayerCompositor::computeCompositingRequirements): (WebCore::RenderLayerCompositor::traverseUnchangedSubtree): (WebCore::RenderLayerCompositor::updateBackingAndHierarchy): (WebCore::RenderLayerCompositor::addDescendantsToOverlapMapRecursive const): (WebCore::RenderLayerCompositor::recursiveRepaintLayer): (WebCore::RenderLayerCompositor::layerHas3DContent const): * rendering/RenderLayoutState.cpp: (WebCore::RenderLayoutState::RenderLayoutState): (WebCore::RenderLayoutState::computeOffsets): (WebCore::RenderLayoutState::addLayoutDelta): * rendering/RenderLayoutState.h: (WebCore::RenderLayoutState::RenderLayoutState): * rendering/RenderObject.cpp: (WebCore::RenderObject::RenderObject): (WebCore::RenderObject::~RenderObject): (WebCore::RenderObject::clearNeedsLayout): * rendering/RenderObject.h: * rendering/RenderQuote.cpp: (WebCore::quotesForLanguage): * rendering/RenderTableCell.h: * rendering/RenderTableSection.cpp: (WebCore::RenderTableSection::computeOverflowFromCells): * rendering/RenderTextLineBoxes.cpp: (WebCore::RenderTextLineBoxes::checkConsistency const): * rendering/RenderTextLineBoxes.h: * rendering/line/BreakingContext.h: (WebCore::tryHyphenating): * rendering/style/GridArea.h: (WebCore::GridSpan::GridSpan): * rendering/style/RenderStyle.cpp: (WebCore::RenderStyle::~RenderStyle): * rendering/style/RenderStyle.h: * rendering/updating/RenderTreeBuilderRuby.cpp: (WebCore::RenderTreeBuilder::Ruby::detach): * rendering/updating/RenderTreePosition.cpp: (WebCore::RenderTreePosition::computeNextSibling): * rendering/updating/RenderTreePosition.h: * svg/SVGToOTFFontConversion.cpp: (WebCore::SVGToOTFFontConverter::Placeholder::Placeholder): (WebCore::SVGToOTFFontConverter::Placeholder::populate): (WebCore::SVGToOTFFontConverter::appendCFFTable): (WebCore::SVGToOTFFontConverter::firstGlyph const): (WebCore::SVGToOTFFontConverter::appendKERNTable): * svg/SVGTransformDistance.cpp: (WebCore::SVGTransformDistance::SVGTransformDistance): (WebCore::SVGTransformDistance::scaledDistance const): (WebCore::SVGTransformDistance::addSVGTransforms): (WebCore::SVGTransformDistance::addToSVGTransform const): (WebCore::SVGTransformDistance::distance const): * svg/graphics/SVGImage.cpp: (WebCore::SVGImage::nativeImage): * testing/InternalSettings.cpp: * workers/service/ServiceWorkerJob.h: * worklets/PaintWorkletGlobalScope.h: (WebCore::PaintWorkletGlobalScope::~PaintWorkletGlobalScope): * xml/XPathStep.cpp: Source/WebKit: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * NetworkProcess/NetworkSession.cpp: (WebKit::NetworkSession::invalidateAndCancel): * NetworkProcess/NetworkSession.h: * NetworkProcess/cache/NetworkCacheStorage.cpp: (WebKit::NetworkCache::Storage::setCapacity): * NetworkProcess/cocoa/NetworkSessionCocoa.mm: (toNSURLSessionResponseDisposition): (WebKit::NetworkSessionCocoa::NetworkSessionCocoa): * Platform/IPC/Connection.cpp: (IPC::Connection::waitForMessage): * Platform/IPC/MessageReceiver.h: (IPC::MessageReceiver::willBeAddedToMessageReceiverMap): (IPC::MessageReceiver::willBeRemovedFromMessageReceiverMap): * Platform/IPC/cocoa/ConnectionCocoa.mm: (IPC::readFromMachPort): * Platform/mac/MachUtilities.cpp: (setMachExceptionPort): * Shared/API/APIClient.h: (API::Client::Client): * Shared/API/Cocoa/WKRemoteObjectCoder.mm: * Shared/Cocoa/ArgumentCodersCocoa.h: * Shared/SharedStringHashTableReadOnly.cpp: * UIProcess/BackingStore.cpp: (WebKit::BackingStore::incorporateUpdate): * UIProcess/GenericCallback.h: * UIProcess/Launcher/mac/ProcessLauncherMac.mm: (WebKit::ProcessLauncher::launchProcess): * UIProcess/PageLoadState.h: (WebKit::PageLoadState::Transaction::Token::Token): * UIProcess/WebPageProxy.cpp: (WebKit::WebPageProxy::~WebPageProxy): * WebProcess/Network/WebResourceLoader.cpp: (WebKit::WebResourceLoader::didReceiveResponse): * WebProcess/Network/WebResourceLoader.h: * WebProcess/Plugins/Netscape/NetscapePluginStream.cpp: (WebKit::NetscapePluginStream::NetscapePluginStream): (WebKit::NetscapePluginStream::notifyAndDestroyStream): * WebProcess/Plugins/Netscape/NetscapePluginStream.h: * WebProcess/WebPage/WebPage.cpp: (WebKit::WebPage::runModal): * WebProcess/WebProcess.cpp: (WebKit::checkDocumentsCaptureStateConsistency): * WebProcess/cocoa/WebProcessCocoa.mm: (WebKit::WebProcess::updateProcessName): Source/WebKitLegacy: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * Storage/StorageAreaImpl.cpp: (WebKit::StorageAreaImpl::StorageAreaImpl): (WebKit::StorageAreaImpl::close): * Storage/StorageAreaImpl.h: Source/WebKitLegacy/mac: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * History/WebHistory.mm: (-[WebHistoryPrivate removeItemForURLString:]): * WebView/WebFrame.mm: Source/WebKitLegacy/win: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. * WebKitQuartzCoreAdditions/CAD3DRenderer.cpp: (WKQCA::CAD3DRenderer::swapChain): (WKQCA::CAD3DRenderer::initialize): * WebKitQuartzCoreAdditions/CAD3DRenderer.h: * WebView.cpp: (WebView::Release): * WebView.h: Source/WTF: Convert ASSERT_DISABLED to ASSERT_ENABLED, and fix some tests of NDEBUG that should actually test for ASSERT_ENABLED. https://bugs.webkit.org/show_bug.cgi?id=205776 Reviewed by Saam Barati. This patch did the following changes: 1. Replaced ASSERT_DISABLED with ASSERT_ENABLED. This change does away with the need for the double negative !ASSERT_DISABLED test that is commonly used all over the code, thereby improving code readability. In Assertions.h, there is also BACKTRACE_DISABLED, ASSERT_MSG_DISABLED, ASSERT_ARG_DISABLED, FATAL_DISABLED, ERROR_DISABLED, LOG_DISABLED, and RELEASE_LOG_DISABLED. We should replace those with ..._ENABLED equivalents as well. We'll do that in another patch. For now, they are left as is to minimize the size of this patch. See https://bugs.webkit.org/show_bug.cgi?id=205780. 2. Fixed some code was guarded with "#ifndef NDEBUG" that should actually be guarded by "#if ASSERT_ENABLED" instead. 3. In cases where the change is minimal, we move some code around so that we can test for "#if ASSERT_ENABLED" instead of "#if !ASSERT_ENABLED". * wtf/Assertions.h: * wtf/AutomaticThread.cpp: (WTF::AutomaticThread::start): * wtf/BitVector.h: * wtf/BlockObjCExceptions.mm: (ReportBlockedObjCException): * wtf/BloomFilter.h: * wtf/CallbackAggregator.h: (WTF::CallbackAggregator::CallbackAggregator): * wtf/CheckedArithmetic.h: (WTF::observesOverflow<AssertNoOverflow>): * wtf/CheckedBoolean.h: (CheckedBoolean::CheckedBoolean): (CheckedBoolean::operator bool): * wtf/CompletionHandler.h: (WTF::CompletionHandler<Out): * wtf/DateMath.cpp: (WTF::initializeDates): * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): * wtf/HashTable.h: (WTF::KeyTraits>::checkKey): (WTF::KeyTraits>::checkTableConsistencyExceptSize const): * wtf/LoggerHelper.h: * wtf/NaturalLoops.h: (WTF::NaturalLoops::headerOf const): * wtf/NeverDestroyed.h: (WTF::LazyNeverDestroyed::construct): * wtf/OptionSet.h: (WTF::OptionSet::OptionSet): * wtf/Platform.h: * wtf/PtrTag.h: * wtf/RefCounted.h: (WTF::RefCountedBase::disableThreadingChecks): (WTF::RefCountedBase::enableThreadingChecksGlobally): (WTF::RefCountedBase::RefCountedBase): (WTF::RefCountedBase::applyRefDerefThreadingCheck const): * wtf/SingleRootGraph.h: (WTF::SingleRootGraph::assertIsConsistent const): * wtf/SizeLimits.cpp: * wtf/StackBounds.h: (WTF::StackBounds::checkConsistency const): * wtf/URLParser.cpp: (WTF::URLParser::URLParser): (WTF::URLParser::domainToASCII): * wtf/ValueCheck.h: * wtf/Vector.h: (WTF::Malloc>::checkConsistency): * wtf/WeakHashSet.h: * wtf/WeakPtr.h: (WTF::WeakPtrImpl::WeakPtrImpl): (WTF::WeakPtrFactory::WeakPtrFactory): * wtf/text/AtomStringImpl.cpp: * wtf/text/AtomStringImpl.h: * wtf/text/StringBuilder.cpp: (WTF::StringBuilder::reifyString const): * wtf/text/StringBuilder.h: * wtf/text/StringCommon.h: (WTF::hasPrefixWithLettersIgnoringASCIICaseCommon): * wtf/text/StringHasher.h: (WTF::StringHasher::addCharacters): * wtf/text/StringImpl.h: * wtf/text/SymbolImpl.h: * wtf/text/UniquedStringImpl.h: Tools: Remove WebsiteDataStore::setServiceWorkerRegistrationDirectory https://bugs.webkit.org/show_bug.cgi?id=205754 Patch by Alex Christensen <achristensen@webkit.org> on 2020-01-06 Reviewed by Youenn Fablet. * TestWebKitAPI/Tests/WebKitCocoa/ServiceWorkerBasic.mm: * WebKitTestRunner/TestController.cpp: (WTR::TestController::websiteDataStore): (WTR::TestController::platformAdjustContext): * WebKitTestRunner/cocoa/TestControllerCocoa.mm: (WTR::initializeWebViewConfiguration): Canonical link: https://commits.webkit.org/218957@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@254087 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-01-06 22:24:50 +00:00
#if ASSERT_ENABLED
Replace tryLargeMemalignVirtual with tryLargeZeroedMemalignVirtual and use it to allocate large zeroed memory in Wasm https://bugs.webkit.org/show_bug.cgi?id=182064 <rdar://problem/36840132> Reviewed by Geoffrey Garen. Source/bmalloc: This patch replaces the tryLargeMemalignVirtual API with tryLargeZeroedMemalignVirtual. By doing that, we're able to remove the AllocationKind enum. To zero the memory, tryLargeZeroedMemalignVirtual uses mmap(... MAP_ANON ...) over previously mmapped memory. This both purges the any resident memory for the virtual range and ensures that the pages in the range are zeroed. Most OSs should implement this by taking a page fault and zero filling on first access. Therefore, this API is returning pages that will result in page faults on first access. Hence, the name 'virtual' in the API. This API differs from the old API in that users of it need not call madvise themselves. The memory is ready to go. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Removed. * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::memalignLarge): (bmalloc::DebugHeap::freeLarge): * bmalloc/DebugHeap.h: * bmalloc/Heap.cpp: (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: * bmalloc/IsoPage.cpp: (bmalloc::IsoPageBase::allocatePageMemory): * bmalloc/VMAllocate.h: (bmalloc::vmZeroAndPurge): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::tryLargeMemalignVirtual): Deleted. * bmalloc/bmalloc.h: Source/JavaScriptCore: This patch switches WebAssembly Memory to always use bmalloc's zeroed virtual allocation API. This makes it so that we don't dirty the memory to zero it. It's a huge compile time speedup on WasmBench on iOS. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::commitZeroPages): Deleted. Source/WTF: * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAllocateVirtualPages): Deleted. * wtf/Gigacage.h: * wtf/OSAllocator.h: Canonical link: https://commits.webkit.org/198171@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@227951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-02-01 05:36:40 +00:00
if (result) {
for (size_t i = 0; i < size / sizeof(uintptr_t); ++i)
ASSERT(static_cast<uintptr_t*>(result)[i] == 0);
}
#endif
return result;
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
}
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void freeVirtualPages(Kind, void* basePtr, size_t size)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Replace tryLargeMemalignVirtual with tryLargeZeroedMemalignVirtual and use it to allocate large zeroed memory in Wasm https://bugs.webkit.org/show_bug.cgi?id=182064 <rdar://problem/36840132> Reviewed by Geoffrey Garen. Source/bmalloc: This patch replaces the tryLargeMemalignVirtual API with tryLargeZeroedMemalignVirtual. By doing that, we're able to remove the AllocationKind enum. To zero the memory, tryLargeZeroedMemalignVirtual uses mmap(... MAP_ANON ...) over previously mmapped memory. This both purges the any resident memory for the virtual range and ensures that the pages in the range are zeroed. Most OSs should implement this by taking a page fault and zero filling on first access. Therefore, this API is returning pages that will result in page faults on first access. Hence, the name 'virtual' in the API. This API differs from the old API in that users of it need not call madvise themselves. The memory is ready to go. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Removed. * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::memalignLarge): (bmalloc::DebugHeap::freeLarge): * bmalloc/DebugHeap.h: * bmalloc/Heap.cpp: (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: * bmalloc/IsoPage.cpp: (bmalloc::IsoPageBase::allocatePageMemory): * bmalloc/VMAllocate.h: (bmalloc::vmZeroAndPurge): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::tryLargeMemalignVirtual): Deleted. * bmalloc/bmalloc.h: Source/JavaScriptCore: This patch switches WebAssembly Memory to always use bmalloc's zeroed virtual allocation API. This makes it so that we don't dirty the memory to zero it. It's a huge compile time speedup on WasmBench on iOS. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::commitZeroPages): Deleted. Source/WTF: * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAllocateVirtualPages): Deleted. * wtf/Gigacage.h: * wtf/OSAllocator.h: Canonical link: https://commits.webkit.org/198171@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@227951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-02-01 05:36:40 +00:00
OSAllocator::decommitAndRelease(basePtr, size);
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
}
} // namespace Gigacage
Gigacages should start allocations from a slide https://bugs.webkit.org/show_bug.cgi?id=193523 Reviewed by Mark Lam. Source/bmalloc: This patch makes it so that Gigacage Heaps slide the start of the cage by some random amount. We still ensure that there is always at least 4/2GB, on MacOS/iOS respectively, of VA space available for allocation. Also, this patch changes some macros into constants since macros are the devil. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::shouldBeEnabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::gigacageSizeToMask): (Gigacage::size): (Gigacage::mask): (Gigacage::basePtr): (Gigacage::ensureGigacage): (Gigacage::wasEnabled): (Gigacage::isCaged): (Gigacage::isEnabled): (Gigacage::caged): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::canPrimitiveGigacageBeDisabled): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): * bmalloc/Sizes.h: (bmalloc::Sizes::maskSizeClass): (bmalloc::Sizes::maskObjectSize): (bmalloc::Sizes::logSizeClass): (bmalloc::Sizes::logObjectSize): (bmalloc::Sizes::sizeClass): (bmalloc::Sizes::objectSize): (bmalloc::Sizes::pageSize): Source/JavaScriptCore: This patch changes some macros into constants since macros are the devil. * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::caged): * llint/LowLevelInterpreter64.asm: Source/WTF: This patch changes some macros into constants since macros are the devil. * wtf/Gigacage.cpp: * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/208097@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@240175 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-01-18 22:48:22 +00:00
#else // defined(USE_SYSTEM_MALLOC) && USE_SYSTEM_MALLOC
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
#include <bmalloc/bmalloc.h>
namespace Gigacage {
// FIXME: Pointers into the primitive gigacage must be scrambled right after being returned from malloc,
// and stay scrambled except just before use.
// https://bugs.webkit.org/show_bug.cgi?id=175035
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void* tryAlignedMalloc(Kind kind, size_t alignment, size_t size)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void* result = bmalloc::api::tryMemalign(alignment, size, bmalloc::heapKind(kind));
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
WTF::compilerFence();
return result;
}
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void alignedFree(Kind kind, void* p)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
if (!p)
return;
RELEASE_ASSERT(isCaged(kind, p));
bmalloc::api::free(p, bmalloc::heapKind(kind));
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
WTF::compilerFence();
}
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void* tryMalloc(Kind kind, size_t size)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void* result = bmalloc::api::tryMalloc(size, bmalloc::heapKind(kind));
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
WTF::compilerFence();
return result;
}
[JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active https://bugs.webkit.org/show_bug.cgi?id=196160 Reviewed by Saam Barati. Source/JavaScriptCore: "realloc" can be effective in terms of peak/current memory footprint when realloc succeeds because, 1. It does not allocate additional memory while expanding a vector 2. It does not deallocate an old memory, just reusing the current memory by expanding, so that memory footprint is tight even before scavenging We found that we can "realloc" large butterflies in certain conditions are met because, 1. If it goes to LargeAllocation, this memory region is never reused until GC sweeps it. 2. Butterflies are owned by owner JSObjects, so we know the lifetime of Butterflies. This patch attempts to use "realloc" onto butterflies if, 1. Butterflies are allocated in LargeAllocation kind 2. Concurrent collector is not active 3. Butterflies do not have property storage The condition (2) is required to avoid deallocating butterflies while the concurrent collector looks into it. The condition (3) is also required to avoid deallocating butterflies while the concurrent compiler looks into it. We also change LargeAllocation mechanism to using "malloc" and "free" instead of "posix_memalign". This allows us to use "realloc" safely in all the platforms. Since LargeAllocation uses alignment to distinguish LargeAllocation and MarkedBlock, we manually adjust 16B alignment by allocating 8B more memory in "malloc". Speedometer2 and JetStream2 are neutral. RAMification shows about 1% progression (even in some of JIT tests). * heap/AlignedMemoryAllocator.h: * heap/CompleteSubspace.cpp: (JSC::CompleteSubspace::tryAllocateSlow): (JSC::CompleteSubspace::reallocateLargeAllocationNonVirtual): * heap/CompleteSubspace.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): (JSC::IsoAlignedMemoryAllocator::tryReallocateMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/LargeAllocation.cpp: (JSC::isAlignedForLargeAllocation): (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::tryReallocate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::destroy): * heap/LargeAllocation.h: (JSC::LargeAllocation::indexInSpace): (JSC::LargeAllocation::setIndexInSpace): (JSC::LargeAllocation::basePointer const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::prepareForConservativeScan): * heap/WeakSet.h: (JSC::WeakSet::isTriviallyDestructible const): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::reallocArrayRightIfPossible): * runtime/JSObject.cpp: (JSC::JSObject::ensureLengthSlow): Source/WTF: * wtf/FastMalloc.h: (WTF::FastMalloc::tryRealloc): * wtf/Gigacage.cpp: (Gigacage::tryRealloc): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/210704@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243688 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-04-01 06:51:11 +00:00
void* tryRealloc(Kind kind, void* pointer, size_t size)
{
void* result = bmalloc::api::tryRealloc(pointer, size, bmalloc::heapKind(kind));
WTF::compilerFence();
return result;
}
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
void free(Kind kind, void* p)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
if (!p)
return;
RELEASE_ASSERT(isCaged(kind, p));
bmalloc::api::free(p, bmalloc::heapKind(kind));
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
WTF::compilerFence();
}
Replace tryLargeMemalignVirtual with tryLargeZeroedMemalignVirtual and use it to allocate large zeroed memory in Wasm https://bugs.webkit.org/show_bug.cgi?id=182064 <rdar://problem/36840132> Reviewed by Geoffrey Garen. Source/bmalloc: This patch replaces the tryLargeMemalignVirtual API with tryLargeZeroedMemalignVirtual. By doing that, we're able to remove the AllocationKind enum. To zero the memory, tryLargeZeroedMemalignVirtual uses mmap(... MAP_ANON ...) over previously mmapped memory. This both purges the any resident memory for the virtual range and ensures that the pages in the range are zeroed. Most OSs should implement this by taking a page fault and zero filling on first access. Therefore, this API is returning pages that will result in page faults on first access. Hence, the name 'virtual' in the API. This API differs from the old API in that users of it need not call madvise themselves. The memory is ready to go. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Removed. * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::memalignLarge): (bmalloc::DebugHeap::freeLarge): * bmalloc/DebugHeap.h: * bmalloc/Heap.cpp: (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: * bmalloc/IsoPage.cpp: (bmalloc::IsoPageBase::allocatePageMemory): * bmalloc/VMAllocate.h: (bmalloc::vmZeroAndPurge): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::tryLargeMemalignVirtual): Deleted. * bmalloc/bmalloc.h: Source/JavaScriptCore: This patch switches WebAssembly Memory to always use bmalloc's zeroed virtual allocation API. This makes it so that we don't dirty the memory to zero it. It's a huge compile time speedup on WasmBench on iOS. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::commitZeroPages): Deleted. Source/WTF: * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAllocateVirtualPages): Deleted. * wtf/Gigacage.h: * wtf/OSAllocator.h: Canonical link: https://commits.webkit.org/198171@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@227951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-02-01 05:36:40 +00:00
void* tryAllocateZeroedVirtualPages(Kind kind, size_t size)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Replace tryLargeMemalignVirtual with tryLargeZeroedMemalignVirtual and use it to allocate large zeroed memory in Wasm https://bugs.webkit.org/show_bug.cgi?id=182064 <rdar://problem/36840132> Reviewed by Geoffrey Garen. Source/bmalloc: This patch replaces the tryLargeMemalignVirtual API with tryLargeZeroedMemalignVirtual. By doing that, we're able to remove the AllocationKind enum. To zero the memory, tryLargeZeroedMemalignVirtual uses mmap(... MAP_ANON ...) over previously mmapped memory. This both purges the any resident memory for the virtual range and ensures that the pages in the range are zeroed. Most OSs should implement this by taking a page fault and zero filling on first access. Therefore, this API is returning pages that will result in page faults on first access. Hence, the name 'virtual' in the API. This API differs from the old API in that users of it need not call madvise themselves. The memory is ready to go. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Removed. * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::memalignLarge): (bmalloc::DebugHeap::freeLarge): * bmalloc/DebugHeap.h: * bmalloc/Heap.cpp: (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: * bmalloc/IsoPage.cpp: (bmalloc::IsoPageBase::allocatePageMemory): * bmalloc/VMAllocate.h: (bmalloc::vmZeroAndPurge): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::tryLargeMemalignVirtual): Deleted. * bmalloc/bmalloc.h: Source/JavaScriptCore: This patch switches WebAssembly Memory to always use bmalloc's zeroed virtual allocation API. This makes it so that we don't dirty the memory to zero it. It's a huge compile time speedup on WasmBench on iOS. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::commitZeroPages): Deleted. Source/WTF: * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAllocateVirtualPages): Deleted. * wtf/Gigacage.h: * wtf/OSAllocator.h: Canonical link: https://commits.webkit.org/198171@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@227951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-02-01 05:36:40 +00:00
void* result = bmalloc::api::tryLargeZeroedMemalignVirtual(WTF::pageSize(), size, bmalloc::heapKind(kind));
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
WTF::compilerFence();
return result;
}
void freeVirtualPages(Kind kind, void* basePtr, size_t size)
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
{
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
if (!basePtr)
return;
RELEASE_ASSERT(isCaged(kind, basePtr));
bmalloc::api::freeLargeVirtual(basePtr, size, bmalloc::heapKind(kind));
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
WTF::compilerFence();
}
} // namespace Gigacage
#endif
Put the ScopedArgumentsTable's ScopeOffset array in some gigacage https://bugs.webkit.org/show_bug.cgi?id=174921 Reviewed by Mark Lam. Source/JavaScriptCore: Uses CagedUniquePtr<> to cage the ScopeOffset array. * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetByValOnScopedArguments): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitScopedArgumentsGetByVal): * runtime/ScopedArgumentsTable.cpp: (JSC::ScopedArgumentsTable::create): (JSC::ScopedArgumentsTable::setLength): * runtime/ScopedArgumentsTable.h: Source/WTF: If you want to std::unique_ptr a class that knows that it should be in the Gigacage, then we would create the Gigacage equivalent of WTF_MAKE_FAST_ALLOCATED and it would just work. But this does not work if we want to std::unique_ptr a primitive type. So, this patch adds a solution for this problem: CagedUniquePtr<>. This will handle allocation (CagedUniquePtr<>::create()) and deallocation (in the style of std::unique_ptr). It has three variants: - Non-array types. - Arrays that don't have destructors. - Arrays that have destructors. Just like std::unique_ptr, the array case is triggered by saying "[]" at the end of the type. Unlike std::unique_ptr and most other smart pointers, the whole point of this smart pointer is to dictate where the thing you're pointing at is allocated. For this reason, it has to know how to do things like the array destructor protocol. So it creates its own: the CagedUniquePtr for arrays with destructors is a fat pointer that remembers the length of the array. CagedUniquePtr<> makes it impossible to leak/release the pointer. This is stricter than what std::unique_ptr does, and is probably appropriate for all of the places where we would use this type. So far, we only use it for ScopedArgumentsTable::m_arguments, but I suspect that it will be useful in other places. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/CagedUniquePtr.h: Added. (WTF::CagedUniquePtr::CagedUniquePtr): (WTF::CagedUniquePtr::create): (WTF::CagedUniquePtr::operator=): (WTF::CagedUniquePtr::~CagedUniquePtr): (WTF::CagedUniquePtr::get const): (WTF::CagedUniquePtr::getMayBeNull const): (WTF::CagedUniquePtr::operator== const): (WTF::CagedUniquePtr::operator!= const): (WTF::CagedUniquePtr::operator bool const): (WTF::CagedUniquePtr::operator* const): (WTF::CagedUniquePtr::operator-> const): (WTF::CagedUniquePtr::operator[] const): (WTF::CagedUniquePtr::destroy): * wtf/Gigacage.cpp: (Gigacage::tryMallocArray): (Gigacage::malloc): (Gigacage::mallocArray): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/192235@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220712 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-14 18:33:16 +00:00
namespace Gigacage {
void* tryMallocArray(Kind kind, size_t numElements, size_t elementSize)
{
CheckedSize checkedSize = elementSize;
Put the ScopedArgumentsTable's ScopeOffset array in some gigacage https://bugs.webkit.org/show_bug.cgi?id=174921 Reviewed by Mark Lam. Source/JavaScriptCore: Uses CagedUniquePtr<> to cage the ScopeOffset array. * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetByValOnScopedArguments): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitScopedArgumentsGetByVal): * runtime/ScopedArgumentsTable.cpp: (JSC::ScopedArgumentsTable::create): (JSC::ScopedArgumentsTable::setLength): * runtime/ScopedArgumentsTable.h: Source/WTF: If you want to std::unique_ptr a class that knows that it should be in the Gigacage, then we would create the Gigacage equivalent of WTF_MAKE_FAST_ALLOCATED and it would just work. But this does not work if we want to std::unique_ptr a primitive type. So, this patch adds a solution for this problem: CagedUniquePtr<>. This will handle allocation (CagedUniquePtr<>::create()) and deallocation (in the style of std::unique_ptr). It has three variants: - Non-array types. - Arrays that don't have destructors. - Arrays that have destructors. Just like std::unique_ptr, the array case is triggered by saying "[]" at the end of the type. Unlike std::unique_ptr and most other smart pointers, the whole point of this smart pointer is to dictate where the thing you're pointing at is allocated. For this reason, it has to know how to do things like the array destructor protocol. So it creates its own: the CagedUniquePtr for arrays with destructors is a fat pointer that remembers the length of the array. CagedUniquePtr<> makes it impossible to leak/release the pointer. This is stricter than what std::unique_ptr does, and is probably appropriate for all of the places where we would use this type. So far, we only use it for ScopedArgumentsTable::m_arguments, but I suspect that it will be useful in other places. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/CagedUniquePtr.h: Added. (WTF::CagedUniquePtr::CagedUniquePtr): (WTF::CagedUniquePtr::create): (WTF::CagedUniquePtr::operator=): (WTF::CagedUniquePtr::~CagedUniquePtr): (WTF::CagedUniquePtr::get const): (WTF::CagedUniquePtr::getMayBeNull const): (WTF::CagedUniquePtr::operator== const): (WTF::CagedUniquePtr::operator!= const): (WTF::CagedUniquePtr::operator bool const): (WTF::CagedUniquePtr::operator* const): (WTF::CagedUniquePtr::operator-> const): (WTF::CagedUniquePtr::operator[] const): (WTF::CagedUniquePtr::destroy): * wtf/Gigacage.cpp: (Gigacage::tryMallocArray): (Gigacage::malloc): (Gigacage::mallocArray): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/192235@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220712 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-14 18:33:16 +00:00
checkedSize *= numElements;
if (checkedSize.hasOverflowed())
return nullptr;
Rename Checked::unsafeGet() to Checked::value() https://bugs.webkit.org/show_bug.cgi?id=226514 Reviewed by Darin Adler. Rename Checked::unsafeGet() to Checked::value(). The "unsafeGet" naming is confusing as this function isn't really unsafe since it will crash if the value has overflowed. Also add an `operator T()` to implicitly convert a Checked to its underlying type without needing to call value(). Source/JavaScriptCore: * b3/B3Const32Value.cpp: (JSC::B3::Const32Value::checkAddConstant const): (JSC::B3::Const32Value::checkSubConstant const): (JSC::B3::Const32Value::checkMulConstant const): * b3/B3Const64Value.cpp: (JSC::B3::Const64Value::checkAddConstant const): (JSC::B3::Const64Value::checkSubConstant const): (JSC::B3::Const64Value::checkMulConstant const): * bytecompiler/BytecodeGenerator.h: (JSC::FinallyContext::numberOfBreaksOrContinues const): * dfg/DFGConstantFoldingPhase.cpp: (JSC::DFG::ConstantFoldingPhase::foldConstants): * dfg/DFGOperations.cpp: (JSC::DFG::JSC_DEFINE_JIT_OPERATION): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSpread): (JSC::FTL::DFG::LowerDFGToB3::compileSpread): (JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargsSpread): (JSC::FTL::DFG::LowerDFGToB3::compileForwardVarargsWithSpread): * ftl/FTLOperations.cpp: (JSC::FTL::JSC_DEFINE_JIT_OPERATION): * heap/Heap.cpp: (JSC::Heap::deprecatedReportExtraMemorySlowCase): (JSC::Heap::extraMemorySize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::reportExtraMemoryVisited): * heap/SlotVisitor.cpp: (JSC::SlotVisitor::propagateExternalMemoryVisitedIfNecessary): * runtime/ArgList.cpp: (JSC::MarkedArgumentBuffer::slowEnsureCapacity): (JSC::MarkedArgumentBuffer::expandCapacity): * runtime/ArrayPrototype.cpp: (JSC::concatAppendOne): (JSC::JSC_DEFINE_HOST_FUNCTION): * runtime/CommonSlowPaths.cpp: (JSC::JSC_DEFINE_COMMON_SLOW_PATH): * runtime/DirectArguments.h: * runtime/HashMapImpl.h: (JSC::HashMapBuffer::allocationSize): (JSC::HashMapImpl::HashMapImpl): * runtime/HashMapImplInlines.h: (JSC::nextCapacity): (JSC::HashMapImpl<HashMapBucketType>::finishCreation): * runtime/JSBigInt.cpp: (JSC::JSBigInt::parseInt): * runtime/JSImmutableButterfly.h: (JSC::JSImmutableButterfly::tryCreate): * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::offsetOfVariable): (JSC::JSLexicalEnvironment::allocationSizeForScopeSize): * runtime/JSObject.h: * runtime/JSPropertyNameEnumerator.cpp: (JSC::JSPropertyNameEnumerator::create): * runtime/JSString.h: * runtime/ScopedArguments.cpp: (JSC::ScopedArguments::createUninitialized): * runtime/StringPrototype.cpp: (JSC::jsSpliceSubstrings): (JSC::jsSpliceSubstringsWithSeparators): * runtime/StructureChain.cpp: (JSC::StructureChain::create): * runtime/VM.h: (JSC::ScratchBuffer::allocationSize): * runtime/WeakMapImpl.h: (JSC::WeakMapBuffer::allocationSize): * wasm/WasmAirIRGenerator.cpp: (JSC::Wasm::AirIRGenerator::AirIRGenerator): (JSC::Wasm::AirIRGenerator::emitCallPatchpoint): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): * wasm/WasmInstance.cpp: * wasm/WasmInstance.h: (JSC::Wasm::Instance::allocationSize): * wasm/WasmLLIntGenerator.cpp: (JSC::Wasm::LLIntGenerator::push): (JSC::Wasm::LLIntGenerator::getDropKeepCount): (JSC::Wasm::LLIntGenerator::walkExpressionStack): (JSC::Wasm::LLIntGenerator::finalize): (JSC::Wasm::LLIntGenerator::callInformationForCaller): (JSC::Wasm::LLIntGenerator::addLoop): (JSC::Wasm::LLIntGenerator::addTopLevel): (JSC::Wasm::LLIntGenerator::addBlock): (JSC::Wasm::LLIntGenerator::addIf): (JSC::Wasm::LLIntGenerator::addElseToUnreachable): * wasm/WasmSignature.h: (JSC::Wasm::Signature::allocatedSize): * wasm/WasmStreamingParser.cpp: (JSC::Wasm::StreamingParser::addBytes): * wasm/WasmTable.cpp: (JSC::Wasm::Table::Table): (JSC::Wasm::Table::grow): (JSC::Wasm::FuncRefTable::FuncRefTable): * yarr/YarrInterpreter.cpp: (JSC::Yarr::Interpreter::DisjunctionContext::allocationSize): (JSC::Yarr::Interpreter::ParenthesesDisjunctionContext::allocationSize): (JSC::Yarr::Interpreter::allocParenthesesDisjunctionContext): (JSC::Yarr::ByteCompiler::atomCharacterClass): (JSC::Yarr::ByteCompiler::atomBackReference): (JSC::Yarr::ByteCompiler::atomParentheticalAssertionEnd): (JSC::Yarr::ByteCompiler::atomParenthesesSubpatternEnd): (JSC::Yarr::ByteCompiler::atomParenthesesOnceEnd): (JSC::Yarr::ByteCompiler::atomParenthesesTerminalEnd): (JSC::Yarr::ByteCompiler::emitDisjunction): * yarr/YarrInterpreter.h: (JSC::Yarr::ByteTerm::ByteTerm): (JSC::Yarr::ByteTerm::CheckInput): (JSC::Yarr::ByteTerm::UncheckInput): * yarr/YarrJIT.cpp: * yarr/YarrParser.h: (JSC::Yarr::Parser::consumeNumber): * yarr/YarrPattern.cpp: (JSC::Yarr::PatternTerm::dumpQuantifier): Source/WebCore: * bindings/js/SerializedScriptValue.cpp: (WebCore::CloneDeserializer::readTerminal): * dom/TextEncoderStreamEncoder.cpp: (WebCore::TextEncoderStreamEncoder::encode): * editing/markup.cpp: (WebCore::StyledMarkupAccumulator::takeResults): * html/FileInputType.cpp: (WebCore::FileInputType::saveFormControlState const): * html/ImageData.cpp: (WebCore::ImageData::create): (WebCore::ImageData::createUninitialized): * html/MediaElementSession.cpp: (WebCore::isElementRectMostlyInMainFrame): * html/canvas/WebGL2RenderingContext.cpp: (WebCore::WebGL2RenderingContext::sliceArrayBufferView): (WebCore::WebGL2RenderingContext::copyBufferSubData): (WebCore::WebGL2RenderingContext::getBufferSubData): (WebCore::WebGL2RenderingContext::validateClearBuffer): * html/canvas/WebGLBuffer.cpp: (WebCore::WebGLBuffer::associateBufferSubDataImpl): (WebCore::WebGLBuffer::associateCopyBufferSubData): * html/canvas/WebGLRenderingContextBase.cpp: (WebCore::clip2D): (WebCore::WebGLRenderingContextBase::validateDrawArrays): (WebCore::WebGLRenderingContextBase::validateDrawElements): (WebCore::WebGLRenderingContextBase::validateTexFuncData): (WebCore::WebGLRenderingContextBase::validateCompressedTexFuncData): (WebCore::WebGLRenderingContextBase::validateSimulatedVertexAttrib0): * html/canvas/WebGLRenderingContextBase.h: (WebCore::WebGLRenderingContextBase::validateTexImageSubRectangle): (WebCore::WebGLRenderingContextBase::checkedAddAndMultiply): * page/FrameView.h: (WebCore::FrameView::incrementVisuallyNonEmptyPixelCount): * page/History.cpp: (WebCore::History::stateObjectAdded): * platform/audio/AudioArray.h: (WebCore::AudioArray::resize): * platform/audio/cocoa/AudioFileReaderCocoa.cpp: (WebCore::tryCreateAudioBufferList): * platform/audio/cocoa/CARingBuffer.cpp: (WebCore::CARingBuffer::adoptStorage): (WebCore::CARingBuffer::initializeAfterAllocation): (WebCore::CARingBuffer::allocate): * platform/audio/cocoa/WebAudioBufferList.cpp: (WebCore::WebAudioBufferList::WebAudioBufferList): * platform/graphics/FormatConverter.h: (WebCore::FormatConverter::FormatConverter): * platform/graphics/GraphicsContextGL.cpp: (WebCore::GraphicsContextGL::computeImageSizeInBytes): * platform/graphics/ImageBackingStore.h: (WebCore::ImageBackingStore::setSize): (WebCore::ImageBackingStore::clear): * platform/graphics/ImageBufferBackend.cpp: (WebCore::ImageBufferBackend::calculateMemoryCost): * platform/graphics/ImageFrame.h: (WebCore::ImageFrame::frameBytes const): * platform/graphics/ImageSource.cpp: (WebCore::ImageSource::maximumSubsamplingLevel): * platform/graphics/PixelBuffer.cpp: (WebCore::PixelBuffer::tryCreateForDecoding): (WebCore::PixelBuffer::tryCreate): * platform/graphics/PixelBuffer.h: (WebCore::PixelBuffer::encode const): (WebCore::PixelBuffer::decode): * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm: (WebCore::ImageDecoderAVFObjC::frameBytesAtIndex const): * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: (WebCore::MediaSampleAVFObjC::setByteRangeOffset): (WebCore::MediaSampleAVFObjC::byteRangeForAttachment const): * platform/graphics/ca/GraphicsLayerCA.cpp: (WebCore::GraphicsLayerCA::updateBackdropFilters): * platform/graphics/ca/LayerPool.cpp: (WebCore::LayerPool::backingStoreBytesForSize): * platform/graphics/cg/GraphicsContextGLCG.cpp: (WebCore::GraphicsContextGLImageExtractor::extractImage): * platform/graphics/cg/ImageBufferCGBackend.cpp: (WebCore::ImageBufferCGBackend::calculateBytesPerRow): * platform/graphics/cg/ImageDecoderCG.cpp: (WebCore::ImageDecoderCG::frameBytesAtIndex const): * platform/graphics/cocoa/SourceBufferParser.cpp: (WebCore::SourceBufferParser::Segment::read const): * platform/graphics/filters/FEColorMatrix.cpp: (WebCore::effectApplyAccelerated): * platform/graphics/filters/FEGaussianBlur.cpp: (WebCore::FEGaussianBlur::platformApplySoftware): * platform/graphics/filters/FETurbulence.cpp: (WebCore::FETurbulence::platformApplySoftware): * platform/graphics/filters/FilterEffect.cpp: (WebCore::FilterEffect::unmultipliedResult): (WebCore::FilterEffect::premultipliedResult): (WebCore::copyPremultiplyingAlpha): (WebCore::copyUnpremultiplyingAlpha): * platform/graphics/gpu/cocoa/GPUBindGroupAllocatorMetal.mm: (WebCore::GPUBindGroupAllocator::allocateAndSetEncoders): (WebCore::GPUBindGroupAllocator::reallocate): * platform/graphics/gpu/cocoa/GPUCommandBufferMetal.mm: (WebCore::GPUCommandBuffer::copyBufferToBuffer): * platform/graphics/gpu/cocoa/GPURenderPassEncoderMetal.mm: (WebCore::GPURenderPassEncoder::drawIndexed): * platform/graphics/gstreamer/ImageDecoderGStreamer.cpp: (WebCore::ImageDecoderGStreamer::frameBytesAtIndex const): * platform/graphics/nicosia/NicosiaBuffer.cpp: (Nicosia::Buffer::Buffer): * platform/graphics/win/Direct2DUtilities.cpp: (WebCore::Direct2D::createDirect2DImageSurfaceWithData): * platform/graphics/win/ImageBufferDirect2DBackend.cpp: (WebCore::ImageBufferDirect2DBackend::compatibleBitmap): * platform/graphics/win/ImageDecoderDirect2D.cpp: (WebCore::ImageDecoderDirect2D::frameBytesAtIndex const): * platform/image-decoders/ScalableImageDecoder.cpp: (WebCore::ScalableImageDecoder::frameBytesAtIndex const): * platform/image-decoders/jpeg2000/JPEG2000ImageDecoder.cpp: (WebCore::sycc444ToRGB): (WebCore::sycc422ToRGB): (WebCore::sycc420ToRGB): * platform/ios/LegacyTileLayerPool.mm: (WebCore::LegacyTileLayerPool::bytesBackingLayerWithPixelSize): * platform/text/TextCodecUTF16.cpp: (WebCore::TextCodecUTF16::encode const): * platform/text/TextCodecUTF8.cpp: (WebCore::TextCodecUTF8::encodeUTF8): * rendering/RenderLayerCompositor.cpp: (WebCore::RenderLayerCompositor::requiresCompositingForCanvas const): * rendering/shapes/Shape.cpp: (WebCore::Shape::createRasterShape): * storage/StorageMap.cpp: (WebCore::StorageMap::setItem): * xml/XSLStyleSheetLibxslt.cpp: (WebCore::XSLStyleSheet::parseString): * xml/XSLTProcessorLibxslt.cpp: (WebCore::xsltParamArrayFromParameterMap): * xml/parser/CharacterReferenceParserInlines.h: (WebCore::consumeCharacterReference): Source/WebKit: * GPUProcess/graphics/RemoteRenderingBackend.cpp: (WebKit::RemoteRenderingBackend::nextDestinationImageBufferAfterApplyingDisplayLists): * NetworkProcess/WebStorage/LocalStorageDatabase.cpp: (WebKit::LocalStorageDatabase::setItem): * NetworkProcess/cache/CacheStorageEngineCache.cpp: (WebKit::CacheStorage::Cache::put): * Platform/IPC/ArgumentCoders.h: * Platform/IPC/cocoa/ConnectionCocoa.mm: (IPC::Connection::sendOutgoingMessage): (IPC::createMessageDecoder): * Platform/IPC/cocoa/MachMessage.cpp: (IPC::MachMessage::create): * Shared/ShareableBitmap.cpp: (WebKit::ShareableBitmap::Handle::encode const): (WebKit::ShareableBitmap::create): (WebKit::ShareableBitmap::createShareable): * Shared/ShareableBitmap.h: (WebKit::ShareableBitmap::bytesPerRow const): (WebKit::ShareableBitmap::sizeInBytes const): * Shared/ShareableResource.cpp: (WebKit::ShareableResource::create): * Shared/cg/ShareableBitmapCG.cpp: (WebKit::ShareableBitmap::calculateBytesPerRow): (WebKit::ShareableBitmap::createGraphicsContext): (WebKit::ShareableBitmap::createCGImage const): * Shared/mac/MediaFormatReader/MediaFormatReader.cpp: (WebKit::MediaFormatReader::copyTrackArray): * Shared/mac/MediaFormatReader/MediaSampleCursor.cpp: (WebKit::MediaSampleCursor::copySampleLocation const): * WebProcess/GPU/graphics/DisplayListWriterHandle.cpp: (WebKit::DisplayListWriterHandle::advance): * WebProcess/GPU/graphics/ImageBufferShareableBitmapBackend.cpp: (WebKit::ImageBufferShareableBitmapBackend::calculateBytesPerRow): * WebProcess/GPU/media/RemoteImageDecoderAVF.cpp: (WebKit::RemoteImageDecoderAVF::frameBytesAtIndex const): * WebProcess/Network/WebSocketChannel.cpp: (WebKit::WebSocketChannel::increaseBufferedAmount): * WebProcess/WebPage/ios/WebPageIOS.mm: (WebKit::WebPage::requestEvasionRectsAboveSelection): (WebKit::WebPage::updateSelectionWithDelta): Source/WTF: * wtf/CheckedArithmetic.h: (WTF::Checked::operator! const): (WTF::Checked::operator bool const): (WTF::Checked::operator T const): (WTF::Checked::value const): (WTF::Checked::operator==): (WTF::Checked::operator< const): (WTF::Checked::operator<= const): (WTF::Checked::operator> const): (WTF::Checked::operator>= const): * wtf/ConcurrentBuffer.h: * wtf/FastMalloc.cpp: (WTF::fastCalloc): (WTF::tryFastCalloc): * wtf/Gigacage.cpp: (Gigacage::tryMallocArray): * wtf/URLHelpers.cpp: (WTF::URLHelpers::userVisibleURL): * wtf/URLParser.cpp: (WTF::URLParser::parseIPv4Piece): * wtf/UniqueArray.h: * wtf/cocoa/NSURLExtras.mm: (WTF::dataWithUserTypedString): * wtf/glib/SocketConnection.cpp: (WTF::SocketConnection::readMessage): (WTF::SocketConnection::sendMessage): * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): * wtf/text/StringBuilderJSON.cpp: (WTF::StringBuilder::appendQuotedJSONString): * wtf/text/StringConcatenate.h: (WTF::tryMakeStringFromAdapters): * wtf/text/StringImpl.h: (WTF::StringImpl::allocationSize): * wtf/text/StringToIntegerConversion.h: (WTF::parseInteger): Tools: * TestWebKitAPI/Tests/WTF/CheckedArithmeticOperations.cpp: (TestWebKitAPI::CheckedArithmeticTester::run): (TestWebKitAPI::AllowMixedSignednessTest::run): (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WebCore/IntRectTests.cpp: (TestWebKitAPI::TEST): * TestWebKitAPI/Tests/WebCore/IntSizeTests.cpp: (TestWebKitAPI::TEST): Canonical link: https://commits.webkit.org/238371@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@278338 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2021-06-02 05:21:13 +00:00
return tryMalloc(kind, checkedSize);
Put the ScopedArgumentsTable's ScopeOffset array in some gigacage https://bugs.webkit.org/show_bug.cgi?id=174921 Reviewed by Mark Lam. Source/JavaScriptCore: Uses CagedUniquePtr<> to cage the ScopeOffset array. * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetByValOnScopedArguments): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitScopedArgumentsGetByVal): * runtime/ScopedArgumentsTable.cpp: (JSC::ScopedArgumentsTable::create): (JSC::ScopedArgumentsTable::setLength): * runtime/ScopedArgumentsTable.h: Source/WTF: If you want to std::unique_ptr a class that knows that it should be in the Gigacage, then we would create the Gigacage equivalent of WTF_MAKE_FAST_ALLOCATED and it would just work. But this does not work if we want to std::unique_ptr a primitive type. So, this patch adds a solution for this problem: CagedUniquePtr<>. This will handle allocation (CagedUniquePtr<>::create()) and deallocation (in the style of std::unique_ptr). It has three variants: - Non-array types. - Arrays that don't have destructors. - Arrays that have destructors. Just like std::unique_ptr, the array case is triggered by saying "[]" at the end of the type. Unlike std::unique_ptr and most other smart pointers, the whole point of this smart pointer is to dictate where the thing you're pointing at is allocated. For this reason, it has to know how to do things like the array destructor protocol. So it creates its own: the CagedUniquePtr for arrays with destructors is a fat pointer that remembers the length of the array. CagedUniquePtr<> makes it impossible to leak/release the pointer. This is stricter than what std::unique_ptr does, and is probably appropriate for all of the places where we would use this type. So far, we only use it for ScopedArgumentsTable::m_arguments, but I suspect that it will be useful in other places. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/CagedUniquePtr.h: Added. (WTF::CagedUniquePtr::CagedUniquePtr): (WTF::CagedUniquePtr::create): (WTF::CagedUniquePtr::operator=): (WTF::CagedUniquePtr::~CagedUniquePtr): (WTF::CagedUniquePtr::get const): (WTF::CagedUniquePtr::getMayBeNull const): (WTF::CagedUniquePtr::operator== const): (WTF::CagedUniquePtr::operator!= const): (WTF::CagedUniquePtr::operator bool const): (WTF::CagedUniquePtr::operator* const): (WTF::CagedUniquePtr::operator-> const): (WTF::CagedUniquePtr::operator[] const): (WTF::CagedUniquePtr::destroy): * wtf/Gigacage.cpp: (Gigacage::tryMallocArray): (Gigacage::malloc): (Gigacage::mallocArray): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/192235@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220712 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-14 18:33:16 +00:00
}
void* malloc(Kind kind, size_t size)
{
void* result = tryMalloc(kind, size);
RELEASE_ASSERT(result);
return result;
}
void* mallocArray(Kind kind, size_t numElements, size_t elementSize)
{
void* result = tryMallocArray(kind, numElements, elementSize);
RELEASE_ASSERT(result);
return result;
}
} // namespace Gigacage