haikuwebkit/Source/WTF/wtf/Gigacage.h

123 lines
4.2 KiB
C
Raw Permalink Normal View History

Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
/*
* Copyright (C) 2017-2021 Apple Inc. All rights reserved.
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
#include <wtf/FastMalloc.h>
Gigacages should start allocations from a slide https://bugs.webkit.org/show_bug.cgi?id=193523 Reviewed by Mark Lam. Source/bmalloc: This patch makes it so that Gigacage Heaps slide the start of the cage by some random amount. We still ensure that there is always at least 4/2GB, on MacOS/iOS respectively, of VA space available for allocation. Also, this patch changes some macros into constants since macros are the devil. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::shouldBeEnabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::gigacageSizeToMask): (Gigacage::size): (Gigacage::mask): (Gigacage::basePtr): (Gigacage::ensureGigacage): (Gigacage::wasEnabled): (Gigacage::isCaged): (Gigacage::isEnabled): (Gigacage::caged): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::canPrimitiveGigacageBeDisabled): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): * bmalloc/Sizes.h: (bmalloc::Sizes::maskSizeClass): (bmalloc::Sizes::maskObjectSize): (bmalloc::Sizes::logSizeClass): (bmalloc::Sizes::logObjectSize): (bmalloc::Sizes::sizeClass): (bmalloc::Sizes::objectSize): (bmalloc::Sizes::pageSize): Source/JavaScriptCore: This patch changes some macros into constants since macros are the devil. * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::caged): * llint/LowLevelInterpreter64.asm: Source/WTF: This patch changes some macros into constants since macros are the devil. * wtf/Gigacage.cpp: * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/208097@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@240175 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-01-18 22:48:22 +00:00
#include <wtf/StdLibExtras.h>
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
#if defined(USE_SYSTEM_MALLOC) && USE_SYSTEM_MALLOC
#define GIGACAGE_ENABLED 0
Gigacages should start allocations from a slide https://bugs.webkit.org/show_bug.cgi?id=193523 Reviewed by Mark Lam. Source/bmalloc: This patch makes it so that Gigacage Heaps slide the start of the cage by some random amount. We still ensure that there is always at least 4/2GB, on MacOS/iOS respectively, of VA space available for allocation. Also, this patch changes some macros into constants since macros are the devil. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::shouldBeEnabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::gigacageSizeToMask): (Gigacage::size): (Gigacage::mask): (Gigacage::basePtr): (Gigacage::ensureGigacage): (Gigacage::wasEnabled): (Gigacage::isCaged): (Gigacage::caged): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): * bmalloc/Sizes.h: (bmalloc::Sizes::maskSizeClass): (bmalloc::Sizes::maskObjectSize): (bmalloc::Sizes::logSizeClass): (bmalloc::Sizes::logObjectSize): (bmalloc::Sizes::sizeClass): (bmalloc::Sizes::objectSize): (bmalloc::Sizes::pageSize): Source/JavaScriptCore: This patch changes some macros into constants since macros are the devil. * llint/LowLevelInterpreter64.asm: Source/WTF: This patch changes some macros into constants since macros are the devil. * wtf/Gigacage.cpp: * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/208086@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@240160 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-01-18 19:38:56 +00:00
namespace Gigacage {
Gigacages should start allocations from a slide https://bugs.webkit.org/show_bug.cgi?id=193523 Reviewed by Mark Lam. Source/bmalloc: This patch makes it so that Gigacage Heaps slide the start of the cage by some random amount. We still ensure that there is always at least 4/2GB, on MacOS/iOS respectively, of VA space available for allocation. Also, this patch changes some macros into constants since macros are the devil. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::shouldBeEnabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::gigacageSizeToMask): (Gigacage::size): (Gigacage::mask): (Gigacage::basePtr): (Gigacage::ensureGigacage): (Gigacage::wasEnabled): (Gigacage::isCaged): (Gigacage::caged): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): * bmalloc/Sizes.h: (bmalloc::Sizes::maskSizeClass): (bmalloc::Sizes::maskObjectSize): (bmalloc::Sizes::logSizeClass): (bmalloc::Sizes::logObjectSize): (bmalloc::Sizes::sizeClass): (bmalloc::Sizes::objectSize): (bmalloc::Sizes::pageSize): Source/JavaScriptCore: This patch changes some macros into constants since macros are the devil. * llint/LowLevelInterpreter64.asm: Source/WTF: This patch changes some macros into constants since macros are the devil. * wtf/Gigacage.cpp: * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/208086@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@240160 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-01-18 19:38:56 +00:00
constexpr bool hasCapacityToUseLargeGigacage = OS_CONSTANT(EFFECTIVE_ADDRESS_WIDTH) > 36;
Gigacages should start allocations from a slide https://bugs.webkit.org/show_bug.cgi?id=193523 Reviewed by Mark Lam. Source/bmalloc: This patch makes it so that Gigacage Heaps slide the start of the cage by some random amount. We still ensure that there is always at least 4/2GB, on MacOS/iOS respectively, of VA space available for allocation. Also, this patch changes some macros into constants since macros are the devil. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::shouldBeEnabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::gigacageSizeToMask): (Gigacage::size): (Gigacage::mask): (Gigacage::basePtr): (Gigacage::ensureGigacage): (Gigacage::wasEnabled): (Gigacage::isCaged): (Gigacage::isEnabled): (Gigacage::caged): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::canPrimitiveGigacageBeDisabled): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): * bmalloc/Sizes.h: (bmalloc::Sizes::maskSizeClass): (bmalloc::Sizes::maskObjectSize): (bmalloc::Sizes::logSizeClass): (bmalloc::Sizes::logObjectSize): (bmalloc::Sizes::sizeClass): (bmalloc::Sizes::objectSize): (bmalloc::Sizes::pageSize): Source/JavaScriptCore: This patch changes some macros into constants since macros are the devil. * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::caged): * llint/LowLevelInterpreter64.asm: Source/WTF: This patch changes some macros into constants since macros are the devil. * wtf/Gigacage.cpp: * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/208097@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@240175 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-01-18 22:48:22 +00:00
const size_t primitiveGigacageMask = 0;
const size_t jsValueGigacageMask = 0;
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
enum Kind {
Primitive,
JSTests: Unreviewed, skipping slow tests. These tests are now timing out. They would have always been slow. The timeouts are probably because OOMs work differently now. * stress/regexp-prototype-exec-on-too-long-rope.js: * stress/string-prototype-charCodeAt-on-too-long-rope.js: Source/bmalloc: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. This adds a StringGigacage. * bmalloc/Gigacage.cpp: * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): Source/JavaScriptCore: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. * runtime/JSString.cpp: (JSC::JSRopeString::resolveRopeToAtomicString const): (JSC::JSRopeString::resolveRope const): * runtime/JSString.h: (JSC::JSString::create): (JSC::JSString::createHasOtherOwner): * runtime/JSStringBuilder.h: * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): Source/WebCore: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. No new tests because no new behavior. * html/canvas/CanvasRenderingContext2D.cpp: (WebCore::normalizeSpaces): Source/WTF: Strings need to be in some kind of gigacage https://bugs.webkit.org/show_bug.cgi?id=174924 Reviewed by Oliver Hunt. This makes all strings allocations come from the string gigacage. Because we expect string allocation to be a hot path, I created specialized allocation paths for the string gigacage. These paths are accessible via <wtf/text/StringMalloc.h>. However, those paths are equivalent to saying Gigacage::malloc and friends with the Gigacage::String kind. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/Deque.h: * wtf/FastMalloc.cpp: (WTF::fastFree): * wtf/FastMalloc.h: (WTF::FastMalloc::malloc): (WTF::FastMalloc::tryMalloc): (WTF::FastMalloc::realloc): (WTF::FastMalloc::free): * wtf/Forward.h: * wtf/Gigacage.cpp: (Gigacage::tryMalloc): * wtf/Gigacage.h: (Gigacage::name): * wtf/Vector.h: (WTF::VectorBufferBase::allocateBuffer): (WTF::VectorBufferBase::tryAllocateBuffer): (WTF::VectorBufferBase::reallocateBuffer): (WTF::VectorBufferBase::deallocateBuffer): (WTF::Malloc>::Vector): (WTF::=): (WTF::Malloc>::contains const): (WTF::Malloc>::findMatching const): (WTF::Malloc>::find const): (WTF::Malloc>::reverseFind const): (WTF::Malloc>::appendIfNotContains): (WTF::Malloc>::fill): (WTF::Malloc>::appendRange): (WTF::Malloc>::expandCapacity): (WTF::Malloc>::tryExpandCapacity): (WTF::Malloc>::resize): (WTF::Malloc>::resizeToFit): (WTF::Malloc>::shrink): (WTF::Malloc>::grow): (WTF::Malloc>::asanSetInitialBufferSizeTo): (WTF::Malloc>::asanSetBufferSizeToFullCapacity): (WTF::Malloc>::asanBufferSizeWillChangeTo): (WTF::Malloc>::reserveCapacity): (WTF::Malloc>::tryReserveCapacity): (WTF::Malloc>::reserveInitialCapacity): (WTF::Malloc>::shrinkCapacity): (WTF::Malloc>::append): (WTF::Malloc>::tryAppend): (WTF::Malloc>::constructAndAppend): (WTF::Malloc>::tryConstructAndAppend): (WTF::Malloc>::appendSlowCase): (WTF::Malloc>::constructAndAppendSlowCase): (WTF::Malloc>::tryConstructAndAppendSlowCase): (WTF::Malloc>::uncheckedAppend): (WTF::Malloc>::appendVector): (WTF::Malloc>::insert): (WTF::Malloc>::insertVector): (WTF::Malloc>::remove): (WTF::Malloc>::removeFirst): (WTF::Malloc>::removeFirstMatching): (WTF::Malloc>::removeAll): (WTF::Malloc>::removeAllMatching): (WTF::Malloc>::reverse): (WTF::Malloc>::map const): (WTF::Malloc>::releaseBuffer): (WTF::Malloc>::checkConsistency): (WTF::swap): (WTF::operator==): (WTF::operator!=): (WTF::removeRepeatedElements): (WTF::minCapacity>::Vector): Deleted. (WTF::minCapacity>::contains const): Deleted. (WTF::minCapacity>::findMatching const): Deleted. (WTF::minCapacity>::find const): Deleted. (WTF::minCapacity>::reverseFind const): Deleted. (WTF::minCapacity>::appendIfNotContains): Deleted. (WTF::minCapacity>::fill): Deleted. (WTF::minCapacity>::appendRange): Deleted. (WTF::minCapacity>::expandCapacity): Deleted. (WTF::minCapacity>::tryExpandCapacity): Deleted. (WTF::minCapacity>::resize): Deleted. (WTF::minCapacity>::resizeToFit): Deleted. (WTF::minCapacity>::shrink): Deleted. (WTF::minCapacity>::grow): Deleted. (WTF::minCapacity>::asanSetInitialBufferSizeTo): Deleted. (WTF::minCapacity>::asanSetBufferSizeToFullCapacity): Deleted. (WTF::minCapacity>::asanBufferSizeWillChangeTo): Deleted. (WTF::minCapacity>::reserveCapacity): Deleted. (WTF::minCapacity>::tryReserveCapacity): Deleted. (WTF::minCapacity>::reserveInitialCapacity): Deleted. (WTF::minCapacity>::shrinkCapacity): Deleted. (WTF::minCapacity>::append): Deleted. (WTF::minCapacity>::tryAppend): Deleted. (WTF::minCapacity>::constructAndAppend): Deleted. (WTF::minCapacity>::tryConstructAndAppend): Deleted. (WTF::minCapacity>::appendSlowCase): Deleted. (WTF::minCapacity>::constructAndAppendSlowCase): Deleted. (WTF::minCapacity>::tryConstructAndAppendSlowCase): Deleted. (WTF::minCapacity>::uncheckedAppend): Deleted. (WTF::minCapacity>::appendVector): Deleted. (WTF::minCapacity>::insert): Deleted. (WTF::minCapacity>::insertVector): Deleted. (WTF::minCapacity>::remove): Deleted. (WTF::minCapacity>::removeFirst): Deleted. (WTF::minCapacity>::removeFirstMatching): Deleted. (WTF::minCapacity>::removeAll): Deleted. (WTF::minCapacity>::removeAllMatching): Deleted. (WTF::minCapacity>::reverse): Deleted. (WTF::minCapacity>::map const): Deleted. (WTF::minCapacity>::releaseBuffer): Deleted. (WTF::minCapacity>::checkConsistency): Deleted. * wtf/text/AtomicStringImpl.h: * wtf/text/CString.cpp: (WTF::CStringBuffer::createUninitialized): * wtf/text/CString.h: * wtf/text/StringBuffer.h: (WTF::StringBuffer::StringBuffer): (WTF::StringBuffer::~StringBuffer): (WTF::StringBuffer::resize): * wtf/text/StringImpl.cpp: (WTF::StringImpl::~StringImpl): (WTF::StringImpl::destroy): (WTF::StringImpl::createUninitializedInternalNonEmpty): (WTF::StringImpl::reallocateInternal): (WTF::StringImpl::releaseAssertCaged const): * wtf/text/StringImpl.h: (WTF::StringImpl::createSubstringSharingImpl): (WTF::StringImpl::tryCreateUninitialized): (WTF::StringImpl::adopt): (WTF::StringImpl::bufferOwnership const): (WTF::StringImpl::assertCaged const): * wtf/text/StringMalloc.cpp: Added. (WTF::tryStringMalloc): (WTF::stringMalloc): (WTF::stringRealloc): (WTF::stringFree): * wtf/text/StringMalloc.h: Added. (WTF::StringMalloc::malloc): (WTF::StringMalloc::tryMalloc): (WTF::StringMalloc::realloc): (WTF::StringMalloc::free): * wtf/text/StringVector.h: Added. * wtf/text/SymbolImpl.h: * wtf/text/UniquedStringImpl.h: * wtf/text/WTFString.h: (WTF::String::adopt): (WTF::String::assertCaged const): (WTF::String::releaseAssertCaged const): Canonical link: https://commits.webkit.org/192824@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@221422 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-31 16:38:16 +00:00
JSValue,
Refactor the Gigacage code to require less pointer casting. https://bugs.webkit.org/show_bug.cgi?id=201521 Reviewed by Saam Barati. Source/bmalloc: 1. Define a Gigacage::Config struct instead of hacking around a g_gigacageBasePtrs array of bytes. 2. Change Gigacage::basePtr() to return a pointer instead of a reference to the requested basePtr. Instead, make it explicit when the client is trying to take the address of the basePtr, or setting it. 3. Renamed wasEnabled() to isEnabled() because it returns the present state of the flag, not some past state. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::primitiveGigacageDisabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::Config::basePtr const): (Gigacage::Config::setBasePtr): (Gigacage::isEnabled): (Gigacage::basePtr): (Gigacage::addressOfBasePtr): (Gigacage::size): (Gigacage::caged): (Gigacage::wasEnabled): Deleted. (Gigacage::setWasEnabled): Deleted. (Gigacage::basePtrs): Deleted. * bmalloc/HeapKind.h: (bmalloc::heapKind): (bmalloc::isActiveHeapKindAfterEnsuringGigacage): (bmalloc::mapToActiveHeapKindAfterEnsuringGigacage): Source/JavaScriptCore: Change LLInt's loadCagedJSValue() to skip the caging if Gigacage is not enabled in the build. This allows us to remove the unneeded stubs in WTF Gigacage.h. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cageConditionally): * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter64.asm: * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): Source/WTF: Remove some unneeded stubs in WTF Gigacage.h. * wtf/Gigacage.cpp: * wtf/Gigacage.h: (Gigacage::name): (Gigacage::isEnabled): (Gigacage::basePtr): Deleted. (Gigacage::basePtrs): Deleted. Canonical link: https://commits.webkit.org/215156@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@249556 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-09-06 00:03:52 +00:00
NumberOfKinds
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
};
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
inline void ensureGigacage() { }
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
inline void disablePrimitiveGigacage() { }
inline bool shouldBeEnabled() { return false; }
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
inline void addPrimitiveDisableCallback(void (*)(void*), void*) { }
inline void removePrimitiveDisableCallback(void (*)(void*), void*) { }
Harden protection of the Gigacage Config parameters. https://bugs.webkit.org/show_bug.cgi?id=201570 <rdar://problem/55134229> Reviewed by Saam Barati. Source/bmalloc: 1. Rename protectGigacageBasePtrs() and unprotectGigacageBasePtrs() to freezeGigacageConfig() and unfreezeGigacageConfig() respectively. Also move the alignment check in protectGigacageBasePtrs() into ensureGigacage(). There's no need to check it more than once. 2. Introduce permanentlyFreezeGigacageConfig() which permanently makes the Config ReadOnly. Once invoked, the Config cannot be made writable again. This is made possible by using vm_protect with a true set_maximum argument. We also add a g_gigacageConfig.isPermanentlyFrozen flag that we assert. Note: this permanence is only true for OS(DARWIN) since vm_protect is Mach API. 3. Rename disableDisablingPrimitiveGigacageIfShouldBeEnabled() to forbidDisablingPrimitiveGigacage() because "disablingDisabling" is a tongue twister. Also, we don't want to make it conditional on "IfShouldBeEnabled". We want forbidDisablingPrimitiveGigacage() to be irreversible. It is up to the client to ensure that the Gigacage is already initialized (if possible) before invoking forbidDisablingPrimitiveGigacage(). Conceptually, the forbidding isn't about guaranteeing that the Gigacage is enabled. It only takes away the option to disable it. That said, forbidDisablingPrimitiveGigacage() is only invoked by clients that care about keeping the Gigacage enabled. So, it does a sanity check (with an assertion) that if !GIGACAGE_ALLOCATION_CAN_FAIL, then the Gigacage should be have been initialized and enabled before invoking it. We also make sure that forbidDisablingPrimitiveGigacage() calls permanentlyFreezeGigacageConfig() unconditionally. It is safe to call it more than once. This guarantees that the Config is permanently frozen after this, even if a bug should inadvertantly set the g_gigacageConfig.disablingPrimitiveGigacageIsForbidden flag before forbidDisablingPrimitiveGigacage() is invoked. 4. Assert that ensureGigacage() is only called once. 5. Assert that shouldBeEnabled() is only called once. Also moved its cached result into the Config so that it can be frozen. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::freezeGigacageConfig): (Gigacage::bmalloc::unfreezeGigacageConfig): (Gigacage::bmalloc::permanentlyFreezeGigacageConfig): (Gigacage::bmalloc::UnfreezeGigacageConfigScope::UnfreezeGigacageConfigScope): (Gigacage::bmalloc::UnfreezeGigacageConfigScope::~UnfreezeGigacageConfigScope): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::verifyGigacageIsEnabled): (Gigacage::forbidDisablingPrimitiveGigacage): (Gigacage::isDisablingPrimitiveGigacageForbidden): (Gigacage::shouldBeEnabled): (Gigacage::bmalloc::protectGigacageBasePtrs): Deleted. (Gigacage::bmalloc::unprotectGigacageBasePtrs): Deleted. (Gigacage::bmalloc::UnprotectGigacageBasePtrsScope::UnprotectGigacageBasePtrsScope): Deleted. (Gigacage::bmalloc::UnprotectGigacageBasePtrsScope::~UnprotectGigacageBasePtrsScope): Deleted. (Gigacage::primitiveGigacageDisabled): Deleted. (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): Deleted. (Gigacage::isDisablingPrimitiveGigacageDisabled): Deleted. * bmalloc/Gigacage.h: (Gigacage::isPrimitiveGigacagePermanentlyEnabled): (Gigacage::canPrimitiveGigacageBeDisabled): (Gigacage::forbidDisablingPrimitiveGigacage): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): Deleted. Source/JavaScriptCore: Just renaming some function names here. * assembler/testmasm.cpp: (JSC::testCagePreservesPACFailureBit): * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cageConditionally): * jsc.cpp: (jscmain): Source/WebKit: Just renaming a function name here. * WebProcess/WebProcess.cpp: Source/WTF: Just renaming some function names here. * wtf/Gigacage.h: (Gigacage::forbidDisablingPrimitiveGigacage): (Gigacage::isDisablingPrimitiveGigacageForbidden): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): Deleted. (Gigacage::isDisablingPrimitiveGigacageDisabled): Deleted. Canonical link: https://commits.webkit.org/215206@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@249608 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-09-07 05:42:11 +00:00
inline void forbidDisablingPrimitiveGigacage() { }
Baseline JIT should do caging https://bugs.webkit.org/show_bug.cgi?id=175037 Reviewed by Mark Lam. Source/bmalloc: This centralizes the notion of permanently enabling the primitive gigacage, which we only do in jsc and WebProcess. This saves the baseline JIT from emitting some code. Otherwise it would always have to emit enabled checks on each typed array access. * bmalloc/Gigacage.cpp: (Gigacage::primitiveGigacageDisabled): (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::isDisablingPrimitiveGigacageDisabled): * bmalloc/Gigacage.h: (Gigacage::isPrimitiveGigacagePermanentlyEnabled): (Gigacage::canPrimitiveGigacageBeDisabled): Source/JavaScriptCore: Adds a AssemblyHelpers::cage and cageConditionally. Uses it in the baseline JIT. Also modifies FTL caging to be more defensive when caging is disabled. Relanded with fixed AssemblyHelpers::cageConditionally(). * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/InlineAccess.cpp: (JSC::InlineAccess::dumpCacheSizesAndCrash): (JSC::InlineAccess::generateSelfPropertyAccess): (JSC::InlineAccess::generateSelfPropertyReplace): (JSC::InlineAccess::generateArrayLength): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::caged): * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cage): (JSC::AssemblyHelpers::cageConditionally): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (jscmain): (primitiveGigacageDisabled): Deleted. Source/WebKit: Use a better API to disable disabling the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::m_webSQLiteDatabaseTracker): (WebKit::primitiveGigacageDisabled): Deleted. Source/WTF: * wtf/Gigacage.h: (Gigacage::disableDisablingPrimitiveGigacageIfShouldBeEnabled): (Gigacage::isDisablingPrimitiveGigacageDisabled): (Gigacage::isPrimitiveGigacagePermanentlyEnabled): (Gigacage::canPrimitiveGigacageBeDisabled): (Gigacage::basePtr): Canonical link: https://commits.webkit.org/192053@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220416 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-08 19:44:20 +00:00
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
ALWAYS_INLINE const char* name(Kind kind)
{
switch (kind) {
case Primitive:
return "Primitive";
case JSValue:
return "JSValue";
Refactor the Gigacage code to require less pointer casting. https://bugs.webkit.org/show_bug.cgi?id=201521 Reviewed by Saam Barati. Source/bmalloc: 1. Define a Gigacage::Config struct instead of hacking around a g_gigacageBasePtrs array of bytes. 2. Change Gigacage::basePtr() to return a pointer instead of a reference to the requested basePtr. Instead, make it explicit when the client is trying to take the address of the basePtr, or setting it. 3. Renamed wasEnabled() to isEnabled() because it returns the present state of the flag, not some past state. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::primitiveGigacageDisabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::Config::basePtr const): (Gigacage::Config::setBasePtr): (Gigacage::isEnabled): (Gigacage::basePtr): (Gigacage::addressOfBasePtr): (Gigacage::size): (Gigacage::caged): (Gigacage::wasEnabled): Deleted. (Gigacage::setWasEnabled): Deleted. (Gigacage::basePtrs): Deleted. * bmalloc/HeapKind.h: (bmalloc::heapKind): (bmalloc::isActiveHeapKindAfterEnsuringGigacage): (bmalloc::mapToActiveHeapKindAfterEnsuringGigacage): Source/JavaScriptCore: Change LLInt's loadCagedJSValue() to skip the caging if Gigacage is not enabled in the build. This allows us to remove the unneeded stubs in WTF Gigacage.h. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cageConditionally): * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter64.asm: * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): Source/WTF: Remove some unneeded stubs in WTF Gigacage.h. * wtf/Gigacage.cpp: * wtf/Gigacage.h: (Gigacage::name): (Gigacage::isEnabled): (Gigacage::basePtr): Deleted. (Gigacage::basePtrs): Deleted. Canonical link: https://commits.webkit.org/215156@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@249556 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-09-06 00:03:52 +00:00
case NumberOfKinds:
break;
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
}
RELEASE_ASSERT_NOT_REACHED();
return nullptr;
}
ALWAYS_INLINE bool contains(const void*) { return false; }
Change Gigacage::Config to use storage in WebConfig::g_config instead of its own. https://bugs.webkit.org/show_bug.cgi?id=212585 <rdar://problem/63812487> Reviewed by Yusuke Suzuki. Source/bmalloc: 1. Gigacage::Config now reserves and expect space to be available in an external WebConfig::g_config buffer. Gigacage does not allocate that buffer. 2. Moved Gigacage::Config to GigacageConfig.h. This allows WTFConfig.h to include GigacageConfig.h instead of all of Gigacage.h. 3. Moved Gigacage::Kind to GigacageKind.h. Otherwise, Gigacage::Kind would need to move to GigacageConfig.h which is a weird place to put it. 4. Removed freezeGigacageConfig(), unfreezeGigacageConfig(), and permanentlyFreezeGigacageConfig(). It is no longer possible to temporarily freeze and unfreeze the Gigacage::Config because it now share the same memory page with higher level Configs. permanentlyFreezeGigacageConfig() is no longer needed because it is subsumed by WTF::Config::permanentlyFreeze(), which will freeze the entire WebConfig::g_config buffer. One difference in behavior here is that Gigacage::Config data used to be permanently frozen as soon as forbidDisablingPrimitiveGigacage() is called. Now, it isn't permanently frozen until the end of the construction of the first JSC::VM instance in the process (just like the other Config records). This still guarantees that it is frozen before any JS script can run. 5. Previously, disablePrimitiveGigacage() works by nullifying the Primitive gigacage base pointer. We can no longer do that because the base pointer will be frozen on VM instantiation. Instead, if not forbidden, we now disable the Primitive gigacage by setting a disablePrimitiveGigacageRequested bool variable that is not frozen in the Gigacage::Config. To check if the Primitive gigacage is enabled, the LLInt, AssemblyHelpers::cageConditionally(), and runtime functions will check the following conditions: g_gigacageConfig.basePtr(Primitive) && (disablingPrimitiveGigacageIsForbidden() || disableNotRequestedForPrimitiveGigacage()) The base pointer being null means the gigacage was never set up. If disablingPrimitiveGigacageIsForbidden() is true, then we don't care whether a disable request has been received. Otherwise, the gigacage is only enabled if it has been set up, and a disable request has not been received. Note that the first 2 terms are frozen in the Gigacage::Config. Only the last term is a runtime variable. If disabling is forbidden, then the runtime variable never comes into play. The FTL does not rely on a runtime check for whether the Primitive gigacage is enabled or not. Instead, it relies on a watchpoint to handle this. So, it just works, and there's no performance penalty with adding the 2 extra terms to check. Note also that the jsc shell and the WebProcess will forbid disabling of the Primitive gigacage. This means the AssemblyHelpers::cageConditionally() will also not generate the runtime checks for the 2 extra terms. Only the LLInt and runtime functions will have to do work to check the 2 extra terms. But because these are not in perf critical paths, this is ok. Note that we're deliberately gating the disablePrimitiveGigacageRequested variable check on disablingPrimitiveGigacageIsForbidden though, logically, the isEnable check does not really depend on whether disabling is forbidden or not. We do this because disablingPrimitiveGigacageIsForbidden is frozen in the Config, and it is, therefore, a stronger guarantee of correctness whereas the variable can be corrupted. 6. Replaced isDisablingPrimitiveGigacageForbidden(), canPrimitiveGigacageBeDisabled(), and isPrimitiveGigacagePermanentlyEnabled() with disablingPrimitiveGigacageIsForbidden(). * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Gigacage.cpp: (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::forbidDisablingPrimitiveGigacage): (Gigacage::bmalloc::freezeGigacageConfig): Deleted. (Gigacage::bmalloc::unfreezeGigacageConfig): Deleted. (Gigacage::bmalloc::permanentlyFreezeGigacageConfig): Deleted. (Gigacage::bmalloc::UnfreezeGigacageConfigScope::UnfreezeGigacageConfigScope): Deleted. (Gigacage::bmalloc::UnfreezeGigacageConfigScope::~UnfreezeGigacageConfigScope): Deleted. (Gigacage::isDisablingPrimitiveGigacageForbidden): Deleted. * bmalloc/Gigacage.h: (Gigacage::disablingPrimitiveGigacageIsForbidden): (Gigacage::disableNotRequestedForPrimitiveGigacage): (Gigacage::isEnabled): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::forbidDisablingPrimitiveGigacage): (): Deleted. (Gigacage::Config::basePtr const): Deleted. (Gigacage::Config::setBasePtr): Deleted. (Gigacage::isPrimitiveGigacagePermanentlyEnabled): Deleted. (Gigacage::canPrimitiveGigacageBeDisabled): Deleted. * bmalloc/GigacageConfig.h: Added. (Gigacage::Config::basePtr const): (Gigacage::Config::setBasePtr): * bmalloc/GigacageKind.h: Added. * bmalloc/Heap.cpp: (bmalloc::Heap::usingGigacage): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: * assembler/testmasm.cpp: (JSC::testCagePreservesPACFailureBit): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::cageTypedArrayStorage): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::caged): * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cageConditionally): * llint/LowLevelInterpreter64.asm: * runtime/JSCConfig.h: (JSC::Config::isPermanentlyFrozen): Source/WTF: We now think of the various Config records as being allocated from parts of a WebConfig::g_config buffer. WTF::Config will manage the mechanics of freezing that buffer. And the JSC VM is still the determiner of if/when to freeze the buffer, and it will do this at the end of the construction of the very first VM instance (as before). Gigacage::Config reserves space in WebConfig::g_config. WTF::Config will honor that reservation and place itself after that. JSC::Config will continue to place itself at WTF::Config::spaceForExtensions. The upside of this approach this is that we can now share the same memory page for all the Configs, and can freeze them in one go. The downside is that g_gigacageConfig, g_wtfConfig, and g_jscConfig now have to be macros. This results in some weirdness e.g. they are no longer qualified by namespaces: referring to WTF::g_wtfConfig is now incorrect. * wtf/Gigacage.h: (Gigacage::disablingPrimitiveGigacageIsForbidden): (Gigacage::isDisablingPrimitiveGigacageForbidden): Deleted. (Gigacage::isPrimitiveGigacagePermanentlyEnabled): Deleted. (Gigacage::canPrimitiveGigacageBeDisabled): Deleted. * wtf/WTFConfig.cpp: (WTF::Config::permanentlyFreeze): * wtf/WTFConfig.h: (): Deleted. Canonical link: https://commits.webkit.org/225481@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@262434 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-06-02 19:43:17 +00:00
ALWAYS_INLINE bool disablingPrimitiveGigacageIsForbidden() { return false; }
ALWAYS_INLINE bool isEnabled() { return false; }
Refactor the Gigacage code to require less pointer casting. https://bugs.webkit.org/show_bug.cgi?id=201521 Reviewed by Saam Barati. Source/bmalloc: 1. Define a Gigacage::Config struct instead of hacking around a g_gigacageBasePtrs array of bytes. 2. Change Gigacage::basePtr() to return a pointer instead of a reference to the requested basePtr. Instead, make it explicit when the client is trying to take the address of the basePtr, or setting it. 3. Renamed wasEnabled() to isEnabled() because it returns the present state of the flag, not some past state. * bmalloc/Gigacage.cpp: (Gigacage::bmalloc::protectGigacageBasePtrs): (Gigacage::bmalloc::unprotectGigacageBasePtrs): (Gigacage::bmalloc::runwaySize): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::primitiveGigacageDisabled): * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::Config::basePtr const): (Gigacage::Config::setBasePtr): (Gigacage::isEnabled): (Gigacage::basePtr): (Gigacage::addressOfBasePtr): (Gigacage::size): (Gigacage::caged): (Gigacage::wasEnabled): Deleted. (Gigacage::setWasEnabled): Deleted. (Gigacage::basePtrs): Deleted. * bmalloc/HeapKind.h: (bmalloc::heapKind): (bmalloc::isActiveHeapKindAfterEnsuringGigacage): (bmalloc::mapToActiveHeapKindAfterEnsuringGigacage): Source/JavaScriptCore: Change LLInt's loadCagedJSValue() to skip the caging if Gigacage is not enabled in the build. This allows us to remove the unneeded stubs in WTF Gigacage.h. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::cageConditionally): * llint/LowLevelInterpreter.asm: * llint/LowLevelInterpreter64.asm: * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): Source/WTF: Remove some unneeded stubs in WTF Gigacage.h. * wtf/Gigacage.cpp: * wtf/Gigacage.h: (Gigacage::name): (Gigacage::isEnabled): (Gigacage::basePtr): Deleted. (Gigacage::basePtrs): Deleted. Canonical link: https://commits.webkit.org/215156@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@249556 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-09-06 00:03:52 +00:00
ALWAYS_INLINE bool isEnabled(Kind) { return false; }
2017-08-24 18:00:09 +00:00
ALWAYS_INLINE size_t mask(Kind) { return 0; }
ALWAYS_INLINE size_t footprint(Kind) { return 0; }
ALWAYS_INLINE size_t maxSize(Kind) { return 0; }
ALWAYS_INLINE size_t size(Kind) { return 0; }
2017-08-24 18:00:09 +00:00
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
template<typename T>
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
inline T* caged(Kind, T* ptr) { return ptr; }
template<typename T>
inline T* cagedMayBeNull(Kind, T* ptr) { return ptr; }
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
inline bool isCaged(Kind, const void*) { return false; }
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
inline void* tryAlignedMalloc(Kind, size_t alignment, size_t size) { return tryFastAlignedMalloc(alignment, size); }
inline void alignedFree(Kind, void* p) { fastAlignedFree(p); }
WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t size);
[JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active https://bugs.webkit.org/show_bug.cgi?id=196160 Reviewed by Saam Barati. Source/JavaScriptCore: "realloc" can be effective in terms of peak/current memory footprint when realloc succeeds because, 1. It does not allocate additional memory while expanding a vector 2. It does not deallocate an old memory, just reusing the current memory by expanding, so that memory footprint is tight even before scavenging We found that we can "realloc" large butterflies in certain conditions are met because, 1. If it goes to LargeAllocation, this memory region is never reused until GC sweeps it. 2. Butterflies are owned by owner JSObjects, so we know the lifetime of Butterflies. This patch attempts to use "realloc" onto butterflies if, 1. Butterflies are allocated in LargeAllocation kind 2. Concurrent collector is not active 3. Butterflies do not have property storage The condition (2) is required to avoid deallocating butterflies while the concurrent collector looks into it. The condition (3) is also required to avoid deallocating butterflies while the concurrent compiler looks into it. We also change LargeAllocation mechanism to using "malloc" and "free" instead of "posix_memalign". This allows us to use "realloc" safely in all the platforms. Since LargeAllocation uses alignment to distinguish LargeAllocation and MarkedBlock, we manually adjust 16B alignment by allocating 8B more memory in "malloc". Speedometer2 and JetStream2 are neutral. RAMification shows about 1% progression (even in some of JIT tests). * heap/AlignedMemoryAllocator.h: * heap/CompleteSubspace.cpp: (JSC::CompleteSubspace::tryAllocateSlow): (JSC::CompleteSubspace::reallocateLargeAllocationNonVirtual): * heap/CompleteSubspace.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): (JSC::IsoAlignedMemoryAllocator::tryReallocateMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/LargeAllocation.cpp: (JSC::isAlignedForLargeAllocation): (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::tryReallocate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::destroy): * heap/LargeAllocation.h: (JSC::LargeAllocation::indexInSpace): (JSC::LargeAllocation::setIndexInSpace): (JSC::LargeAllocation::basePointer const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::prepareForConservativeScan): * heap/WeakSet.h: (JSC::WeakSet::isTriviallyDestructible const): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::reallocArrayRightIfPossible): * runtime/JSObject.cpp: (JSC::JSObject::ensureLengthSlow): Source/WTF: * wtf/FastMalloc.h: (WTF::FastMalloc::tryRealloc): * wtf/Gigacage.cpp: (Gigacage::tryRealloc): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/210704@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243688 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-04-01 06:51:11 +00:00
WTF_EXPORT_PRIVATE void* tryRealloc(Kind, void*, size_t);
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
inline void free(Kind, void* p) { fastFree(p); }
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
Replace tryLargeMemalignVirtual with tryLargeZeroedMemalignVirtual and use it to allocate large zeroed memory in Wasm https://bugs.webkit.org/show_bug.cgi?id=182064 <rdar://problem/36840132> Reviewed by Geoffrey Garen. Source/bmalloc: This patch replaces the tryLargeMemalignVirtual API with tryLargeZeroedMemalignVirtual. By doing that, we're able to remove the AllocationKind enum. To zero the memory, tryLargeZeroedMemalignVirtual uses mmap(... MAP_ANON ...) over previously mmapped memory. This both purges the any resident memory for the virtual range and ensures that the pages in the range are zeroed. Most OSs should implement this by taking a page fault and zero filling on first access. Therefore, this API is returning pages that will result in page faults on first access. Hence, the name 'virtual' in the API. This API differs from the old API in that users of it need not call madvise themselves. The memory is ready to go. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Removed. * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::memalignLarge): (bmalloc::DebugHeap::freeLarge): * bmalloc/DebugHeap.h: * bmalloc/Heap.cpp: (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: * bmalloc/IsoPage.cpp: (bmalloc::IsoPageBase::allocatePageMemory): * bmalloc/VMAllocate.h: (bmalloc::vmZeroAndPurge): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::tryLargeMemalignVirtual): Deleted. * bmalloc/bmalloc.h: Source/JavaScriptCore: This patch switches WebAssembly Memory to always use bmalloc's zeroed virtual allocation API. This makes it so that we don't dirty the memory to zero it. It's a huge compile time speedup on WasmBench on iOS. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::commitZeroPages): Deleted. Source/WTF: * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAllocateVirtualPages): Deleted. * wtf/Gigacage.h: * wtf/OSAllocator.h: Canonical link: https://commits.webkit.org/198171@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@227951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-02-01 05:36:40 +00:00
WTF_EXPORT_PRIVATE void* tryAllocateZeroedVirtualPages(Kind, size_t size);
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size);
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
} // namespace Gigacage
#else
#include <bmalloc/Gigacage.h>
namespace Gigacage {
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
WTF_EXPORT_PRIVATE void* tryAlignedMalloc(Kind, size_t alignment, size_t size);
WTF_EXPORT_PRIVATE void alignedFree(Kind, void*);
WTF_EXPORT_PRIVATE void* tryMalloc(Kind, size_t);
[JSC] Butterfly allocation from LargeAllocation should try "realloc" behavior if collector thread is not active https://bugs.webkit.org/show_bug.cgi?id=196160 Reviewed by Saam Barati. Source/JavaScriptCore: "realloc" can be effective in terms of peak/current memory footprint when realloc succeeds because, 1. It does not allocate additional memory while expanding a vector 2. It does not deallocate an old memory, just reusing the current memory by expanding, so that memory footprint is tight even before scavenging We found that we can "realloc" large butterflies in certain conditions are met because, 1. If it goes to LargeAllocation, this memory region is never reused until GC sweeps it. 2. Butterflies are owned by owner JSObjects, so we know the lifetime of Butterflies. This patch attempts to use "realloc" onto butterflies if, 1. Butterflies are allocated in LargeAllocation kind 2. Concurrent collector is not active 3. Butterflies do not have property storage The condition (2) is required to avoid deallocating butterflies while the concurrent collector looks into it. The condition (3) is also required to avoid deallocating butterflies while the concurrent compiler looks into it. We also change LargeAllocation mechanism to using "malloc" and "free" instead of "posix_memalign". This allows us to use "realloc" safely in all the platforms. Since LargeAllocation uses alignment to distinguish LargeAllocation and MarkedBlock, we manually adjust 16B alignment by allocating 8B more memory in "malloc". Speedometer2 and JetStream2 are neutral. RAMification shows about 1% progression (even in some of JIT tests). * heap/AlignedMemoryAllocator.h: * heap/CompleteSubspace.cpp: (JSC::CompleteSubspace::tryAllocateSlow): (JSC::CompleteSubspace::reallocateLargeAllocationNonVirtual): * heap/CompleteSubspace.h: * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::tryAllocateMemory): (JSC::FastMallocAlignedMemoryAllocator::freeMemory): (JSC::FastMallocAlignedMemoryAllocator::tryReallocateMemory): * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::tryAllocateMemory): (JSC::GigacageAlignedMemoryAllocator::freeMemory): (JSC::GigacageAlignedMemoryAllocator::tryReallocateMemory): * heap/GigacageAlignedMemoryAllocator.h: * heap/IsoAlignedMemoryAllocator.cpp: (JSC::IsoAlignedMemoryAllocator::tryAllocateMemory): (JSC::IsoAlignedMemoryAllocator::freeMemory): (JSC::IsoAlignedMemoryAllocator::tryReallocateMemory): * heap/IsoAlignedMemoryAllocator.h: * heap/LargeAllocation.cpp: (JSC::isAlignedForLargeAllocation): (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::tryReallocate): (JSC::LargeAllocation::LargeAllocation): (JSC::LargeAllocation::destroy): * heap/LargeAllocation.h: (JSC::LargeAllocation::indexInSpace): (JSC::LargeAllocation::setIndexInSpace): (JSC::LargeAllocation::basePointer const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::sweepLargeAllocations): (JSC::MarkedSpace::prepareForConservativeScan): * heap/WeakSet.h: (JSC::WeakSet::isTriviallyDestructible const): * runtime/Butterfly.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::reallocArrayRightIfPossible): * runtime/JSObject.cpp: (JSC::JSObject::ensureLengthSlow): Source/WTF: * wtf/FastMalloc.h: (WTF::FastMalloc::tryRealloc): * wtf/Gigacage.cpp: (Gigacage::tryRealloc): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/210704@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@243688 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2019-04-01 06:51:11 +00:00
WTF_EXPORT_PRIVATE void* tryRealloc(Kind, void*, size_t);
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
WTF_EXPORT_PRIVATE void free(Kind, void*);
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
Replace tryLargeMemalignVirtual with tryLargeZeroedMemalignVirtual and use it to allocate large zeroed memory in Wasm https://bugs.webkit.org/show_bug.cgi?id=182064 <rdar://problem/36840132> Reviewed by Geoffrey Garen. Source/bmalloc: This patch replaces the tryLargeMemalignVirtual API with tryLargeZeroedMemalignVirtual. By doing that, we're able to remove the AllocationKind enum. To zero the memory, tryLargeZeroedMemalignVirtual uses mmap(... MAP_ANON ...) over previously mmapped memory. This both purges the any resident memory for the virtual range and ensures that the pages in the range are zeroed. Most OSs should implement this by taking a page fault and zero filling on first access. Therefore, this API is returning pages that will result in page faults on first access. Hence, the name 'virtual' in the API. This API differs from the old API in that users of it need not call madvise themselves. The memory is ready to go. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Removed. * bmalloc/DebugHeap.cpp: (bmalloc::DebugHeap::memalignLarge): (bmalloc::DebugHeap::freeLarge): * bmalloc/DebugHeap.h: * bmalloc/Heap.cpp: (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: * bmalloc/IsoPage.cpp: (bmalloc::IsoPageBase::allocatePageMemory): * bmalloc/VMAllocate.h: (bmalloc::vmZeroAndPurge): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/bmalloc.cpp: (bmalloc::api::tryLargeZeroedMemalignVirtual): (bmalloc::api::freeLargeVirtual): (bmalloc::api::tryLargeMemalignVirtual): Deleted. * bmalloc/bmalloc.h: Source/JavaScriptCore: This patch switches WebAssembly Memory to always use bmalloc's zeroed virtual allocation API. This makes it so that we don't dirty the memory to zero it. It's a huge compile time speedup on WasmBench on iOS. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::commitZeroPages): Deleted. Source/WTF: * wtf/Gigacage.cpp: (Gigacage::tryAllocateZeroedVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAllocateVirtualPages): Deleted. * wtf/Gigacage.h: * wtf/OSAllocator.h: Canonical link: https://commits.webkit.org/198171@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@227951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2018-02-01 05:36:40 +00:00
WTF_EXPORT_PRIVATE void* tryAllocateZeroedVirtualPages(Kind, size_t size);
Primitive auxiliaries and JSValue auxiliaries should have separate gigacages https://bugs.webkit.org/show_bug.cgi?id=174919 Reviewed by Keith Miller. Source/bmalloc: This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of HeapKind, PrimitiveGigacage and JSValueGigacage. The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and NO_INLINE into BNO_INLINE. * bmalloc.xcodeproj/project.pbxproj: * bmalloc/Allocator.cpp: (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::refillAllocator): (bmalloc::Allocator::allocateLarge): (bmalloc::Allocator::allocateLogSizeClass): * bmalloc/AsyncTask.h: * bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h. * bmalloc/Cache.cpp: (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): * bmalloc/Deallocator.cpp: * bmalloc/Gigacage.cpp: (Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks): (Gigacage::ensureGigacage): (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::Callbacks::Callbacks): Deleted. (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. * bmalloc/Gigacage.h: (Gigacage::name): (Gigacage::basePtr): (Gigacage::forEachKind): (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::gigacageBasePtr): * bmalloc/Heap.h: * bmalloc/HeapKind.h: (bmalloc::isGigacage): (bmalloc::gigacageKind): (bmalloc::heapKind): * bmalloc/Inline.h: Removed. * bmalloc/Map.h: * bmalloc/PerProcess.h: (bmalloc::PerProcess<T>::getFastCase): (bmalloc::PerProcess<T>::get): (bmalloc::PerProcess<T>::getSlowCase): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::getFastCase): * bmalloc/Vector.h: (bmalloc::Vector<T>::push): (bmalloc::Vector<T>::shrinkCapacity): (bmalloc::Vector<T>::growCapacity): Source/JavaScriptCore: This adapts JSC to there being two gigacages. To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than singletons. I don't think we were gaining anything by making them be singletons. This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of gigacages. We'll have one of those allocators per cage. From there, this change teaches everyone who previously knew about cages that there are two cages. This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's not so obvious, so this change introduces some helpers to make it easy to define what cage you want to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and CagedPtr. This removes one layer of "get()" calls from a bunch of places. * JavaScriptCore.xcodeproj/project.pbxproj: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileNewTypedArray): (JSC::DFG::SpeculativeJIT::emitAllocateButterfly): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/FastMallocAlignedMemoryAllocator.cpp: (JSC::FastMallocAlignedMemoryAllocator::instance): Deleted. * heap/FastMallocAlignedMemoryAllocator.h: * heap/GigacageAlignedMemoryAllocator.cpp: (JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator): (JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory): (JSC::GigacageAlignedMemoryAllocator::dump const): (JSC::GigacageAlignedMemoryAllocator::instance): Deleted. * heap/GigacageAlignedMemoryAllocator.h: * jsc.cpp: (primitiveGigacageDisabled): (jscmain): (gigacageDisabled): Deleted. * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): * runtime/AuxiliaryBarrier.h: * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/CagedBarrierPtr.h: Added. (JSC::CagedBarrierPtr::CagedBarrierPtr): (JSC::CagedBarrierPtr::clear): (JSC::CagedBarrierPtr::set): (JSC::CagedBarrierPtr::get const): (JSC::CagedBarrierPtr::getMayBeNull const): (JSC::CagedBarrierPtr::operator== const): (JSC::CagedBarrierPtr::operator!= const): (JSC::CagedBarrierPtr::operator bool const): (JSC::CagedBarrierPtr::setWithoutBarrier): (JSC::CagedBarrierPtr::operator* const): (JSC::CagedBarrierPtr::operator-> const): (JSC::CagedBarrierPtr::operator[] const): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): (JSC::DirectArguments::unmapArgument): * runtime/DirectArguments.h: (JSC::DirectArguments::isMappedArgument const): * runtime/GenericArguments.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): (JSC::GenericArguments<Type>::setModifiedArgumentDescriptor): (JSC::GenericArguments<Type>::isModifiedArgumentDescriptor): * runtime/HashMapImpl.cpp: (JSC::HashMapImpl<HashMapBucket>::visitChildren): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): (JSC::HashMapImpl::buffer const): (JSC::HashMapImpl::rehash): * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): (JSC::JSArray::setLength): (JSC::JSArray::pop): (JSC::JSArray::push): (JSC::JSArray::fastSlice): (JSC::JSArray::shiftCountWithArrayStorage): (JSC::JSArray::shiftCountWithAnyIndexingType): (JSC::JSArray::unshiftCountWithAnyIndexingType): (JSC::JSArray::fillArgList): (JSC::JSArray::copyToArguments): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.cpp: (JSC::JSObject::heapSnapshot): (JSC::JSObject::getOwnPropertySlotByIndex): (JSC::JSObject::putByIndex): (JSC::JSObject::enterDictionaryIndexingMode): (JSC::JSObject::createInitialIndexedStorage): (JSC::JSObject::createArrayStorage): (JSC::JSObject::convertUndecidedToInt32): (JSC::JSObject::convertUndecidedToDouble): (JSC::JSObject::convertUndecidedToContiguous): (JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements): (JSC::JSObject::convertUndecidedToArrayStorage): (JSC::JSObject::convertInt32ToDouble): (JSC::JSObject::convertInt32ToContiguous): (JSC::JSObject::convertInt32ToArrayStorage): (JSC::JSObject::convertDoubleToContiguous): (JSC::JSObject::convertDoubleToArrayStorage): (JSC::JSObject::convertContiguousToArrayStorage): (JSC::JSObject::setIndexQuicklyToUndecided): (JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode): (JSC::JSObject::deletePropertyByIndex): (JSC::JSObject::getOwnPropertyNames): (JSC::JSObject::putIndexedDescriptor): (JSC::JSObject::defineOwnIndexedProperty): (JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes): (JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength): (JSC::JSObject::getNewVectorLength): (JSC::JSObject::ensureLengthSlow): (JSC::JSObject::reallocateAndShrinkButterfly): (JSC::JSObject::allocateMoreOutOfLineStorage): (JSC::JSObject::getEnumerableLength): * runtime/JSObject.h: (JSC::JSObject::getArrayLength const): (JSC::JSObject::getVectorLength): (JSC::JSObject::putDirectIndex): (JSC::JSObject::canGetIndexQuickly): (JSC::JSObject::getIndexQuickly): (JSC::JSObject::tryGetIndexQuickly const): (JSC::JSObject::canSetIndexQuickly): (JSC::JSObject::setIndexQuickly): (JSC::JSObject::initializeIndex): (JSC::JSObject::initializeIndexWithoutBarrier): (JSC::JSObject::hasSparseMap): (JSC::JSObject::inSparseIndexingMode): (JSC::JSObject::butterfly const): (JSC::JSObject::butterfly): (JSC::JSObject::outOfLineStorage const): (JSC::JSObject::outOfLineStorage): (JSC::JSObject::ensureInt32): (JSC::JSObject::ensureDouble): (JSC::JSObject::ensureContiguous): (JSC::JSObject::ensureArrayStorage): (JSC::JSObject::arrayStorage): (JSC::JSObject::arrayStorageOrNull): (JSC::JSObject::ensureLength): * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::primitiveGigacageDisabledCallback): (JSC::VM::primitiveGigacageDisabled): (JSC::VM::gigacageDisabledCallback): Deleted. (JSC::VM::gigacageDisabled): Deleted. * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): (JSC::VM::firePrimitiveGigacageEnabledIfNecessary): (JSC::VM::primitiveGigacageEnabled): (JSC::VM::fireGigacageEnabledIfNecessary): Deleted. (JSC::VM::gigacageEnabled): Deleted. * wasm/WasmMemory.cpp: (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::grow): Source/WebCore: No new tests because no change in behavior. Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be specific about which Gigacage we're using. * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): Source/WebKit: The disable callback is all about the primitive gigacage. * WebProcess/WebProcess.cpp: (WebKit::primitiveGigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): (WebKit::gigacageDisabled): Deleted. Source/WTF: This mirrors the changes from bmalloc/Gigacage.h. Also it teaches CagedPtr how to reason about multiple gigacages. * wtf/CagedPtr.h: (WTF::CagedPtr::get const): (WTF::CagedPtr::operator[] const): * wtf/Gigacage.cpp: (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: (Gigacage::disablePrimitiveGigacage): (Gigacage::addPrimitiveDisableCallback): (Gigacage::removePrimitiveDisableCallback): (Gigacage::name): (Gigacage::basePtr): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): (Gigacage::disableGigacage): Deleted. (Gigacage::addDisableCallback): Deleted. (Gigacage::removeDisableCallback): Deleted. Canonical link: https://commits.webkit.org/192001@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220352 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-07 21:31:49 +00:00
WTF_EXPORT_PRIVATE void freeVirtualPages(Kind, void* basePtr, size_t size);
Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region) https://bugs.webkit.org/show_bug.cgi?id=174727 Reviewed by Mark Lam. Source/bmalloc: This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps (isolated heaps) have a very simple relationship with each other and with the rest of bmalloc: - You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h. - Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind. - Each isoheap gets its own Heap. - Each Heap gets a scavenger thread. - Some things, like Zone/VMHeap/Scavenger, are per-process. Most of the per-HeapKind functionality is handled by PerHeapKind<>. This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps: the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual region that we now use for variable-length random-access allocations. No Primary allocations will go into the Gigacage. * CMakeLists.txt: * bmalloc.xcodeproj/project.pbxproj: * bmalloc/AllocationKind.h: Added. * bmalloc/Allocator.cpp: (bmalloc::Allocator::Allocator): (bmalloc::Allocator::tryAllocate): (bmalloc::Allocator::allocateImpl): (bmalloc::Allocator::reallocate): (bmalloc::Allocator::refillAllocatorSlowCase): (bmalloc::Allocator::allocateLarge): * bmalloc/Allocator.h: * bmalloc/BExport.h: Added. * bmalloc/Cache.cpp: (bmalloc::Cache::scavenge): (bmalloc::Cache::Cache): (bmalloc::Cache::tryAllocateSlowCaseNullCache): (bmalloc::Cache::allocateSlowCaseNullCache): (bmalloc::Cache::deallocateSlowCaseNullCache): (bmalloc::Cache::reallocateSlowCaseNullCache): (bmalloc::Cache::operator new): Deleted. (bmalloc::Cache::operator delete): Deleted. * bmalloc/Cache.h: (bmalloc::Cache::tryAllocate): (bmalloc::Cache::allocate): (bmalloc::Cache::deallocate): (bmalloc::Cache::reallocate): * bmalloc/Deallocator.cpp: (bmalloc::Deallocator::Deallocator): (bmalloc::Deallocator::scavenge): (bmalloc::Deallocator::processObjectLog): (bmalloc::Deallocator::deallocateSlowCase): * bmalloc/Deallocator.h: * bmalloc/Gigacage.cpp: Added. (Gigacage::Callback::Callback): (Gigacage::Callback::function): (Gigacage::Callbacks::Callbacks): (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): * bmalloc/Gigacage.h: Added. (Gigacage::caged): (Gigacage::isCaged): * bmalloc/Heap.cpp: (bmalloc::Heap::Heap): (bmalloc::Heap::usingGigacage): (bmalloc::Heap::concurrentScavenge): (bmalloc::Heap::splitAndAllocate): (bmalloc::Heap::tryAllocateLarge): (bmalloc::Heap::allocateLarge): (bmalloc::Heap::shrinkLarge): (bmalloc::Heap::deallocateLarge): * bmalloc/Heap.h: (bmalloc::Heap::mutex): (bmalloc::Heap::kind const): (bmalloc::Heap::setScavengerThreadQOSClass): Deleted. * bmalloc/HeapKind.h: Added. * bmalloc/ObjectType.cpp: (bmalloc::objectType): * bmalloc/ObjectType.h: * bmalloc/PerHeapKind.h: Added. (bmalloc::PerHeapKindBase::PerHeapKindBase): (bmalloc::PerHeapKindBase::size): (bmalloc::PerHeapKindBase::at): (bmalloc::PerHeapKindBase::at const): (bmalloc::PerHeapKindBase::operator[]): (bmalloc::PerHeapKindBase::operator[] const): (bmalloc::StaticPerHeapKind::StaticPerHeapKind): (bmalloc::PerHeapKind::PerHeapKind): (bmalloc::PerHeapKind::~PerHeapKind): * bmalloc/PerThread.h: (bmalloc::PerThread<T>::destructor): (bmalloc::PerThread<T>::getSlowCase): (bmalloc::PerThreadStorage<Cache>::get): Deleted. (bmalloc::PerThreadStorage<Cache>::init): Deleted. * bmalloc/Scavenger.cpp: Added. (bmalloc::Scavenger::Scavenger): (bmalloc::Scavenger::scavenge): * bmalloc/Scavenger.h: Added. (bmalloc::Scavenger::setScavengerThreadQOSClass): (bmalloc::Scavenger::requestedScavengerThreadQOSClass const): * bmalloc/VMHeap.cpp: (bmalloc::VMHeap::VMHeap): (bmalloc::VMHeap::tryAllocateLargeChunk): * bmalloc/VMHeap.h: * bmalloc/Zone.cpp: (bmalloc::Zone::Zone): * bmalloc/Zone.h: * bmalloc/bmalloc.h: (bmalloc::api::tryMalloc): (bmalloc::api::malloc): (bmalloc::api::tryMemalign): (bmalloc::api::memalign): (bmalloc::api::realloc): (bmalloc::api::tryLargeMemalignVirtual): (bmalloc::api::free): (bmalloc::api::freeLargeVirtual): (bmalloc::api::scavengeThisThread): (bmalloc::api::scavenge): (bmalloc::api::isEnabled): (bmalloc::api::setScavengerThreadQOSClass): * bmalloc/mbmalloc.cpp: Source/JavaScriptCore: This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses themselves. Basically, we do masking to ensure that the pointer points into the gigacage. This is neutral on JetStream. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * b3/B3InsertionSet.cpp: (JSC::B3::InsertionSet::execute): * dfg/DFGAbstractInterpreterInlines.h: (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects): * dfg/DFGArgumentsEliminationPhase.cpp: * dfg/DFGClobberize.cpp: (JSC::DFG::readsOverlap): * dfg/DFGClobberize.h: (JSC::DFG::clobberize): * dfg/DFGDoesGC.cpp: (JSC::DFG::doesGC): * dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added. (JSC::DFG::performFixedButterflyAccessUncaging): * dfg/DFGFixedButterflyAccessUncagingPhase.h: Added. * dfg/DFGFixupPhase.cpp: (JSC::DFG::FixupPhase::fixupNode): * dfg/DFGHeapLocation.cpp: (WTF::printInternal): * dfg/DFGHeapLocation.h: * dfg/DFGNodeType.h: * dfg/DFGPlan.cpp: (JSC::DFG::Plan::compileInThreadImpl): * dfg/DFGPredictionPropagationPhase.cpp: * dfg/DFGSafeToExecute.h: (JSC::DFG::safeToExecute): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetButterfly): * dfg/DFGSpeculativeJIT32_64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * dfg/DFGTypeCheckHoistingPhase.cpp: (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks): (JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks): * ftl/FTLCapabilities.cpp: (JSC::FTL::canCompile): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileNode): (JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly): (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): (JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket): (JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname): (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): (JSC::FTL::DFG::LowerDFGToB3::caged): * heap/GigacageSubspace.cpp: Added. (JSC::GigacageSubspace::GigacageSubspace): (JSC::GigacageSubspace::~GigacageSubspace): (JSC::GigacageSubspace::tryAllocateAlignedMemory): (JSC::GigacageSubspace::freeAlignedMemory): (JSC::GigacageSubspace::canTradeBlocksWith): * heap/GigacageSubspace.h: Added. * heap/Heap.cpp: (JSC::Heap::Heap): (JSC::Heap::lastChanceToFinalize): (JSC::Heap::finalize): (JSC::Heap::sweepInFinalize): (JSC::Heap::updateAllocationLimits): (JSC::Heap::shouldDoFullCollection): (JSC::Heap::collectIfNecessaryOrDefer): (JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted. (JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted. (JSC::Heap::sweepLargeAllocations): Deleted. (JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted. * heap/Heap.h: * heap/LargeAllocation.cpp: (JSC::LargeAllocation::tryCreate): (JSC::LargeAllocation::destroy): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::tryAllocateWithoutCollecting): (JSC::MarkedAllocator::tryAllocateBlock): * heap/MarkedBlock.cpp: (JSC::MarkedBlock::tryCreate): (JSC::MarkedBlock::Handle::Handle): (JSC::MarkedBlock::Handle::~Handle): (JSC::MarkedBlock::Handle::didAddToAllocator): (JSC::MarkedBlock::Handle::subspace const): Deleted. * heap/MarkedBlock.h: (JSC::MarkedBlock::Handle::subspace const): * heap/MarkedSpace.cpp: (JSC::MarkedSpace::~MarkedSpace): (JSC::MarkedSpace::freeMemory): (JSC::MarkedSpace::prepareForAllocation): (JSC::MarkedSpace::addMarkedAllocator): (JSC::MarkedSpace::findEmptyBlockToSteal): Deleted. * heap/MarkedSpace.h: (JSC::MarkedSpace::firstAllocator const): (JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted. * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::canTradeBlocksWith): (JSC::Subspace::tryAllocateAlignedMemory): (JSC::Subspace::freeAlignedMemory): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): * heap/Subspace.h: (JSC::Subspace::didCreateFirstAllocator): * heap/SubspaceInlines.h: (JSC::Subspace::forEachAllocator): (JSC::Subspace::forEachMarkedBlock): (JSC::Subspace::forEachNotEmptyMarkedBlock): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitDoubleLoad): (JSC::JIT::emitContiguousLoad): (JSC::JIT::emitArrayStorageLoad): (JSC::JIT::emitGenericContiguousPutByVal): (JSC::JIT::emitArrayStoragePutByVal): (JSC::JIT::emit_op_get_from_scope): (JSC::JIT::emit_op_put_to_scope): (JSC::JIT::emitIntTypedArrayGetByVal): (JSC::JIT::emitFloatTypedArrayGetByVal): (JSC::JIT::emitIntTypedArrayPutByVal): (JSC::JIT::emitFloatTypedArrayPutByVal): * jsc.cpp: (fillBufferWithContentsOfFile): (functionReadFile): (gigacageDisabled): (jscmain): * llint/LowLevelInterpreter64.asm: * runtime/ArrayBuffer.cpp: (JSC::ArrayBufferContents::tryAllocate): (JSC::ArrayBuffer::createAdopted): (JSC::ArrayBuffer::createFromBytes): (JSC::ArrayBuffer::tryCreate): * runtime/IndexingHeader.h: * runtime/InitializeThreading.cpp: (JSC::initializeThreading): * runtime/JSArrayBuffer.cpp: * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): (JSC::JSArrayBufferView::finalize): * runtime/JSLock.cpp: (JSC::JSLock::didAcquireLock): * runtime/JSObject.h: * runtime/Options.cpp: (JSC::recomputeDependentOptions): * runtime/Options.h: * runtime/ScopedArgumentsTable.h: * runtime/VM.cpp: (JSC::VM::VM): (JSC::VM::~VM): (JSC::VM::gigacageDisabledCallback): (JSC::VM::gigacageDisabled): * runtime/VM.h: (JSC::VM::fireGigacageEnabledIfNecessary): (JSC::VM::gigacageEnabled): * wasm/WasmB3IRGenerator.cpp: (JSC::Wasm::B3IRGenerator::B3IRGenerator): (JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer): * wasm/WasmCodeBlock.cpp: (JSC::Wasm::CodeBlock::isSafeToRun): * wasm/WasmMemory.cpp: (JSC::Wasm::makeString): (JSC::Wasm::Memory::create): (JSC::Wasm::Memory::~Memory): (JSC::Wasm::Memory::addressIsInActiveFastMemory): (JSC::Wasm::Memory::grow): (JSC::Wasm::Memory::initializePreallocations): Deleted. (JSC::Wasm::Memory::maxFastMemoryCount): Deleted. * wasm/WasmMemory.h: * wasm/js/JSWebAssemblyInstance.cpp: (JSC::JSWebAssemblyInstance::create): * wasm/js/JSWebAssemblyMemory.cpp: (JSC::JSWebAssemblyMemory::grow): (JSC::JSWebAssemblyMemory::finishCreation): * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no change in behavior. Needed to teach Metal how to allocate in the Gigacage. * platform/graphics/cocoa/GPUBufferMetal.mm: (WebCore::GPUBuffer::GPUBuffer): (WebCore::GPUBuffer::contents): Source/WebKit: The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So, we add a callback that crashes the process. * WebProcess/WebProcess.cpp: (WebKit::gigacageDisabled): (WebKit::m_webSQLiteDatabaseTracker): Source/WTF: For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc. Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses, it's better to hide the question of whether or not it's enabled inside this API. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/FastMalloc.cpp: * wtf/Gigacage.cpp: Added. (Gigacage::tryMalloc): (Gigacage::tryAllocateVirtualPages): (Gigacage::freeVirtualPages): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): * wtf/Gigacage.h: Added. (Gigacage::ensureGigacage): (Gigacage::disableGigacage): (Gigacage::addDisableCallback): (Gigacage::removeDisableCallback): (Gigacage::caged): (Gigacage::isCaged): (Gigacage::tryAlignedMalloc): (Gigacage::alignedFree): (Gigacage::free): Canonical link: https://commits.webkit.org/191825@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220118 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-02 01:50:16 +00:00
} // namespace Gigacage
#endif
Put the ScopedArgumentsTable's ScopeOffset array in some gigacage https://bugs.webkit.org/show_bug.cgi?id=174921 Reviewed by Mark Lam. Source/JavaScriptCore: Uses CagedUniquePtr<> to cage the ScopeOffset array. * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::compileGetByValOnScopedArguments): * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileGetByVal): * jit/JITPropertyAccess.cpp: (JSC::JIT::emitScopedArgumentsGetByVal): * runtime/ScopedArgumentsTable.cpp: (JSC::ScopedArgumentsTable::create): (JSC::ScopedArgumentsTable::setLength): * runtime/ScopedArgumentsTable.h: Source/WTF: If you want to std::unique_ptr a class that knows that it should be in the Gigacage, then we would create the Gigacage equivalent of WTF_MAKE_FAST_ALLOCATED and it would just work. But this does not work if we want to std::unique_ptr a primitive type. So, this patch adds a solution for this problem: CagedUniquePtr<>. This will handle allocation (CagedUniquePtr<>::create()) and deallocation (in the style of std::unique_ptr). It has three variants: - Non-array types. - Arrays that don't have destructors. - Arrays that have destructors. Just like std::unique_ptr, the array case is triggered by saying "[]" at the end of the type. Unlike std::unique_ptr and most other smart pointers, the whole point of this smart pointer is to dictate where the thing you're pointing at is allocated. For this reason, it has to know how to do things like the array destructor protocol. So it creates its own: the CagedUniquePtr for arrays with destructors is a fat pointer that remembers the length of the array. CagedUniquePtr<> makes it impossible to leak/release the pointer. This is stricter than what std::unique_ptr does, and is probably appropriate for all of the places where we would use this type. So far, we only use it for ScopedArgumentsTable::m_arguments, but I suspect that it will be useful in other places. * WTF.xcodeproj/project.pbxproj: * wtf/CMakeLists.txt: * wtf/CagedUniquePtr.h: Added. (WTF::CagedUniquePtr::CagedUniquePtr): (WTF::CagedUniquePtr::create): (WTF::CagedUniquePtr::operator=): (WTF::CagedUniquePtr::~CagedUniquePtr): (WTF::CagedUniquePtr::get const): (WTF::CagedUniquePtr::getMayBeNull const): (WTF::CagedUniquePtr::operator== const): (WTF::CagedUniquePtr::operator!= const): (WTF::CagedUniquePtr::operator bool const): (WTF::CagedUniquePtr::operator* const): (WTF::CagedUniquePtr::operator-> const): (WTF::CagedUniquePtr::operator[] const): (WTF::CagedUniquePtr::destroy): * wtf/Gigacage.cpp: (Gigacage::tryMallocArray): (Gigacage::malloc): (Gigacage::mallocArray): * wtf/Gigacage.h: Canonical link: https://commits.webkit.org/192235@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@220712 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-08-14 18:33:16 +00:00
namespace Gigacage {
WTF_EXPORT_PRIVATE void* tryMallocArray(Kind, size_t numElements, size_t elementSize);
WTF_EXPORT_PRIVATE void* malloc(Kind, size_t);
WTF_EXPORT_PRIVATE void* mallocArray(Kind, size_t numElements, size_t elementSize);
} // namespace Gigacage