haikuwebkit/Source/WTF/wtf/SinglyLinkedListWithTail.h

64 lines
2.0 KiB
C
Raw Permalink Normal View History

CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00
/*
* Copyright (C) 2017 Apple Inc. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#pragma once
GC should support isoheaps https://bugs.webkit.org/show_bug.cgi?id=179288 Reviewed by Saam Barati. Source/JavaScriptCore: This expands the power of the Subspace API in JSC: - Everything associated with describing the types of objects is now part of the HeapCellType class. We have different HeapCellTypes for different destruction strategies. Any Subspace can use any HeapCellType; these are orthogonal things. - There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual pages but releases the physical pages as part of the respective allocator's scavenging policy (the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for IsoSubspace). So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it for more things. This does not have any effect on JetStream (0.18% faster with p = 0.69). * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/ObjectAllocationProfileInlines.h: (JSC::ObjectAllocationProfile::initializeProfile): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): * heap/AlignedMemoryAllocator.cpp: (JSC::AlignedMemoryAllocator::registerAllocator): (JSC::AlignedMemoryAllocator::registerSubspace): * heap/AlignedMemoryAllocator.h: (JSC::AlignedMemoryAllocator::firstAllocator const): * heap/AllocationFailureMode.h: Added. * heap/CompleteSubspace.cpp: Added. (JSC::CompleteSubspace::CompleteSubspace): (JSC::CompleteSubspace::~CompleteSubspace): (JSC::CompleteSubspace::allocatorFor): (JSC::CompleteSubspace::allocate): (JSC::CompleteSubspace::allocateNonVirtual): (JSC::CompleteSubspace::allocatorForSlow): (JSC::CompleteSubspace::allocateSlow): (JSC::CompleteSubspace::tryAllocateSlow): * heap/CompleteSubspace.h: Added. (JSC::CompleteSubspace::offsetOfAllocatorForSizeStep): (JSC::CompleteSubspace::allocatorForSizeStep): (JSC::CompleteSubspace::allocatorForNonVirtual): * heap/HeapCellType.cpp: Added. (JSC::HeapCellType::HeapCellType): (JSC::HeapCellType::~HeapCellType): (JSC::HeapCellType::finishSweep): (JSC::HeapCellType::destroy): * heap/HeapCellType.h: Added. (JSC::HeapCellType::attributes const): * heap/IsoAlignedMemoryAllocator.cpp: Added. (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::dump const): * heap/IsoAlignedMemoryAllocator.h: Added. * heap/IsoSubspace.cpp: Added. (JSC::IsoSubspace::IsoSubspace): (JSC::IsoSubspace::~IsoSubspace): (JSC::IsoSubspace::allocatorFor): (JSC::IsoSubspace::allocatorForNonVirtual): (JSC::IsoSubspace::allocate): (JSC::IsoSubspace::allocateNonVirtual): * heap/IsoSubspace.h: Added. (JSC::IsoSubspace::size const): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::setSubspace): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): Deleted. (JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const): (JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator): * heap/MarkedAllocatorInlines.h: (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::tryAllocate): Deleted. * heap/MarkedBlock.h: * heap/MarkedBlockInlines.h: (JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType): (JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::addMarkedAllocator): * heap/MarkedSpace.h: * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::initialize): (JSC::Subspace::finishSweep): (JSC::Subspace::destroy): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): (): Deleted. (JSC::Subspace::allocate): Deleted. (JSC::Subspace::tryAllocate): Deleted. (JSC::Subspace::allocatorForSlow): Deleted. (JSC::Subspace::allocateSlow): Deleted. (JSC::Subspace::tryAllocateSlow): Deleted. (JSC::Subspace::didAllocate): Deleted. * heap/Subspace.h: (JSC::Subspace::heapCellType const): (JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const): (JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator): (JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted. (JSC::Subspace::allocatorForSizeStep): Deleted. (JSC::Subspace::tryAllocatorFor): Deleted. (JSC::Subspace::allocatorFor): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): (JSC::AssemblyHelpers::emitAllocateVariableSizedCell): * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): * runtime/DirectArguments.h: (JSC::DirectArguments::subspaceFor): * runtime/DirectEvalExecutable.h: * runtime/EvalExecutable.h: * runtime/ExecutableBase.h: (JSC::ExecutableBase::subspaceFor): * runtime/FunctionExecutable.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): * runtime/IndirectEvalExecutable.h: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): * runtime/JSCell.h: (JSC::subspaceFor): * runtime/JSCellInlines.h: (JSC::JSCell::subspaceFor): (JSC::tryAllocateCellHelper): (JSC::allocateCell): (JSC::tryAllocateCell): * runtime/JSDestructibleObject.h: (JSC::JSDestructibleObject::subspaceFor): * runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp. (JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::finishSweep): (JSC::JSDestructibleObjectHeapCellType::destroy): (JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::finishSweep): Deleted. (JSC::JSDestructibleObjectSubspace::destroy): Deleted. * runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h. * runtime/JSDestructibleObjectSubspace.cpp: Removed. * runtime/JSDestructibleObjectSubspace.h: Removed. * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::subspaceFor): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::subspaceFor): * runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp. (JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::finishSweep): (JSC::JSSegmentedVariableObjectHeapCellType::destroy): (JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted. (JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted. * runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h. * runtime/JSSegmentedVariableObjectSubspace.cpp: Removed. * runtime/JSSegmentedVariableObjectSubspace.h: Removed. * runtime/JSString.h: (JSC::JSString::subspaceFor): * runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp. (JSC::JSStringHeapCellType::JSStringHeapCellType): (JSC::JSStringHeapCellType::~JSStringHeapCellType): (JSC::JSStringHeapCellType::finishSweep): (JSC::JSStringHeapCellType::destroy): (JSC::JSStringSubspace::JSStringSubspace): Deleted. (JSC::JSStringSubspace::~JSStringSubspace): Deleted. (JSC::JSStringSubspace::finishSweep): Deleted. (JSC::JSStringSubspace::destroy): Deleted. * runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h. * runtime/JSStringSubspace.cpp: Removed. * runtime/JSStringSubspace.h: Removed. * runtime/ModuleProgramExecutable.h: * runtime/NativeExecutable.h: * runtime/ProgramExecutable.h: * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/ScopedArguments.h: (JSC::ScopedArguments::subspaceFor): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): * wasm/js/JSWebAssemblyCodeBlock.h: * wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp. (JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep): (JSC::JSWebAssemblyCodeBlockHeapCellType::destroy): (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted. * wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed. * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed. * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no new behavior. Adopting changes in JSC Subspace API. * ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added. * ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added. * bindings/js/JSDOMWrapper.cpp: (WebCore::outputConstraintSubspaceFor): (WebCore::globalObjectOutputConstraintSubspaceFor): * bindings/js/JSDOMWrapper.h: * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * bindings/js/WebCoreJSClientData.h: (WebCore::JSVMClientData::outputConstraintSpace): (WebCore::JSVMClientData::globalObjectOutputConstraintSpace): * bindings/scripts/CodeGeneratorJS.pm: (GenerateHeader): Source/WTF: One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that things get added to it at the end rather that at the beginning. In this patch, I use this to put the same node on multiple lists, which our existing linked list templates also don't support. This adds a new linked list that does those things: - It supports append(). It could also support prepend(), but currently there is no need for it. - It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the list uses to set next. * WTF.xcodeproj/project.pbxproj: * wtf/SinglyLinkedListWithTail.h: Added. (WTF::SinglyLinkedListWithTail::isEmpty const): (WTF::SinglyLinkedListWithTail::append): (WTF::SinglyLinkedListWithTail::first const): (WTF::SinglyLinkedListWithTail::last const): Canonical link: https://commits.webkit.org/196172@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-30 04:39:50 +00:00
#include <wtf/Assertions.h>
#include <wtf/StdLibExtras.h>
CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00
GC should support isoheaps https://bugs.webkit.org/show_bug.cgi?id=179288 Reviewed by Saam Barati. Source/JavaScriptCore: This expands the power of the Subspace API in JSC: - Everything associated with describing the types of objects is now part of the HeapCellType class. We have different HeapCellTypes for different destruction strategies. Any Subspace can use any HeapCellType; these are orthogonal things. - There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual pages but releases the physical pages as part of the respective allocator's scavenging policy (the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for IsoSubspace). So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it for more things. This does not have any effect on JetStream (0.18% faster with p = 0.69). * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/ObjectAllocationProfileInlines.h: (JSC::ObjectAllocationProfile::initializeProfile): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): * heap/AlignedMemoryAllocator.cpp: (JSC::AlignedMemoryAllocator::registerAllocator): (JSC::AlignedMemoryAllocator::registerSubspace): * heap/AlignedMemoryAllocator.h: (JSC::AlignedMemoryAllocator::firstAllocator const): * heap/AllocationFailureMode.h: Added. * heap/CompleteSubspace.cpp: Added. (JSC::CompleteSubspace::CompleteSubspace): (JSC::CompleteSubspace::~CompleteSubspace): (JSC::CompleteSubspace::allocatorFor): (JSC::CompleteSubspace::allocate): (JSC::CompleteSubspace::allocateNonVirtual): (JSC::CompleteSubspace::allocatorForSlow): (JSC::CompleteSubspace::allocateSlow): (JSC::CompleteSubspace::tryAllocateSlow): * heap/CompleteSubspace.h: Added. (JSC::CompleteSubspace::offsetOfAllocatorForSizeStep): (JSC::CompleteSubspace::allocatorForSizeStep): (JSC::CompleteSubspace::allocatorForNonVirtual): * heap/HeapCellType.cpp: Added. (JSC::HeapCellType::HeapCellType): (JSC::HeapCellType::~HeapCellType): (JSC::HeapCellType::finishSweep): (JSC::HeapCellType::destroy): * heap/HeapCellType.h: Added. (JSC::HeapCellType::attributes const): * heap/IsoAlignedMemoryAllocator.cpp: Added. (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::dump const): * heap/IsoAlignedMemoryAllocator.h: Added. * heap/IsoSubspace.cpp: Added. (JSC::IsoSubspace::IsoSubspace): (JSC::IsoSubspace::~IsoSubspace): (JSC::IsoSubspace::allocatorFor): (JSC::IsoSubspace::allocatorForNonVirtual): (JSC::IsoSubspace::allocate): (JSC::IsoSubspace::allocateNonVirtual): * heap/IsoSubspace.h: Added. (JSC::IsoSubspace::size const): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::setSubspace): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): Deleted. (JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const): (JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator): * heap/MarkedAllocatorInlines.h: (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::tryAllocate): Deleted. * heap/MarkedBlock.h: * heap/MarkedBlockInlines.h: (JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType): (JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::addMarkedAllocator): * heap/MarkedSpace.h: * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::initialize): (JSC::Subspace::finishSweep): (JSC::Subspace::destroy): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): (): Deleted. (JSC::Subspace::allocate): Deleted. (JSC::Subspace::tryAllocate): Deleted. (JSC::Subspace::allocatorForSlow): Deleted. (JSC::Subspace::allocateSlow): Deleted. (JSC::Subspace::tryAllocateSlow): Deleted. (JSC::Subspace::didAllocate): Deleted. * heap/Subspace.h: (JSC::Subspace::heapCellType const): (JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const): (JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator): (JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted. (JSC::Subspace::allocatorForSizeStep): Deleted. (JSC::Subspace::tryAllocatorFor): Deleted. (JSC::Subspace::allocatorFor): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): (JSC::AssemblyHelpers::emitAllocateVariableSizedCell): * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): * runtime/DirectArguments.h: (JSC::DirectArguments::subspaceFor): * runtime/DirectEvalExecutable.h: * runtime/EvalExecutable.h: * runtime/ExecutableBase.h: (JSC::ExecutableBase::subspaceFor): * runtime/FunctionExecutable.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): * runtime/IndirectEvalExecutable.h: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): * runtime/JSCell.h: (JSC::subspaceFor): * runtime/JSCellInlines.h: (JSC::JSCell::subspaceFor): (JSC::tryAllocateCellHelper): (JSC::allocateCell): (JSC::tryAllocateCell): * runtime/JSDestructibleObject.h: (JSC::JSDestructibleObject::subspaceFor): * runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp. (JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::finishSweep): (JSC::JSDestructibleObjectHeapCellType::destroy): (JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::finishSweep): Deleted. (JSC::JSDestructibleObjectSubspace::destroy): Deleted. * runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h. * runtime/JSDestructibleObjectSubspace.cpp: Removed. * runtime/JSDestructibleObjectSubspace.h: Removed. * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::subspaceFor): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::subspaceFor): * runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp. (JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::finishSweep): (JSC::JSSegmentedVariableObjectHeapCellType::destroy): (JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted. (JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted. * runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h. * runtime/JSSegmentedVariableObjectSubspace.cpp: Removed. * runtime/JSSegmentedVariableObjectSubspace.h: Removed. * runtime/JSString.h: (JSC::JSString::subspaceFor): * runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp. (JSC::JSStringHeapCellType::JSStringHeapCellType): (JSC::JSStringHeapCellType::~JSStringHeapCellType): (JSC::JSStringHeapCellType::finishSweep): (JSC::JSStringHeapCellType::destroy): (JSC::JSStringSubspace::JSStringSubspace): Deleted. (JSC::JSStringSubspace::~JSStringSubspace): Deleted. (JSC::JSStringSubspace::finishSweep): Deleted. (JSC::JSStringSubspace::destroy): Deleted. * runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h. * runtime/JSStringSubspace.cpp: Removed. * runtime/JSStringSubspace.h: Removed. * runtime/ModuleProgramExecutable.h: * runtime/NativeExecutable.h: * runtime/ProgramExecutable.h: * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/ScopedArguments.h: (JSC::ScopedArguments::subspaceFor): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): * wasm/js/JSWebAssemblyCodeBlock.h: * wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp. (JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep): (JSC::JSWebAssemblyCodeBlockHeapCellType::destroy): (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted. * wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed. * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed. * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no new behavior. Adopting changes in JSC Subspace API. * ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added. * ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added. * bindings/js/JSDOMWrapper.cpp: (WebCore::outputConstraintSubspaceFor): (WebCore::globalObjectOutputConstraintSubspaceFor): * bindings/js/JSDOMWrapper.h: * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * bindings/js/WebCoreJSClientData.h: (WebCore::JSVMClientData::outputConstraintSpace): (WebCore::JSVMClientData::globalObjectOutputConstraintSpace): * bindings/scripts/CodeGeneratorJS.pm: (GenerateHeader): Source/WTF: One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that things get added to it at the end rather that at the beginning. In this patch, I use this to put the same node on multiple lists, which our existing linked list templates also don't support. This adds a new linked list that does those things: - It supports append(). It could also support prepend(), but currently there is no need for it. - It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the list uses to set next. * WTF.xcodeproj/project.pbxproj: * wtf/SinglyLinkedListWithTail.h: Added. (WTF::SinglyLinkedListWithTail::isEmpty const): (WTF::SinglyLinkedListWithTail::append): (WTF::SinglyLinkedListWithTail::first const): (WTF::SinglyLinkedListWithTail::last const): Canonical link: https://commits.webkit.org/196172@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-30 04:39:50 +00:00
namespace WTF {
CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00
GC should support isoheaps https://bugs.webkit.org/show_bug.cgi?id=179288 Reviewed by Saam Barati. Source/JavaScriptCore: This expands the power of the Subspace API in JSC: - Everything associated with describing the types of objects is now part of the HeapCellType class. We have different HeapCellTypes for different destruction strategies. Any Subspace can use any HeapCellType; these are orthogonal things. - There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual pages but releases the physical pages as part of the respective allocator's scavenging policy (the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for IsoSubspace). So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it for more things. This does not have any effect on JetStream (0.18% faster with p = 0.69). * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/ObjectAllocationProfileInlines.h: (JSC::ObjectAllocationProfile::initializeProfile): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): * heap/AlignedMemoryAllocator.cpp: (JSC::AlignedMemoryAllocator::registerAllocator): (JSC::AlignedMemoryAllocator::registerSubspace): * heap/AlignedMemoryAllocator.h: (JSC::AlignedMemoryAllocator::firstAllocator const): * heap/AllocationFailureMode.h: Added. * heap/CompleteSubspace.cpp: Added. (JSC::CompleteSubspace::CompleteSubspace): (JSC::CompleteSubspace::~CompleteSubspace): (JSC::CompleteSubspace::allocatorFor): (JSC::CompleteSubspace::allocate): (JSC::CompleteSubspace::allocateNonVirtual): (JSC::CompleteSubspace::allocatorForSlow): (JSC::CompleteSubspace::allocateSlow): (JSC::CompleteSubspace::tryAllocateSlow): * heap/CompleteSubspace.h: Added. (JSC::CompleteSubspace::offsetOfAllocatorForSizeStep): (JSC::CompleteSubspace::allocatorForSizeStep): (JSC::CompleteSubspace::allocatorForNonVirtual): * heap/HeapCellType.cpp: Added. (JSC::HeapCellType::HeapCellType): (JSC::HeapCellType::~HeapCellType): (JSC::HeapCellType::finishSweep): (JSC::HeapCellType::destroy): * heap/HeapCellType.h: Added. (JSC::HeapCellType::attributes const): * heap/IsoAlignedMemoryAllocator.cpp: Added. (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::dump const): * heap/IsoAlignedMemoryAllocator.h: Added. * heap/IsoSubspace.cpp: Added. (JSC::IsoSubspace::IsoSubspace): (JSC::IsoSubspace::~IsoSubspace): (JSC::IsoSubspace::allocatorFor): (JSC::IsoSubspace::allocatorForNonVirtual): (JSC::IsoSubspace::allocate): (JSC::IsoSubspace::allocateNonVirtual): * heap/IsoSubspace.h: Added. (JSC::IsoSubspace::size const): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::setSubspace): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): Deleted. (JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const): (JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator): * heap/MarkedAllocatorInlines.h: (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::tryAllocate): Deleted. * heap/MarkedBlock.h: * heap/MarkedBlockInlines.h: (JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType): (JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::addMarkedAllocator): * heap/MarkedSpace.h: * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::initialize): (JSC::Subspace::finishSweep): (JSC::Subspace::destroy): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): (): Deleted. (JSC::Subspace::allocate): Deleted. (JSC::Subspace::tryAllocate): Deleted. (JSC::Subspace::allocatorForSlow): Deleted. (JSC::Subspace::allocateSlow): Deleted. (JSC::Subspace::tryAllocateSlow): Deleted. (JSC::Subspace::didAllocate): Deleted. * heap/Subspace.h: (JSC::Subspace::heapCellType const): (JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const): (JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator): (JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted. (JSC::Subspace::allocatorForSizeStep): Deleted. (JSC::Subspace::tryAllocatorFor): Deleted. (JSC::Subspace::allocatorFor): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): (JSC::AssemblyHelpers::emitAllocateVariableSizedCell): * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): * runtime/DirectArguments.h: (JSC::DirectArguments::subspaceFor): * runtime/DirectEvalExecutable.h: * runtime/EvalExecutable.h: * runtime/ExecutableBase.h: (JSC::ExecutableBase::subspaceFor): * runtime/FunctionExecutable.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): * runtime/IndirectEvalExecutable.h: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): * runtime/JSCell.h: (JSC::subspaceFor): * runtime/JSCellInlines.h: (JSC::JSCell::subspaceFor): (JSC::tryAllocateCellHelper): (JSC::allocateCell): (JSC::tryAllocateCell): * runtime/JSDestructibleObject.h: (JSC::JSDestructibleObject::subspaceFor): * runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp. (JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::finishSweep): (JSC::JSDestructibleObjectHeapCellType::destroy): (JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::finishSweep): Deleted. (JSC::JSDestructibleObjectSubspace::destroy): Deleted. * runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h. * runtime/JSDestructibleObjectSubspace.cpp: Removed. * runtime/JSDestructibleObjectSubspace.h: Removed. * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::subspaceFor): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::subspaceFor): * runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp. (JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::finishSweep): (JSC::JSSegmentedVariableObjectHeapCellType::destroy): (JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted. (JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted. * runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h. * runtime/JSSegmentedVariableObjectSubspace.cpp: Removed. * runtime/JSSegmentedVariableObjectSubspace.h: Removed. * runtime/JSString.h: (JSC::JSString::subspaceFor): * runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp. (JSC::JSStringHeapCellType::JSStringHeapCellType): (JSC::JSStringHeapCellType::~JSStringHeapCellType): (JSC::JSStringHeapCellType::finishSweep): (JSC::JSStringHeapCellType::destroy): (JSC::JSStringSubspace::JSStringSubspace): Deleted. (JSC::JSStringSubspace::~JSStringSubspace): Deleted. (JSC::JSStringSubspace::finishSweep): Deleted. (JSC::JSStringSubspace::destroy): Deleted. * runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h. * runtime/JSStringSubspace.cpp: Removed. * runtime/JSStringSubspace.h: Removed. * runtime/ModuleProgramExecutable.h: * runtime/NativeExecutable.h: * runtime/ProgramExecutable.h: * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/ScopedArguments.h: (JSC::ScopedArguments::subspaceFor): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): * wasm/js/JSWebAssemblyCodeBlock.h: * wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp. (JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep): (JSC::JSWebAssemblyCodeBlockHeapCellType::destroy): (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted. * wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed. * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed. * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no new behavior. Adopting changes in JSC Subspace API. * ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added. * ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added. * bindings/js/JSDOMWrapper.cpp: (WebCore::outputConstraintSubspaceFor): (WebCore::globalObjectOutputConstraintSubspaceFor): * bindings/js/JSDOMWrapper.h: * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * bindings/js/WebCoreJSClientData.h: (WebCore::JSVMClientData::outputConstraintSpace): (WebCore::JSVMClientData::globalObjectOutputConstraintSpace): * bindings/scripts/CodeGeneratorJS.pm: (GenerateHeader): Source/WTF: One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that things get added to it at the end rather that at the beginning. In this patch, I use this to put the same node on multiple lists, which our existing linked list templates also don't support. This adds a new linked list that does those things: - It supports append(). It could also support prepend(), but currently there is no need for it. - It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the list uses to set next. * WTF.xcodeproj/project.pbxproj: * wtf/SinglyLinkedListWithTail.h: Added. (WTF::SinglyLinkedListWithTail::isEmpty const): (WTF::SinglyLinkedListWithTail::append): (WTF::SinglyLinkedListWithTail::first const): (WTF::SinglyLinkedListWithTail::last const): Canonical link: https://commits.webkit.org/196172@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-30 04:39:50 +00:00
template<typename T>
class SinglyLinkedListWithTail {
CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00
public:
GC should support isoheaps https://bugs.webkit.org/show_bug.cgi?id=179288 Reviewed by Saam Barati. Source/JavaScriptCore: This expands the power of the Subspace API in JSC: - Everything associated with describing the types of objects is now part of the HeapCellType class. We have different HeapCellTypes for different destruction strategies. Any Subspace can use any HeapCellType; these are orthogonal things. - There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual pages but releases the physical pages as part of the respective allocator's scavenging policy (the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for IsoSubspace). So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it for more things. This does not have any effect on JetStream (0.18% faster with p = 0.69). * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/ObjectAllocationProfileInlines.h: (JSC::ObjectAllocationProfile::initializeProfile): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): * heap/AlignedMemoryAllocator.cpp: (JSC::AlignedMemoryAllocator::registerAllocator): (JSC::AlignedMemoryAllocator::registerSubspace): * heap/AlignedMemoryAllocator.h: (JSC::AlignedMemoryAllocator::firstAllocator const): * heap/AllocationFailureMode.h: Added. * heap/CompleteSubspace.cpp: Added. (JSC::CompleteSubspace::CompleteSubspace): (JSC::CompleteSubspace::~CompleteSubspace): (JSC::CompleteSubspace::allocatorFor): (JSC::CompleteSubspace::allocate): (JSC::CompleteSubspace::allocateNonVirtual): (JSC::CompleteSubspace::allocatorForSlow): (JSC::CompleteSubspace::allocateSlow): (JSC::CompleteSubspace::tryAllocateSlow): * heap/CompleteSubspace.h: Added. (JSC::CompleteSubspace::offsetOfAllocatorForSizeStep): (JSC::CompleteSubspace::allocatorForSizeStep): (JSC::CompleteSubspace::allocatorForNonVirtual): * heap/HeapCellType.cpp: Added. (JSC::HeapCellType::HeapCellType): (JSC::HeapCellType::~HeapCellType): (JSC::HeapCellType::finishSweep): (JSC::HeapCellType::destroy): * heap/HeapCellType.h: Added. (JSC::HeapCellType::attributes const): * heap/IsoAlignedMemoryAllocator.cpp: Added. (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::dump const): * heap/IsoAlignedMemoryAllocator.h: Added. * heap/IsoSubspace.cpp: Added. (JSC::IsoSubspace::IsoSubspace): (JSC::IsoSubspace::~IsoSubspace): (JSC::IsoSubspace::allocatorFor): (JSC::IsoSubspace::allocatorForNonVirtual): (JSC::IsoSubspace::allocate): (JSC::IsoSubspace::allocateNonVirtual): * heap/IsoSubspace.h: Added. (JSC::IsoSubspace::size const): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::setSubspace): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): Deleted. (JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const): (JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator): * heap/MarkedAllocatorInlines.h: (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::tryAllocate): Deleted. * heap/MarkedBlock.h: * heap/MarkedBlockInlines.h: (JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType): (JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::addMarkedAllocator): * heap/MarkedSpace.h: * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::initialize): (JSC::Subspace::finishSweep): (JSC::Subspace::destroy): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): (): Deleted. (JSC::Subspace::allocate): Deleted. (JSC::Subspace::tryAllocate): Deleted. (JSC::Subspace::allocatorForSlow): Deleted. (JSC::Subspace::allocateSlow): Deleted. (JSC::Subspace::tryAllocateSlow): Deleted. (JSC::Subspace::didAllocate): Deleted. * heap/Subspace.h: (JSC::Subspace::heapCellType const): (JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const): (JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator): (JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted. (JSC::Subspace::allocatorForSizeStep): Deleted. (JSC::Subspace::tryAllocatorFor): Deleted. (JSC::Subspace::allocatorFor): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): (JSC::AssemblyHelpers::emitAllocateVariableSizedCell): * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): * runtime/DirectArguments.h: (JSC::DirectArguments::subspaceFor): * runtime/DirectEvalExecutable.h: * runtime/EvalExecutable.h: * runtime/ExecutableBase.h: (JSC::ExecutableBase::subspaceFor): * runtime/FunctionExecutable.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): * runtime/IndirectEvalExecutable.h: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): * runtime/JSCell.h: (JSC::subspaceFor): * runtime/JSCellInlines.h: (JSC::JSCell::subspaceFor): (JSC::tryAllocateCellHelper): (JSC::allocateCell): (JSC::tryAllocateCell): * runtime/JSDestructibleObject.h: (JSC::JSDestructibleObject::subspaceFor): * runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp. (JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::finishSweep): (JSC::JSDestructibleObjectHeapCellType::destroy): (JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::finishSweep): Deleted. (JSC::JSDestructibleObjectSubspace::destroy): Deleted. * runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h. * runtime/JSDestructibleObjectSubspace.cpp: Removed. * runtime/JSDestructibleObjectSubspace.h: Removed. * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::subspaceFor): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::subspaceFor): * runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp. (JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::finishSweep): (JSC::JSSegmentedVariableObjectHeapCellType::destroy): (JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted. (JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted. * runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h. * runtime/JSSegmentedVariableObjectSubspace.cpp: Removed. * runtime/JSSegmentedVariableObjectSubspace.h: Removed. * runtime/JSString.h: (JSC::JSString::subspaceFor): * runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp. (JSC::JSStringHeapCellType::JSStringHeapCellType): (JSC::JSStringHeapCellType::~JSStringHeapCellType): (JSC::JSStringHeapCellType::finishSweep): (JSC::JSStringHeapCellType::destroy): (JSC::JSStringSubspace::JSStringSubspace): Deleted. (JSC::JSStringSubspace::~JSStringSubspace): Deleted. (JSC::JSStringSubspace::finishSweep): Deleted. (JSC::JSStringSubspace::destroy): Deleted. * runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h. * runtime/JSStringSubspace.cpp: Removed. * runtime/JSStringSubspace.h: Removed. * runtime/ModuleProgramExecutable.h: * runtime/NativeExecutable.h: * runtime/ProgramExecutable.h: * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/ScopedArguments.h: (JSC::ScopedArguments::subspaceFor): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): * wasm/js/JSWebAssemblyCodeBlock.h: * wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp. (JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep): (JSC::JSWebAssemblyCodeBlockHeapCellType::destroy): (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted. * wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed. * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed. * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no new behavior. Adopting changes in JSC Subspace API. * ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added. * ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added. * bindings/js/JSDOMWrapper.cpp: (WebCore::outputConstraintSubspaceFor): (WebCore::globalObjectOutputConstraintSubspaceFor): * bindings/js/JSDOMWrapper.h: * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * bindings/js/WebCoreJSClientData.h: (WebCore::JSVMClientData::outputConstraintSpace): (WebCore::JSVMClientData::globalObjectOutputConstraintSpace): * bindings/scripts/CodeGeneratorJS.pm: (GenerateHeader): Source/WTF: One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that things get added to it at the end rather that at the beginning. In this patch, I use this to put the same node on multiple lists, which our existing linked list templates also don't support. This adds a new linked list that does those things: - It supports append(). It could also support prepend(), but currently there is no need for it. - It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the list uses to set next. * WTF.xcodeproj/project.pbxproj: * wtf/SinglyLinkedListWithTail.h: Added. (WTF::SinglyLinkedListWithTail::isEmpty const): (WTF::SinglyLinkedListWithTail::append): (WTF::SinglyLinkedListWithTail::first const): (WTF::SinglyLinkedListWithTail::last const): Canonical link: https://commits.webkit.org/196172@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-30 04:39:50 +00:00
bool isEmpty() const { return !m_first; }
template<typename SetNextFunc>
void append(SetNextFunc&& setNextFunc, T* node)
{
if (!m_first) {
RELEASE_ASSERT(!m_last);
m_first = node;
m_last = node;
return;
}
std::forward<SetNextFunc>(setNextFunc)(m_last, node);
m_last = node;
}
CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00
GC should support isoheaps https://bugs.webkit.org/show_bug.cgi?id=179288 Reviewed by Saam Barati. Source/JavaScriptCore: This expands the power of the Subspace API in JSC: - Everything associated with describing the types of objects is now part of the HeapCellType class. We have different HeapCellTypes for different destruction strategies. Any Subspace can use any HeapCellType; these are orthogonal things. - There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual pages but releases the physical pages as part of the respective allocator's scavenging policy (the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for IsoSubspace). So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it for more things. This does not have any effect on JetStream (0.18% faster with p = 0.69). * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/ObjectAllocationProfileInlines.h: (JSC::ObjectAllocationProfile::initializeProfile): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): * heap/AlignedMemoryAllocator.cpp: (JSC::AlignedMemoryAllocator::registerAllocator): (JSC::AlignedMemoryAllocator::registerSubspace): * heap/AlignedMemoryAllocator.h: (JSC::AlignedMemoryAllocator::firstAllocator const): * heap/AllocationFailureMode.h: Added. * heap/CompleteSubspace.cpp: Added. (JSC::CompleteSubspace::CompleteSubspace): (JSC::CompleteSubspace::~CompleteSubspace): (JSC::CompleteSubspace::allocatorFor): (JSC::CompleteSubspace::allocate): (JSC::CompleteSubspace::allocateNonVirtual): (JSC::CompleteSubspace::allocatorForSlow): (JSC::CompleteSubspace::allocateSlow): (JSC::CompleteSubspace::tryAllocateSlow): * heap/CompleteSubspace.h: Added. (JSC::CompleteSubspace::offsetOfAllocatorForSizeStep): (JSC::CompleteSubspace::allocatorForSizeStep): (JSC::CompleteSubspace::allocatorForNonVirtual): * heap/HeapCellType.cpp: Added. (JSC::HeapCellType::HeapCellType): (JSC::HeapCellType::~HeapCellType): (JSC::HeapCellType::finishSweep): (JSC::HeapCellType::destroy): * heap/HeapCellType.h: Added. (JSC::HeapCellType::attributes const): * heap/IsoAlignedMemoryAllocator.cpp: Added. (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::dump const): * heap/IsoAlignedMemoryAllocator.h: Added. * heap/IsoSubspace.cpp: Added. (JSC::IsoSubspace::IsoSubspace): (JSC::IsoSubspace::~IsoSubspace): (JSC::IsoSubspace::allocatorFor): (JSC::IsoSubspace::allocatorForNonVirtual): (JSC::IsoSubspace::allocate): (JSC::IsoSubspace::allocateNonVirtual): * heap/IsoSubspace.h: Added. (JSC::IsoSubspace::size const): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::setSubspace): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): Deleted. (JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const): (JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator): * heap/MarkedAllocatorInlines.h: (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::tryAllocate): Deleted. * heap/MarkedBlock.h: * heap/MarkedBlockInlines.h: (JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType): (JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::addMarkedAllocator): * heap/MarkedSpace.h: * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::initialize): (JSC::Subspace::finishSweep): (JSC::Subspace::destroy): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): (): Deleted. (JSC::Subspace::allocate): Deleted. (JSC::Subspace::tryAllocate): Deleted. (JSC::Subspace::allocatorForSlow): Deleted. (JSC::Subspace::allocateSlow): Deleted. (JSC::Subspace::tryAllocateSlow): Deleted. (JSC::Subspace::didAllocate): Deleted. * heap/Subspace.h: (JSC::Subspace::heapCellType const): (JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const): (JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator): (JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted. (JSC::Subspace::allocatorForSizeStep): Deleted. (JSC::Subspace::tryAllocatorFor): Deleted. (JSC::Subspace::allocatorFor): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): (JSC::AssemblyHelpers::emitAllocateVariableSizedCell): * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): * runtime/DirectArguments.h: (JSC::DirectArguments::subspaceFor): * runtime/DirectEvalExecutable.h: * runtime/EvalExecutable.h: * runtime/ExecutableBase.h: (JSC::ExecutableBase::subspaceFor): * runtime/FunctionExecutable.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): * runtime/IndirectEvalExecutable.h: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): * runtime/JSCell.h: (JSC::subspaceFor): * runtime/JSCellInlines.h: (JSC::JSCell::subspaceFor): (JSC::tryAllocateCellHelper): (JSC::allocateCell): (JSC::tryAllocateCell): * runtime/JSDestructibleObject.h: (JSC::JSDestructibleObject::subspaceFor): * runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp. (JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::finishSweep): (JSC::JSDestructibleObjectHeapCellType::destroy): (JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::finishSweep): Deleted. (JSC::JSDestructibleObjectSubspace::destroy): Deleted. * runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h. * runtime/JSDestructibleObjectSubspace.cpp: Removed. * runtime/JSDestructibleObjectSubspace.h: Removed. * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::subspaceFor): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::subspaceFor): * runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp. (JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::finishSweep): (JSC::JSSegmentedVariableObjectHeapCellType::destroy): (JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted. (JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted. * runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h. * runtime/JSSegmentedVariableObjectSubspace.cpp: Removed. * runtime/JSSegmentedVariableObjectSubspace.h: Removed. * runtime/JSString.h: (JSC::JSString::subspaceFor): * runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp. (JSC::JSStringHeapCellType::JSStringHeapCellType): (JSC::JSStringHeapCellType::~JSStringHeapCellType): (JSC::JSStringHeapCellType::finishSweep): (JSC::JSStringHeapCellType::destroy): (JSC::JSStringSubspace::JSStringSubspace): Deleted. (JSC::JSStringSubspace::~JSStringSubspace): Deleted. (JSC::JSStringSubspace::finishSweep): Deleted. (JSC::JSStringSubspace::destroy): Deleted. * runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h. * runtime/JSStringSubspace.cpp: Removed. * runtime/JSStringSubspace.h: Removed. * runtime/ModuleProgramExecutable.h: * runtime/NativeExecutable.h: * runtime/ProgramExecutable.h: * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/ScopedArguments.h: (JSC::ScopedArguments::subspaceFor): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): * wasm/js/JSWebAssemblyCodeBlock.h: * wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp. (JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep): (JSC::JSWebAssemblyCodeBlockHeapCellType::destroy): (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted. * wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed. * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed. * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no new behavior. Adopting changes in JSC Subspace API. * ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added. * ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added. * bindings/js/JSDOMWrapper.cpp: (WebCore::outputConstraintSubspaceFor): (WebCore::globalObjectOutputConstraintSubspaceFor): * bindings/js/JSDOMWrapper.h: * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * bindings/js/WebCoreJSClientData.h: (WebCore::JSVMClientData::outputConstraintSpace): (WebCore::JSVMClientData::globalObjectOutputConstraintSpace): * bindings/scripts/CodeGeneratorJS.pm: (GenerateHeader): Source/WTF: One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that things get added to it at the end rather that at the beginning. In this patch, I use this to put the same node on multiple lists, which our existing linked list templates also don't support. This adds a new linked list that does those things: - It supports append(). It could also support prepend(), but currently there is no need for it. - It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the list uses to set next. * WTF.xcodeproj/project.pbxproj: * wtf/SinglyLinkedListWithTail.h: Added. (WTF::SinglyLinkedListWithTail::isEmpty const): (WTF::SinglyLinkedListWithTail::append): (WTF::SinglyLinkedListWithTail::first const): (WTF::SinglyLinkedListWithTail::last const): Canonical link: https://commits.webkit.org/196172@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-30 04:39:50 +00:00
T* first() const { return m_first; }
T* last() const { return m_last; }
private:
T* m_first { nullptr };
T* m_last { nullptr };
CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00
};
GC should support isoheaps https://bugs.webkit.org/show_bug.cgi?id=179288 Reviewed by Saam Barati. Source/JavaScriptCore: This expands the power of the Subspace API in JSC: - Everything associated with describing the types of objects is now part of the HeapCellType class. We have different HeapCellTypes for different destruction strategies. Any Subspace can use any HeapCellType; these are orthogonal things. - There are now two variants of Subspace: CompleteSubspace, which can allocate any size objects using any AlignedMemoryAllocator; and IsoSubspace, which can allocate just one size of object and uses a special virtual memory pool for that purpose. Like bmalloc's IsoHeap, IsoSubspace hoards virtual pages but releases the physical pages as part of the respective allocator's scavenging policy (the Scavenger in bmalloc for IsoHeap and the incremental sweep and full sweep in Riptide for IsoSubspace). So far, this patch just puts subtypes of ExecutableBase in IsoSubspaces. If it works, we can use it for more things. This does not have any effect on JetStream (0.18% faster with p = 0.69). * JavaScriptCore.xcodeproj/project.pbxproj: * Sources.txt: * bytecode/AccessCase.cpp: (JSC::AccessCase::generateImpl): * bytecode/ObjectAllocationProfileInlines.h: (JSC::ObjectAllocationProfile::initializeProfile): * dfg/DFGSpeculativeJIT.cpp: (JSC::DFG::SpeculativeJIT::emitAllocateRawObject): (JSC::DFG::SpeculativeJIT::compileMakeRope): (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage): (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage): * dfg/DFGSpeculativeJIT64.cpp: (JSC::DFG::SpeculativeJIT::compile): * ftl/FTLAbstractHeapRepository.h: * ftl/FTLLowerDFGToB3.cpp: (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject): (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl): (JSC::FTL::DFG::LowerDFGToB3::allocateObject): (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize): * heap/AlignedMemoryAllocator.cpp: (JSC::AlignedMemoryAllocator::registerAllocator): (JSC::AlignedMemoryAllocator::registerSubspace): * heap/AlignedMemoryAllocator.h: (JSC::AlignedMemoryAllocator::firstAllocator const): * heap/AllocationFailureMode.h: Added. * heap/CompleteSubspace.cpp: Added. (JSC::CompleteSubspace::CompleteSubspace): (JSC::CompleteSubspace::~CompleteSubspace): (JSC::CompleteSubspace::allocatorFor): (JSC::CompleteSubspace::allocate): (JSC::CompleteSubspace::allocateNonVirtual): (JSC::CompleteSubspace::allocatorForSlow): (JSC::CompleteSubspace::allocateSlow): (JSC::CompleteSubspace::tryAllocateSlow): * heap/CompleteSubspace.h: Added. (JSC::CompleteSubspace::offsetOfAllocatorForSizeStep): (JSC::CompleteSubspace::allocatorForSizeStep): (JSC::CompleteSubspace::allocatorForNonVirtual): * heap/HeapCellType.cpp: Added. (JSC::HeapCellType::HeapCellType): (JSC::HeapCellType::~HeapCellType): (JSC::HeapCellType::finishSweep): (JSC::HeapCellType::destroy): * heap/HeapCellType.h: Added. (JSC::HeapCellType::attributes const): * heap/IsoAlignedMemoryAllocator.cpp: Added. (JSC::IsoAlignedMemoryAllocator::IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::~IsoAlignedMemoryAllocator): (JSC::IsoAlignedMemoryAllocator::tryAllocateAlignedMemory): (JSC::IsoAlignedMemoryAllocator::freeAlignedMemory): (JSC::IsoAlignedMemoryAllocator::dump const): * heap/IsoAlignedMemoryAllocator.h: Added. * heap/IsoSubspace.cpp: Added. (JSC::IsoSubspace::IsoSubspace): (JSC::IsoSubspace::~IsoSubspace): (JSC::IsoSubspace::allocatorFor): (JSC::IsoSubspace::allocatorForNonVirtual): (JSC::IsoSubspace::allocate): (JSC::IsoSubspace::allocateNonVirtual): * heap/IsoSubspace.h: Added. (JSC::IsoSubspace::size const): * heap/MarkedAllocator.cpp: (JSC::MarkedAllocator::MarkedAllocator): (JSC::MarkedAllocator::setSubspace): (JSC::MarkedAllocator::allocateSlowCase): (JSC::MarkedAllocator::tryAllocateSlowCase): Deleted. (JSC::MarkedAllocator::allocateSlowCaseImpl): Deleted. * heap/MarkedAllocator.h: (JSC::MarkedAllocator::nextAllocatorInAlignedMemoryAllocator const): (JSC::MarkedAllocator::setNextAllocatorInAlignedMemoryAllocator): * heap/MarkedAllocatorInlines.h: (JSC::MarkedAllocator::allocate): (JSC::MarkedAllocator::tryAllocate): Deleted. * heap/MarkedBlock.h: * heap/MarkedBlockInlines.h: (JSC::MarkedBlock::Handle::finishSweepKnowingHeapCellType): (JSC::MarkedBlock::Handle::finishSweepKnowingSubspace): Deleted. * heap/MarkedSpace.cpp: (JSC::MarkedSpace::addMarkedAllocator): * heap/MarkedSpace.h: * heap/Subspace.cpp: (JSC::Subspace::Subspace): (JSC::Subspace::initialize): (JSC::Subspace::finishSweep): (JSC::Subspace::destroy): (JSC::Subspace::prepareForAllocation): (JSC::Subspace::findEmptyBlockToSteal): (): Deleted. (JSC::Subspace::allocate): Deleted. (JSC::Subspace::tryAllocate): Deleted. (JSC::Subspace::allocatorForSlow): Deleted. (JSC::Subspace::allocateSlow): Deleted. (JSC::Subspace::tryAllocateSlow): Deleted. (JSC::Subspace::didAllocate): Deleted. * heap/Subspace.h: (JSC::Subspace::heapCellType const): (JSC::Subspace::nextSubspaceInAlignedMemoryAllocator const): (JSC::Subspace::setNextSubspaceInAlignedMemoryAllocator): (JSC::Subspace::offsetOfAllocatorForSizeStep): Deleted. (JSC::Subspace::allocatorForSizeStep): Deleted. (JSC::Subspace::tryAllocatorFor): Deleted. (JSC::Subspace::allocatorFor): Deleted. * jit/AssemblyHelpers.h: (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize): (JSC::AssemblyHelpers::emitAllocateVariableSized): (JSC::AssemblyHelpers::emitAllocateVariableSizedCell): * jit/JITOpcodes.cpp: (JSC::JIT::emit_op_new_object): * runtime/ButterflyInlines.h: (JSC::Butterfly::createUninitialized): (JSC::Butterfly::tryCreate): (JSC::Butterfly::growArrayRight): * runtime/DirectArguments.cpp: (JSC::DirectArguments::overrideThings): * runtime/DirectArguments.h: (JSC::DirectArguments::subspaceFor): * runtime/DirectEvalExecutable.h: * runtime/EvalExecutable.h: * runtime/ExecutableBase.h: (JSC::ExecutableBase::subspaceFor): * runtime/FunctionExecutable.h: * runtime/GenericArgumentsInlines.h: (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor): * runtime/HashMapImpl.h: (JSC::HashMapBuffer::create): * runtime/IndirectEvalExecutable.h: * runtime/JSArray.cpp: (JSC::JSArray::tryCreateUninitializedRestricted): (JSC::JSArray::unshiftCountSlowCase): * runtime/JSArray.h: (JSC::JSArray::tryCreate): * runtime/JSArrayBufferView.cpp: (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext): * runtime/JSCell.h: (JSC::subspaceFor): * runtime/JSCellInlines.h: (JSC::JSCell::subspaceFor): (JSC::tryAllocateCellHelper): (JSC::allocateCell): (JSC::tryAllocateCell): * runtime/JSDestructibleObject.h: (JSC::JSDestructibleObject::subspaceFor): * runtime/JSDestructibleObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.cpp. (JSC::JSDestructibleObjectHeapCellType::JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::~JSDestructibleObjectHeapCellType): (JSC::JSDestructibleObjectHeapCellType::finishSweep): (JSC::JSDestructibleObjectHeapCellType::destroy): (JSC::JSDestructibleObjectSubspace::JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::~JSDestructibleObjectSubspace): Deleted. (JSC::JSDestructibleObjectSubspace::finishSweep): Deleted. (JSC::JSDestructibleObjectSubspace::destroy): Deleted. * runtime/JSDestructibleObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSDestructibleObjectSubspace.h. * runtime/JSDestructibleObjectSubspace.cpp: Removed. * runtime/JSDestructibleObjectSubspace.h: Removed. * runtime/JSLexicalEnvironment.h: (JSC::JSLexicalEnvironment::subspaceFor): * runtime/JSSegmentedVariableObject.h: (JSC::JSSegmentedVariableObject::subspaceFor): * runtime/JSSegmentedVariableObjectHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.cpp. (JSC::JSSegmentedVariableObjectHeapCellType::JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::~JSSegmentedVariableObjectHeapCellType): (JSC::JSSegmentedVariableObjectHeapCellType::finishSweep): (JSC::JSSegmentedVariableObjectHeapCellType::destroy): (JSC::JSSegmentedVariableObjectSubspace::JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::~JSSegmentedVariableObjectSubspace): Deleted. (JSC::JSSegmentedVariableObjectSubspace::finishSweep): Deleted. (JSC::JSSegmentedVariableObjectSubspace::destroy): Deleted. * runtime/JSSegmentedVariableObjectHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSSegmentedVariableObjectSubspace.h. * runtime/JSSegmentedVariableObjectSubspace.cpp: Removed. * runtime/JSSegmentedVariableObjectSubspace.h: Removed. * runtime/JSString.h: (JSC::JSString::subspaceFor): * runtime/JSStringHeapCellType.cpp: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.cpp. (JSC::JSStringHeapCellType::JSStringHeapCellType): (JSC::JSStringHeapCellType::~JSStringHeapCellType): (JSC::JSStringHeapCellType::finishSweep): (JSC::JSStringHeapCellType::destroy): (JSC::JSStringSubspace::JSStringSubspace): Deleted. (JSC::JSStringSubspace::~JSStringSubspace): Deleted. (JSC::JSStringSubspace::finishSweep): Deleted. (JSC::JSStringSubspace::destroy): Deleted. * runtime/JSStringHeapCellType.h: Copied from Source/JavaScriptCore/runtime/JSStringSubspace.h. * runtime/JSStringSubspace.cpp: Removed. * runtime/JSStringSubspace.h: Removed. * runtime/ModuleProgramExecutable.h: * runtime/NativeExecutable.h: * runtime/ProgramExecutable.h: * runtime/RegExpMatchesArray.h: (JSC::tryCreateUninitializedRegExpMatchesArray): * runtime/ScopedArguments.h: (JSC::ScopedArguments::subspaceFor): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: (JSC::VM::gigacageAuxiliarySpace): * wasm/js/JSWebAssemblyCodeBlock.h: * wasm/js/JSWebAssemblyCodeBlockHeapCellType.cpp: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.cpp. (JSC::JSWebAssemblyCodeBlockHeapCellType::JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::~JSWebAssemblyCodeBlockHeapCellType): (JSC::JSWebAssemblyCodeBlockHeapCellType::finishSweep): (JSC::JSWebAssemblyCodeBlockHeapCellType::destroy): (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): Deleted. (JSC::JSWebAssemblyCodeBlockSubspace::destroy): Deleted. * wasm/js/JSWebAssemblyCodeBlockHeapCellType.h: Copied from Source/JavaScriptCore/wasm/js/JSWebAssemblyCodeBlockSubspace.h. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Removed. * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Removed. * wasm/js/JSWebAssemblyMemory.h: (JSC::JSWebAssemblyMemory::subspaceFor): Source/WebCore: No new tests because no new behavior. Adopting changes in JSC Subspace API. * ForwardingHeaders/runtime/JSDestructibleObjectHeapCellType.h: Added. * ForwardingHeaders/runtime/JSSegmentedVariableObjectHeapCellType.h: Added. * bindings/js/JSDOMWrapper.cpp: (WebCore::outputConstraintSubspaceFor): (WebCore::globalObjectOutputConstraintSubspaceFor): * bindings/js/JSDOMWrapper.h: * bindings/js/WebCoreJSClientData.cpp: (WebCore::JSVMClientData::JSVMClientData): * bindings/js/WebCoreJSClientData.h: (WebCore::JSVMClientData::outputConstraintSpace): (WebCore::JSVMClientData::globalObjectOutputConstraintSpace): * bindings/scripts/CodeGeneratorJS.pm: (GenerateHeader): Source/WTF: One of my favorite data structures in the GC is a singly-linked list that knows its tail, so that things get added to it at the end rather that at the beginning. In this patch, I use this to put the same node on multiple lists, which our existing linked list templates also don't support. This adds a new linked list that does those things: - It supports append(). It could also support prepend(), but currently there is no need for it. - It supports nodes that are on multiple lists. The GC uses std::mem_fn() to create a lambda that the list uses to set next. * WTF.xcodeproj/project.pbxproj: * wtf/SinglyLinkedListWithTail.h: Added. (WTF::SinglyLinkedListWithTail::isEmpty const): (WTF::SinglyLinkedListWithTail::append): (WTF::SinglyLinkedListWithTail::first const): (WTF::SinglyLinkedListWithTail::last const): Canonical link: https://commits.webkit.org/196172@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@225314 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-11-30 04:39:50 +00:00
} // namespace WTF
using WTF::SinglyLinkedListWithTail;
CallLinkInfos belonging to Wasm->JS stubs need to be informed when we clearCode() from all Executables https://bugs.webkit.org/show_bug.cgi?id=171707 <rdar://problem/31891649> Reviewed by Filip Pizlo. This patch fixes a bug where a Wasm->JS IC call stub would go stale and point into a CodeBlock no longer owned by any executable. The problematic scenario is this: 1. We generate the call IC which has a branch on a callee check. This callee owns the Executable in question. If the branch succeeds, it will call code belonging to a particular CodeBlock associated with that Executable. 2. Heap::deleteAllCodeBlocks is called. This leads the Executable to clear its various CodeBlock references. 3. Wasm has no idea this happened, so now it has stale ICs that point into code from a CodeBlock no longer belonging to an Executable. This patch fixes the bug by informing all JSWebAssemblyCodeBlocks to unlink their CallLinkInfo when Heap::deleteAllCodeBlocks is called. We track all JSWebAssemblyCodeBlocks by creating a new subspace for them. This allows us to quickly iterate over the live JSWebAssemblyCodeBlocks in the heap. * CMakeLists.txt: * JavaScriptCore.xcodeproj/project.pbxproj: * heap/Heap.cpp: (JSC::Heap::deleteAllCodeBlocks): * heap/Subspace.h: * heap/SubspaceInlines.h: (JSC::Subspace::forEachLiveCell): * runtime/VM.cpp: (JSC::VM::VM): * runtime/VM.h: * wasm/js/JSWebAssemblyCodeBlock.cpp: (JSC::JSWebAssemblyCodeBlock::clearJSCallICs): * wasm/js/JSWebAssemblyCodeBlock.h: (JSC::JSWebAssemblyCodeBlock::createStructure): Deleted. (JSC::JSWebAssemblyCodeBlock::functionImportCount): Deleted. (JSC::JSWebAssemblyCodeBlock::module): Deleted. (JSC::JSWebAssemblyCodeBlock::jsEntrypointCalleeFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmEntrypointLoadLocationFromFunctionIndexSpace): Deleted. (JSC::JSWebAssemblyCodeBlock::wasmToJsCallStubForImport): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportWasmToJSStub): Deleted. (JSC::JSWebAssemblyCodeBlock::codeBlock): Deleted. (JSC::JSWebAssemblyCodeBlock::offsetOfImportStubs): Deleted. (JSC::JSWebAssemblyCodeBlock::allocationSize): Deleted. (JSC::JSWebAssemblyCodeBlock::importWasmToJSStub): Deleted. * wasm/js/JSWebAssemblyCodeBlockSubspace.cpp: Added. (JSC::JSWebAssemblyCodeBlockSubspace::JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::~JSWebAssemblyCodeBlockSubspace): (JSC::JSWebAssemblyCodeBlockSubspace::finishSweep): (JSC::JSWebAssemblyCodeBlockSubspace::destroy): * wasm/js/JSWebAssemblyCodeBlockSubspace.h: Added. Canonical link: https://commits.webkit.org/188765@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@216481 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2017-05-09 07:15:01 +00:00